Automatically manage cloud compute for optimized costs and agility
Make sense of spend and chargeback to align with your business
Maximize AWS Spot savings and minimize disruptions for optimized scaling
Autonomously identify cost spikes early, with zero configuration
Huq produces a wide range of economic and societal benchmarks using mobility data that serves as a proxy for footfall in retail, transport, town centers, and workplaces. This anonymized resource – updated in near real time – incorporates high volumes of rich, descriptive data that enables complex analysis of consumer behavior and how that evolves. It’s an offering that’s becoming even more important for Huq’s customers as they navigate the wide-scale disruption to movement and footfall brought on by COVID-19.
“Our customers require increasingly fast turnaround times and more regular access to high-frequency data,” says Isambard Poulson, Co-founder, and CTO at Huq. “While previously, a hedge-fund customer might use our data to make mid-term or long-term decisions, customers of all sizes are now keen to understand changes in near real time: how new policies impact behavioral patterns, for example, providing an up-to-date view of how a particular country’s economy is faring as a result of COVID-19.”
Huq prides itself on being able to adapt to its customers’ needs. To make this happen, the company has to be extremely flexible. “Our mission is to make our data easily accessible by offering the right level of granularity for a customer’s needs,” says Isambard. “That means people with limited data engineering skills can use our analytics products via dashboard solutions, or simply receive a visualized PDF report, while more sophisticated customers might access up to billions of rows of anonymized geospatial data per day.” To accommodate these needs, Huq decided to migrate to Google Cloud in 2019 so that it could more easily adapt its data pipeline and react quickly to evolving customer demands.
Finding the right solutions based on a tailored approach
Isambard first turned to DoiT in May 2020 on the recommendation of another DoiT client, the CTO of a company in the investment sector with similar needs. “Having gone through previous evaluations with potential partners, I was initially skeptical of what they could offer us, but the client had a similar profile to us, so we knew it could be a good fit,” says Isambard. “It was beneficial to have a personal recommendation.” Huq has an agnostic attitude to providers, preferring open-source to proprietary technologies, so when looking for implementation support, it wanted a partner that kept its options open. “We didn’t want to be tied into a relationship where we might lose control of our internal processes,” says Isambard.
The DoiT International team impressed Huq with its tailored approach. “Many partners come in with an agenda, making suggestions based on their existing product knowledge rather than listening to your needs,” says Isambard. “DoiT took the time to gain a deep understanding of our requirements and offer solutions that aligned with those needs. We didn’t feel tied into anything we couldn’t walk away from.”
“Our business model is built around flexibility,” says Rebeca Fenoy-Anthony, Cloud Business Lead at DoiT International. “As a reseller for Google Cloud, our on-demand engagement model enables our customers to enter into a relationship with us, without any fixed constraints.”
These initial meetings reassured Huq that DoiT was the right fit. “The onboarding process was very smooth, and the tools that were immediately going to be available to us were highly compelling,” says Isambard.
Offering support that inspires clients with confidence
When DoiT came on board, Huq had already begun its migration to Google Cloud. “The first goal for the project was to migrate our core data processing pipeline from AWS to Google Cloud using Apache Beam and Google Dataflow,” says Isambard. Huq’s enriched data asset comprises 24 billion new rows of anonymized geospatial data every month, along with 200 million rows of continually-refreshed spatial reference data. Its data record incorporates 73 properties with diverse data types over five years of history, so its key requirements are being able to quickly run queries over this extensive data asset, and build and maintain multiple custom data pipelines.
The Huq team had completed the early stages of the implementation itself, but still had technical hurdles to overcome. DoiT helped to provide solutions, such as optimizing the machine types Huq uses on Dataflow. With DoiT’s support, Huq built a data pipeline that rapidly ingests, processes, analyzes, and exports almost one billion geo-location records every day. To do that, Huq uses Google Cloud Composer for its workflow orchestration service to schedule tasks such as loading the core data and managing the daily processing pipelines. On arrival, the source data is first loaded into Google Cloud Storage and enriched with Dataflow. Once the data has been processed, multiple pipelines transport it back into Cloud Storage or Google BigQuery for clients to access, or export it as Google Data Studio reports.
“As well as providing rapid answers to our questions, DoiT International reassured us that we were moving in the right direction,” says Isambard. “That gave us the confidence to continue our migration, and we now have an architecture that offers scale and flexibility on an economically-viable basis.”
Staying on top of costs with reOptimize CMP and superQuery IDE
One of the key goals for DoiT was to optimize Huq’s costs on Google Cloud. “As a rapidly growing company, we want to grow our volume without increasing costs,” says Isambard. That’s critical because, without optimization, the cost of running queries could become too high for Huq to continue offering the high-volume, quick-response approach its clients rely on. “It’s not just a question of our bottom line or profitability,” says Isambard. “Without optimizations, we’re not able to iterate rapidly and deliver for our clients. It has a tangible impact on our outputs.”
To do that, Huq uses a DoiT International technology platform called reOptimize, which provides enhanced visibility into cloud usage by breaking it down according to services, project, or cost allocation, as well as predictive analytics and other cost-optimization functions. In addition to offering tips on how to use Google BigQuery efficiently, such as using Flex Slots to respond to rapid demand, working with DoiT gives Huq access to superQuery, an AI-powered integrated development environment (IDE) for Google BigQuery.
“We use Google BigQuery extensively, so having unlimited access to the superQuery IDE tool is very helpful for us,” says Isambard. “It enables us to optimize the way we query our data, making it more efficient so we can respond more flexibly to our customers.”
“On the cloud, it can be hard to keep control of costs,” says Rebeca. “However, with a combination of the right engineering power, the right tools, and the right alerts, you can stay on top of those costs and make sure the cloud is working for you.”
Helping Huq to develop new services and stay competitive
With the help of DoiT International, Huq has been able to cut its cloud bill in half even while its data needs quadrupled. “Our volumes of data have grown rapidly since we started working with DoiT,” says Isambard. “We’re consuming about four times more data on a daily basis, from around 200 million data points a day to 800, but at the same time, our costs have fallen by 50%. In the context of a challenging economic environment, that’s something to be proud of.”
Now that it’s finished its migration and is able to rapidly provide access for customers at a wide range of granularities, Huq wants to focus on scaling up its business. “With our new stack, we could increase the volume of data we process 100-fold, without any changes to our stack” says Isambard. That means Huq can focus on creating new products for its customers, opening up the possibility of tracking particular sectors in real time.
“I’m really looking forward to seeing what we can achieve with Google Cloud and DoiT International,” says Isambard. “We’re in the process of continuous evolution, and DoiT will support us in the process of expanding our products, helping us to explore the technologies we need to generate those new services.”
With personalized recommendations for optimizing BigQuery costs and high-impact insights into team usage, Playgendary has a better grasp of their BigQuery spend and how their BI team operates.
E-commerce retailer Niceshops is creating more advanced data engineering and analytics pipelines by tapping into DoiT’s expertise.
Time management platform Clockwise leverages DoiT’s expertise as an AWS Premier Partner and the DoiT Product Portfolio to gain granular visibility into its AWS usage.
From cost optimization to cloud migration, machine learning and CloudOps, we’re here to make the public cloud easy — without the costs.
Ready to get started?
