How Huq built a powerful data platform to deliver near real-time location data at scale

Finding the right solutions based on a tailored approach 

Isambard first turned to DoiT in May 2020 on the recommendation of another DoiT client, the CTO of a company in the investment sector with similar needs. “Having gone through previous evaluations with potential partners, I was initially skeptical of what they could offer us, but the client had a similar profile to us, so we knew it could be a good fit,” says Isambard. “It was beneficial to have a personal recommendation.” Huq has an agnostic attitude to providers, preferring open-source to proprietary technologies, so when looking for implementation support, it wanted a partner that kept its options open. “We didn’t want to be tied into a relationship where we might lose control of our internal processes,” says Isambard.

The DoiT International team impressed Huq with its tailored approach. “Many partners come in with an agenda, making suggestions based on their existing product knowledge rather than listening to your needs,” says Isambard. “DoiT took the time to gain a deep understanding of our requirements and offer solutions that aligned with those needs. We didn’t feel tied into anything we couldn’t walk away from.”

“Our business model is built around flexibility,” says Rebeca Fenoy-Anthony, Cloud Business Lead at DoiT International. “As a reseller for Google Cloud, our on-demand engagement model enables our customers to enter into a relationship with us, without any fixed constraints.”

These initial meetings reassured Huq that DoiT was the right fit. “The onboarding process was very smooth, and the tools that were immediately going to be available to us were highly compelling,” says Isambard.

Offering support that inspires clients with confidence

When DoiT came on board, Huq had already begun its migration to Google Cloud. “The first goal for the project was to migrate our core data processing pipeline from AWS to Google Cloud using Apache Beam and Google Dataflow,” says Isambard. Huq’s enriched data asset comprises 24 billion new rows of anonymized geospatial data every month, along with 200 million rows of continually-refreshed spatial reference data. Its data record incorporates 73 properties with diverse data types over five years of history, so its key requirements are being able to quickly run queries over this extensive data asset, and build and maintain multiple custom data pipelines.

The Huq team had completed the early stages of the implementation itself, but still had technical hurdles to overcome. DoiT helped to provide solutions, such as optimizing the machine types Huq uses on Dataflow. With DoiT’s support, Huq built a data pipeline that rapidly ingests, processes, analyzes, and exports almost one billion geo-location records every day. To do that, Huq uses Google Cloud Composer for its workflow orchestration service to schedule tasks such as loading the core data and managing the daily processing pipelines. On arrival, the source data is first loaded into Google Cloud Storage and enriched with Dataflow. Once the data has been processed, multiple pipelines transport it back into Cloud Storage or Google BigQuery for clients to access, or export it as Google Data Studio reports.

“As well as providing rapid answers to our questions, DoiT International reassured us that we were moving in the right direction,” says Isambard. “That gave us the confidence to continue our migration, and we now have an architecture that offers scale and flexibility on an economically-viable basis.”

Staying on top of costs with reOptimize CMP and superQuery IDE

One of the key goals for DoiT was to optimize Huq’s costs on Google Cloud. “As a rapidly growing company, we want to grow our volume without increasing costs,” says Isambard. That’s critical because, without optimization, the cost of running queries could become too high for Huq to continue offering the high-volume, quick-response approach its clients rely on. “It’s not just a question of our bottom line or profitability,” says Isambard. “Without optimizations, we’re not able to iterate rapidly and deliver for our clients. It has a tangible impact on our outputs.”

To do that, Huq uses a DoiT International technology platform called reOptimize, which provides enhanced visibility into cloud usage by breaking it down according to services, project, or cost allocation, as well as predictive analytics and other cost-optimization functions. In addition to offering tips on how to use Google BigQuery efficiently, such as using Flex Slots to respond to rapid demand, working with DoiT gives Huq access to superQuery, an AI-powered integrated development environment (IDE) for Google BigQuery.

“We use Google BigQuery extensively, so having unlimited access to the superQuery IDE tool is very helpful for us,” says Isambard. “It enables us to optimize the way we query our data, making it more efficient so we can respond more flexibly to our customers.”

“On the cloud, it can be hard to keep control of costs,” says Rebeca. “However, with a combination of the right engineering power, the right tools, and the right alerts, you can stay on top of those costs and make sure the cloud is working for you.”