Tapping into DoiT’s knowledge to build new data pipelines

With managed services and support from DoiT, Niceshops began expanding its data platform, creating more advanced data engineering and analytics pipelines. New data sources included competitors’ price monitoring solutions, marketing insights and financial reports. The Niceshops team also built new pipelines in-house to replace its legacy third-party ingestion tools. With enterprise level support around the clock, constant assessment of Niceshops’ solutions and processes against best practices, and continuous automated cost monitoring optimization, DoiT helped the team make the most of their Google Cloud setup.

“Building data pipelines has multiple components to it, from orchestration to scheduling and monitoring,” says Dovjak. “In a training session, DoiT shared useful tips and best practices with us, helping us to develop new pipelines on our own quickly.”

Speeding up development with first-class support

Whenever the Niceshops team ran into issues they couldn’t resolve in-house, DoiT was the first point of contact. The support tickets that were submitted via the DoiT Console were usually resolved on the same day, enabling Niceshops to speed up the development process.

“Whenever we had an issue, DoiT basically already had the answer ready because they had experienced the exact same issue with other customers several times before,” says Tosic.

Stefan Gajanovic, Data Engineer, Niceshops, adds: “Working with DoiT keeps us on the right track, and ensures that we’re not doing the wrong thing or using tools that aren’t scalable or not meant for a particular use case. That’s especially important with the Google Cloud toolkit, which grows and changes every day.”

Identifying and cutting the costs of ML models and data ingestion

Turning data into analytics comes with a price tag. Visibility is the first step towards cost reduction, but in a complex cloud environment, it’s not always easy to see where specific costs occur. DoiT helped the Niceshops team gain visibility into their cloud spend and identify inefficiencies.

“With DoiT’s help, we were able to set up a monitoring framework that gives us a daily overview of our costs. This helped us detect and avoid cost peaks and improve our queries,“ says Gajanovic. “We’re estimating that we can lower our data ingestion expenses by 50%, thanks to support from DoiT.”

The DoiT team also helped Niceshops adopt the best practices for using BigQuery Machine Learning (ML) models in Looker. “We discovered in-house that our BigQuery ML models were not configured in Looker correctly; they were executing continuously instead of monthly, which was driving up our BQ consumption significantly” says Tosic. “DoiT’s Looker experts helped us troubleshoot these issues and adapt our development processes, which helped us reduce our cloud spend by 30%.”