Lightricks Ingests 1B Events a Day on Google Cloud

DoiT guides the migration to BigQuery, Dataflow, and GKE, eliminating bottlenecks and powering real-time analytics across Lightricks' creator apps.

The Challenge

Lightricks' original hybrid system struggled to meet growing demand and support increasingly complex machine learning and analytics needs. The infrastructure created bottlenecks during critical business moments, with data uploads reaching limits and clusters shutting down during funding rounds and peak usage periods.

The Solution

Lightricks migrated to Google Cloud, leveraging BigQuery and Dataflow for automated data ingestion and analysis. The team deployed Google Kubernetes Engine for containerized infrastructure and utilized Vertex AI for machine learning models. DoiT International provided ongoing architectural support and problem-solving expertise.

Results

Processes around 1 billion events per day with BigQuery and Dataflow
Deployed working Kubernetes infrastructure on GKE in just weeks with minimal engineering resources
Eliminated infrastructure bottlenecks that previously caused system failures during critical business periods
Enabled real-time business intelligence for optimizing ad campaigns on terabytes of data

When, for instance, we wanted to create a cluster on GKE and attach it to our machine learning systems, DoiT ensured that our data lakes that we use for research and our on-premises compute functionality synched. We have an elaborate machine learning construction and DoiT provides ongoing support for everything from architecture to problem-solving.
Ofir Bibi, VP Research, Lightricks

Meeting the Challenge of Scale

Lightricks quickly found market success with apps like Facetune, Videoleap, and Photoleap. However, rapid growth exposed limitations in their hybrid cloud-on-premises infrastructure. The system struggled with GPU usage demands and couldn't support increasingly complex machine learning and analytics requirements. Critical failures occurred during funding rounds and peak usage periods, creating urgent need for a more robust platform.

Automating Data Ingestion at Scale

BigQuery and Dataflow became the foundation of Lightricks' data platform transformation. The company now ingests around 10,000 events per second, totaling a billion events daily. Dataflow's autoscale feature eliminated previous bottlenecks where data uploads reached system limits. This automation enables real-time business intelligence, allowing teams to optimize ad campaigns on terabytes of data and instantly target users from successful marketing campaigns.

Empowering Small Teams with Big Infrastructure

Google Kubernetes Engine dramatically reduced infrastructure complexity for Lightricks' lean engineering team. Within weeks, just a few engineers and DevOps personnel deployed a working Kubernetes infrastructure that would have been impossible with their previous system. The separation of storage and compute eliminated the infrastructure obstacles that previously hindered development, allowing teams to focus on delivering business value instead of maintaining systems.

Enhancing Machine Learning Capabilities

Google Cloud solved Lightricks' compute availability challenges for machine learning workloads. The team migrated from experimental cloud training in 2014 to a robust platform where compute resources are available on demand. Marketing, product optimization, and recommendation engine teams now create machine learning models on Compute Engine, with migration to Vertex AI enabling even faster scaling for recommendation systems and user interaction optimization.

Securing Third-Party Integrations

Google Cloud's integration capabilities enable Lightricks to securely connect with third-party services like Cloudinary and Elasticsearch. The platform provides secure traffic forwarding outside private networks without exposing systems to the public Internet. This security framework supports Lightricks' backend development as they build services on open-source technologies while maintaining robust protection.

Future Growth and Platform Expansion

Lightricks plans major backend service expansion in 2022, including shared profiles across apps and media upload capabilities. This growth will generate more data and require enhanced machine learning models. With Google Cloud's support, the company can achieve rapid, cost-effective scaling while developing their creators' platform and maintaining state-of-the-art content creation services.

See how DoiT helps cloud teams control spend

Explore how DoiT Cloud Intelligence helps teams improve visibility, governance, and unit economics across cloud environments.

Visit DoiT Talk to our team

What they say

What I really like about DoiT's approach is that you're very hands-on and proactive. Satyam would ping me a few times a sprint, letting me know about the most current features, checking in on how things are going. When we are going through a peak time, that proactiveness makes a real difference. Satyam always comes through whenever we need support and helps us leverage the right experts to get us where we need to be.

Chiamaka Ibeme, Engineering Manager, Platform

SELECT has made important cost data readily accessible. I will often pull it up during engineering design reviews so we can quickly evaluate cost impact and projections and factor that into our design decisions.

Douglas Zickuhr, Senior Data Platform Engineer at Personio

I love clicking through SELECT to understand how our environment and workloads are evolving. I probably check it every day. It's coffee and SELECT for me every morning.

Ian Fahey, Senior Analytics Engineer at Loop

You guys have the best UI experience that I've had of any software. It's like you just read my mind where, like, oh, I wish I could click there. Oh, I can

Diana Koshy, Sr. Director of Data Engineering at Kargo

SELECT feels like exactly what Paul and I would have built if we had locked ourselves in a room for 18 months to create our ideal monitoring solution.

Devin McGee, Data Engineering Lead at Home Chef

SELECT dramatically lowers the cognitive load to understanding Snowflake costs. I'm able to sit there and easily understand what's driving the cost. Not to blow smoke up your ass, but it's just so easy to do in your platform

Blake Baggett, Head of Data Operations at Entain

One of the most helpful cost rituals we've setup from SELECT is the weekly spend digest sent to Slack. I can start high level and ensure things are in check. If not, I can very quickly drill down into specific workloads which may have driven the cost spike and remediate them before they become a bigger issue.

Michael Revelo, Manager of data and analytics engineering at ClickUp

Through SELECT's automated savings feature and deep cost visibility, we were able to instantly lower our Snowflake spend by over 40% and achieve a 20X ROI on our SELECT investment.

Skyler Chi, SVP, GTM Productivity & Excellence at Exiger

Our costs had jumped up 3X as we scaled, so we're talking about 60% savings in Snowflake spend after adopting SELECT.

Edward Mancey, GTM Lead at Synthesia

DoiT was a true partner, not a vendor. They helped us understand the problem, refine the vision, and build something production-ready far faster than we could have on our own. Their expertise, responsiveness, and commitment made all the difference.

Dr. Anish Kapur, Founder & CEO, Promptly

DoiT's Customer Success and Forward Deployed Engineering teams work very closely with us. The regular sessions with our CSM keep us focused on the right priorities, and the FDEs provide the deep technical guidance we need to validate decisions and optimize our environment. That combination has been genuinely valuable for us.

Alexander Lundberg Santos, Platform Engineer at Extenda Retail

When we started working with DoiT, we deployed Flexsave to save time and reduce complexity. We still use it today. But what really stands out is the expert support. Having someone to collaborate with on deep cloud cost topics, someone who really understands the nuances, is incredibly valuable.

Jesper Terkelsen, CTO at Monta

DoiT's Cloud Accelerator turned our AI idea into a shipped product, saving at least three months of development and delivering reliable, explainable insights our customers trust.

Scott Desgrosseilliers, CEO and co-founder, Wicked Reports

Your cloud bill shouldn't be a mystery

Let us show you what ships this week.

Book a demo