Portfolio item

Sensor data ingestion

A client of ours was going through a fast growth phase. Although this generally makes most people in a company happy, the engineers were facing a challenge: How do we scale our infrastructure to accommodate this growth?

The client offers dashboards for users to track their energy production and consumption. With more and more clients signing up, ingestion of data in their data warehouse was quickly becoming the bottleneck.

Data Minded stepped in to see what we could do without burdening the, already very busy, engineering team. After initial meetings with the client to understand the situation, we went to work. We built an architecture based on Cassandra and benchmarked its performance on low-cost cloud infrastructure. The picture below shows a histogram of the amount of rows ingested:

Our solution proved to be fast, very fast. Especially if you take into account the modest hardware it ran on. Not only could our system handle 30x more data than their existing system, it was also significantly cheaper. Based on calculations from the client, their existing database is at least 50x more expensive.

Last but not least, we made sure the engineering team was up-to-speed with the new system. We minimised the changes needed to their applications. And we took ownership of the parts that did change.