Félix is a Data Engineer at MessageBird, an IT Telecom Startup based in Amsterdam. He joined the organisation ~2.5 years ago. With the data team of MessageBird he is working on building and managing various data pipelines with a focus on scalability and real-time.
He also worked on designing and deploying the ClickHouse cluster and now is busy with running and scaling it.
At MessageBird we send hundreds of millions of messages across the world every month, which generate billions of events. Providing real-time* analytics is key to being able to quickly deliver those messages to our customers. From measuring and optimising the performance of our platform, to providing insights to our customers, ClickHouse is being used far and wide inside our organisation.
In this talk, we look into how Clickhouse allows us to ingest a large amount of data and run complex analytical interactive queries. We also present the business needs that brought ClickHouse to our attention and detail the journey to its deployment. We cover the problems we faced, and how we dealt with them. We talk about our current Cloud production setup and how we deployed and use it. Last but not least, we talk about mistakes we did along the way and what we learned from running and maintaining a ClickHouse cluster by ourselves.