Building globally available storage layers

Trends in Architecture and Design
4 April 1:50PM - 2:40PM @ Ballroom F

Experience level: 
50 minutes conference
More than two years ago Spil Games started building a global cross datacenter storage solution with MySQL as one its key components. This session will focus on our road to get there and share some of the best practices we got out of this. To give our users the best experience possible we needed to expand our datacenters globally and bring sharded data closer to our end users using a storage abstraction layer called SSP (Spil Storage Layer). This required a new way of thinking and a complete overhaul of our architecture. This meant we also had to change or replace many of our infrastructure components. When we started building this new architecture we also took into account that failure is bound to happen sometime. The new architecture has been built with failure in mind: we can cope with split brain scenarios and, to a certain degree, even with a full datacenter outage. We achieved all this by building Master datacenters containing the persistent storage and Satellite datacenters containing memory-based storage. This approach required new techniques like write-through for inter-datacenter writes and enabled us to implement asynchronous (fire and forget) writes to our persistent layer. At the same time we had to cope with non-shardable data that was required to be available in all our datacenters at all times and, of course, be available in a similar fashion as our sharded data. We solved this by creating a second storage layer ROAR (Read Often, Alter Rarely) that could cope with high concurrency reads. The architecture is being built with MySQL, Erlang, Memcache, Openstack and hybird cloud as its building blocks. Spil Games is a (social) gaming company that grew in a short time from an internet startup to a global online gaming leader with more than 180 million users.


Senior Database Administrator (freelance), DBArt
Has worked over nine years at Spil Games and contributed as a developer, IT Architect, Database Administrator, Database Specialist and Head of Database Engineering to the growth of the company. Recently he has started his own company that enables him to work on various MySQL and database performance related projects.