Should You Use ClickHouse as a Main Operational Database?

ClickHouse as a main operational databaseFirst of all, this post is not a recommendation but more like a “what if” story. What if we use ClickHouse (which is a columnar analytical database) as our main datastore? Well, typically, an analytical database is not a replacement for a transactional or key/value datastore. However, ClickHouse is super efficient for timeseries and provides “sharding” out of the box (scalability beyond one node).  So can we use it as our main datastore?

Let’s imagine we are running a webservice and provide a public API. Public API as -a-service has become a good business model: examples include social networks like Facebook/Twitter, messaging as a service like Twilio, and even credit card authorization platforms like Marqeta. Let’s also imagine we need to store all messages (SMS messages, email messages, etc) we are sending and allow our customers to get various information about the message. This information can be a mix of analytical (OLAP) queries (i.e. how many messages was send for some time period and how much it cost) and a typical key/value queries like: “return 1 message by the message id”.

Using a columnar analytical database can be a big challenge here. Although such databases can be very efficient with counts and averages, some queries will be slow or simply non existent. Analytical databases are optimized for a low number of slow queries. The most important limitations of the analytical databases are:

  1. Deletes and updates are non-existent or slow
  2. Inserts are efficient for bulk inserts only
  3. No secondary indexes means that point selects (select by ID) tend to be very slow

This is all true for ClickHouse, however, we may be able to live with it for our task.

To simulate text messages I have used ~3 billion of reddit comments (10 years from 2007 to 2017), downloaded from pushshift.io . Vadim published a blog post about analyzing reddit comments with ClickHouse. In my case, I’m using this data as a simulation of text messages, and will show how we can use ClickHouse as a backend for an API.

Loading the JSON data to Clickhouse

I used the following table in Clickhouse to load all data:

Then I used the following command to load the JSON data (downloaded from pushshift.io) to ClickHouse:

The data on disk in ClickHouse is not significantly larger than compressed files, which is great:

We have ~4 billion rows:

The data is partitioned and sorted by created_utc so queries which include created_utc will be able to using partition pruning: therefore skip the not-needed partitions. However, let’s say our API needs to support the following features, which are not common for analytical databases:

  1. Selecting a single comment/message by ID
  2. Retrieving the last 10 or 100 of the messages/comments
  3. Updating a single message in the past (e.g. in the case of messages, we may need to update the final price; in the case of comments, we may need to upvote or downvote a comment)
  4. Deleting messages
  5. Text search

With the latest ClickHouse version, all of these features are available, but some of them may not perform fast enough.

Retrieving a single row in ClickHouse

Again, this is not a typical operation in any analytical database, those databases are simply not optimized for it. ClickHouse does not have secondary indexes, and we are using created_utc as a primary key (sort by). So, selecting a message by just ID will require a full table scan:

Only if we know the timestamp (created_utc)… Then it will be lighting fast: ClickHouse will use the primary key:

Actually, we can simulate an additional index set by creating a materialized view in ClickHouse:

Here I’m creating a materialized view and populating it initially from the main (rc) table. The view will be updated automatically when there are any inserts into table reddit.rc. The view is actually another MergeTree table sorted by id. Now we can use this query:

This is a single query which will join our materialized view to pass the created_utc (timestamp) to the original table. It is a little bit slower but still less than 100ms response time.

Using this trick (materialized views) we can potentially simulate other indexes.

Retrieving the last 10 messages

This is where ClickHouse is not very efficient. Let’s say we want to retrieve the last 10 comments:

In a conventional relational database (like MySQL) this can be done by reading a btree index sequentially from the end, as the index is sorted (like “tail” command on linux). In a partitioned massively parallel database system, the storage format and sorting algorithm may not be optimized for that operation as we are reading multiple partitions in parallel. Currently, an issue has been opened to make the “tailing” based on the primary key much faster: slow order by primary key with small limit on big data. As a temporary workaround we can do something like this:

It is still a five seconds query. Hopefully, this type of query will become faster in ClickHouse.

Updating / deleting data in ClickHouse

The latest ClickHouse version allows running update/delete in the form of “ALTER TABLE .. UPDATE / DELETE” (it is called mutations in ClickHouse terms). For example, we may want to upvote a specific comment.

“Mutation” queries will return immediately and will be executed asynchronously. We can see the progress by reading from the system.mutations table:

Now we can try deleting comments that have been marked for deletion (body showing “[deleted]”):

After a while, we can do the count again:

As we can see our “mutation” is done.

Text analysis

ClickHouse does not offer full text search, however we can use some text functions. In my previous blog post about ClickHouse I used it to find the most popular wikipedia page of the month. This time I’m trying to find the news keywords of the year using all reddit comments: basically I’m calculating the most frequently used new words for the specific year (algorithm based on an article about finding trending topics using Google Books n-grams data). To do that I’m using the ClickHouse function alphaTokens(body) which will split the “body” field into words. From there, I can count the words or use arrayJoin to create a list (similar to MySQL’s group_concat function). Here is the example:

First I created a table word_by_year_news:

This will store all frequent words (I’m filtering by subreddits; the examples are: “news, politics and worldnews” or “programming”) as well as its occurrence this year; actually I want to store “relative” occurrence which is called “ratio” above: for each word I divide its occurrence by the number of total words this year (this is needed as the number of comments grows significantly year by year).

Now we can actually calculate the words of the year:

And the results are (here I’m grouping words for each year):

For “programming” subreddit:

For news subreddit:

Conclusion

ClickHouse is a great massively parallel analytical system. It is extremely efficient and can potentially (with some hacks) be used as a main backend database powering a public API gateway serving both realtime and analytical queries. At the same time, it was not originally designed that way. Let me know in the comments if you are using ClickHouse for this or similar projects.


Photo by John Baker on Unsplash

 

 

Share this post

Comments (4)

  • Denis Golius Reply

    Alexander Rubin showed the worst things… So maybe you are not realy good in CH?

    January 15, 2019 at 3:15 am
  • Kevin English Reply

    GROUP BY is not OLAP. Please stop associating single-index column stores with OLAP. They are miles apart. If you can not analyze and pivot in N dimensions, it is not OLAP.

    February 25, 2019 at 3:49 pm
  • David Reply

    Additional index simulation is just what I was looking for. Thanks!

    May 25, 2019 at 1:33 pm
  • sourabh S Reply

    we run adservers which process almost a several hunderd million to close to a billion requests per day.
    we are using clickhouse for our analytical storage engine for over 1.5 yeras now.

    few months ago when updated/deletes came out for clickhouse we tried to do exactly what is mentioned above .i.e convert everything to clickhouse from mysql , including user,product table etc.

    we used clickhouse as our primary storage (replicated engines with kafka) in the development mode everything was running smoothly even the updates and deletes , so we were happy and pushed the code to production.. however this turned out to be a major blunder and almost ruined our production server.

    What happened was there were to many updates queries piling up and choking up the server ,clickhouse was not able to process them in time. as a result , updates were taking more then 30 minutes .

    we knew from he start that updates would be slow but we dint expected this. in short clickhouse must not be used for the table where updates are happening frequently.

    Merge engine is very unreliable about when would it merge the duplicate entries. usage of ‘final’ keyword query is very bad ideas once you table starts growing.also not to mention frequent data corruption/loss due to replication errors.

    July 6, 2019 at 9:41 pm

Leave a Reply