KeyDB: the multithreaded fork of Redis

At Hosted Power, we want to distinct ourselves when it comes to performance. We thrive to be the fastest and most performance one on the market.

As a consequence, innovation is embedded as a necessity in our organization to keep improving our processes and tools that create these performant environments for your website(s).

This is where Redis comes to mind; it is an in-memory datastore used for caching within our environment that allows us to quickly access and retrieve data without the need to access the disk, ensuring minimal response times.

For a long time, the community has been pushing Redis to make their system multithreaded but for reasons such as ease-of-programming and cost-effective deployment, Redis decided to stay with their single threaded architecture. 

Eventually multithreaded alternatives to the caching system started to arise that would trigger the professional environments to also research and test these alternatives.

As we noticed benchmark results from alternatives showing 5 to 25 times the performance Redis should achieve, we started testing ourselves. There was one alternative that showed to be the most stable and most compatible one for a professional environment, called KeyDB.

KeyDB is a multithreaded fork of Redis, meaning it is fully compatible with the Redis protocol, changes and modules while promising to stay aligned with the upstream changes Redis will be doing in the future.

KeyDB, besides being multhithreaded, is also giving a higher performance on less hardware, reduction of the cluster size due to active replication (meaning 2 master nodes who accept both read and write operations), the ability of creating your own command with ModJS as well as promising interesting upcoming features sucs as FLASH storage, json support and multi-tenant support. This all sound promising, yet we still need confirmation by our own test results that back up these expectations. 

To generate the best possible test results, we used a benchmark tool utilized and developed by RedisLabs called memtier_benchmark. It allows you to generate various traffic patterns against Redis instances. This allowed us to recreate different kind of scenarios varying in the number of clients and the amount of requests generated by these clients in different setups ranging from a 2XCPU-4GB RAM setup to a 8xCPU-32GB RAM.

The results did not dissappoint. We started testing withing an environment with only 2 cores and 4GB RAM. Here we started testing by using a 100 clients that each generated a 100, 1000 and 10 000 requests. 

As the output shows, Redis outperforms KeyDB by a lot while maintaining a lower average latency. 

These results were to be expected as KeyDB should improve drastically while more threads are being assigned to it. The testing with 4 cores already support these hypotheses:

Starting at 4 cores, KeyDB begins to outperform Redis in all metrices, executes 66% more operations per seconds and has a higher throughput of data per second than Redis is capable. As to be expected, upping the threads to 6 showed even greater results.

KeyDB doubles the value Redis can produce in throughput of data and operations per second. A clear pattern is shown here, to confirm this we have upped the threads once more to the amount of 8.

Conclusions can be drawn here, as it starts getting beneficial from a setup with at least 4 cores while the greatest turnover is still generated with a setup of 6 threads. Latency is not a metric shown in the graphs about, but where outlined in the test results that showed KeyDB still maintaining a lower average latency.

The future for this alternative is promising and since it is already outperforming an optimized setup of Redis, looking for ways to embed this in our deployment will be an upcoming feature that will make websites on our own TurboStack® even more performant than before.