Short term Redis plans

Short term Redis plans Monday, 07 November 11

Users often ask me what is the Redis development roadmap, so it is probably time to write a blog post about our short/mid term plans, with the most important points.

There are two major features that we are pushing forward: scripting and cluster, so let's start from this two.

Scripting

Scripting implements Lua scripting support for Redis. We already have detailed documentation for this feature that describes what is already implemented and what will likely go inside the first stable release of Redis featuring scripting. The only part you should consider outdated both in the doc and the implementation is how scripts running for too much time are handled (this topic was extensively covered in the Redis google group).

Redis scripting will appear in Redis 2.6, and I'm trying hard to ship Redis 2.6 RC1 for the end of this year (2011). There are many features that are planned for 2.6 as well, I'm not sure I'll be able to address everything but at some point I think I'll try to do a time-driven release for 2.6 and just put inside scripting and everything else that is already implemented/stable.

Redis Cluster

Redis cluster is definitely the next big thing, and you can read our Redis Cluster draft specification to get an idea about what it can do and what not. But in short Redis Cluster is a distributed implementation of a subset of Redis standalone. Not all commands will be supported, especially we don't support things like multi-key operations. In general we are just implementing the subset of Redis that we are sure can be made working in a solid way in a cluster setup, with predictable behaviors.

Redis cluster will stress consistency in favor of ability to resist to netsplits and failures in general. Basically it will tollerate well a few instances going down, but will not survive to big netsplits like other eventually consistent systems are able to do.

Redis cluster is as important as scripting, but will be delayed to Redis 3.0 since scripting it is much simpler to implement and in our opinion of almost equal importance for most users (if not more) so we prioritized scripting first.

The current status of Redis cluster is that you can play with it already but not everything in the specification is implemented. It will take a few more months in order to reach beta, and then we'll work on the details in order to ship something solid. We'll try hard to resist to ship a system that is not mature just to say we are already cluster-ready. Redis is fortunately very useful already so we want to make sure to ship the cluster version only when it will likely resolve problems instead of creating new ones.

The good news is that the Redis Cluster design is particularly simple in almost all the aspects, this helps our hope to ship a good system in a reasonable amount of time.

Replication improvements

As part of the work we are doing for Redis Cluster we'll need to improve replication. This part of Redis almost always received improvements in the course of releases, but with Redis Cluster we need an even better one. For instance it is planned to avoid a full resync every time the link goes down if the downtime was reasonable and the differences can be accumulated. In short when the slave disconnects the master does not kill the client representation of the slave, but continues sending data (that gets bufferized). When the slave reconnects we recognized it form a new per-instance ID that always changes after a restart (or after a SLAVEOF NO ONE command), and perform the incremental resync.

This changes will either be shipped with Redis 3.0 or with a future version (the current replication is not optimal but probably already good enough for the first Redis Cluster release, so it is not clear if we'll be able to fix it before or after the first release).

Persistence improvements

Currently we have two persistence modes: append only files and RDB persistence. Both have different tradeoffs. It is not clear what we'll do about it but it is possible that we will either unify the two models and/or improve AOF a lot so that it does no longer need the online rewrite process in most use cases (but the log can be rewritten by an external process or simply a Redis thread).

Everything is very hypothetical in this area for now, but there are al lot of ideas that we accumulated in the latest years that are wroth to experiment with for sure.

We want also work both in the communication (most users don't understand that Redis with both AOF and RDB enabled is very durable already, and this is the setup we suggest) and the implementation to make sure that Redis AOF can be a very durable solution, as durable as the best SQL databases out there.

This is definitely a post-cluster stuff.

More introspection

There is a plan to use Pub/Sub in order to communicate events happening inside Redis, like a key that expired, clients connecting / disconnecting, operations performed against keys. We'll probably allow the user to script this feature with Lua so that you can, for instance, push all the keys expired inside a list as well, or other things that can't be reliably done with clients and Pub/Sub since the client is not guaranteed to get all the messages (it can get disconnected for some reason).

High resolution expires

I'm working at it already, we'll have high precision expires in Redis 2.6. So you can set an expire just for a few milliseconds for a given key. The current resolution is 1 second that is ok for most applications but not for all.

Performances improvements when reading/writing big objects

If you check the 'slowset' branch there is work in this direction already. As part of this work I'm creating a speed regression test, since we really lack it. Note: with big objects I mean sets/gets in the range of 100k or 1 MB per element. Redis performs very well already with reads/writes of a few Kb values.

Many other smaller things

See the list of issues filtered by "new feature" tag on github to get an idea about the smallest things that are going to be implemented.

I hope this helped please ask me questions in the blog comments if you want more details. I'll reply tomorrow morning likely since here is already late ;)

home