antirez weblog

I was not able to write this post yesterday as I was just too busy delivering a new feature, just before the feature freeze that will start 1th April. So as you can guess, even if this violates the chronological order, this is going to be the first entry of the weekly update.

Redis is now a Publish/Subscribe server

This feature is a user-requested one. Actually it is the mix of two different requests:

• Many users wanted a notification mechanism to trap changes in the key space. Like waiting for a given key to be changed, deleted, and so forth. We already had BLPOP (blocking list-pop operation) but this is really not good enough for this task, for a number of reasons: usually you want to notify multiple clients at the same time, and usually you don't want to queue notifications, but just, who is listening will get it. Users resorted to all kind of tricks to get this effect, including the use of MONITOR as a stream of changes! But this was not the right solution as you can guess.
• Everybody trying to do something like a chat or other many-to-many message passing things with Redis started using lists and ended with problems.

So I started designing some more general way that the requests I was receiving in order to model many message passing needs that are not well suited for BLPOP (that is instead very good for queues of messages, but not all problems can be modeled this way), and the design I ended up with was promptly recognized by a few users as Publish/Subscribe (I was not aware of it to be honest).

But wait, what is Redis? a NoSQL database (and what is this supposed to mean after all? I'm starting to dislike the term), a Cache, or a messaging server given that there are many installations of Resque around?

Redis is a event-driven substrate where it is simple to plug other ideas, and this ideas are often useful in the same context. This is how it is possible to write modular, scalable applications today: you need a very fast "global state" (the DB side), caching, ways to create queues of jobs as soon or later you need to process things in the background, and sometimes you need the ability to pass messages around in a pub/sub many-to-many scenario. And this things are converging, they are just different sides of the same things, at least when the DB part is designed as it is in Redis.

The whole Pub/Sub implementation in Redis is 150 lines of code, it's like if Redis was already a Pub/Sub server without the API to use it, so all I did was to add the glue code to expose some of the unexploited internals.

If you want to explore this new tool, there is a great deal of info! We have the official documentation, support in redis-rb (that were already merged by Ezra but not visible in Github for some reason?), and a great example written by Peter Noordhuis using Redis and EventMachine implementing a web chat.

New semantic: delete keys on empty values

This week I finally introduced a important semantical change in Redis: now all the operations modifying a value, resulting in an empty list/set/sorted-set/hash, will also delete the key.

So for instance if you have a key with a one-element list, and call LPOP against it, you get the element, and the key gets removed at all: since all our commands are designed to work the same against non existing keys and empty values, most of the code will just work the same way as it used to, but there are no longer memory pollution issues.

Less latency under high amount of expiring keys

The active expiring of keys is now done in a more incremental way (less keys, more often) in order to reduce some latency issue. In order to notice this change you need to have strict milliseconds requirements, but it is definitely an improvement anyway.

Also a new INFO field was added, called expired_keys, that is a (64 bit) counter of all the keys expired for a timeout. This way it is possible to monitor how a caching server is "working", for instance checking how many keys are freed every second because they expired.

CONFIG!

We have an interesting new command called CONFIG, still not documented, but the manual page is in my short term TODO list. A small redis-cli trace will show what you can do with it (that is, configuring a few Redis parameters while the server is running):

$ ./redis-cli config get "*"
1. dbfilename
2. dump.rdb
3. requirepass
4. (nil)
5. masterauth
6. (nil)
7. maxmemory
8. 0
$ ./redis-cli config set maxmemory 1000
OK
$ ./redis-cli set foo bar
(error) ERR command not allowed when used memory > 'maxmemory'
$ ./redis-cli config set maxmemory 1000000000
OK
$ ./redis-cli set foo bar
OK

I'll add a few more parameters before 2.0 release (for instance saving parameters). It is also possible to reset all the INFO statistics with CONFIG RESETSTAT.

HINCRBY!

Pieter Noordhuis implemented HINCRBY for hashes (already merged) and provided some enhancement for zipmaps (defragmentation code) that is yet to merge (but I've no doubt that it will get inside, given the quality of Pieter work).

Feature Freeze time is near

Confused by the speed of development? Now you'll have a long period where everything will be a lot quieter ;) That is: feature freeze. It's time to consolidate what we have, to fix the latest VM bugs we are aware of, to write tests for everything we have added in the latest weeks.

When I'll be confident enough that all the known bugs are gone and the code looks solid, I'll release RC-1, and the work will start again. But the next big thing post 2.0 is not really about redis-server, but more about redis-cluster. Once we'll have 2.0 stable, we have an interesting "data node", and it will be really time to provide scalability, auto sharding, and fault tolerance, on top of what we already have.

1.2.6 released

This week also saw the release of 1.2.6, a bugfix + improvements release of the Redis stable version, so I'll end this post with the changelog of what you can find in the old stable 1.2.x Redis.