An update on Redis and Lua Friday, 13 May 11

I released the scripting branch just two weeks ago, but it is already a super hot topic: people are hacking with it, using it in production, dropping forked versions of Redis with C implemented specific commands in favor of simple Lua scripts. I was very in doubt about the opportunity of adding scripting to Redis, it was clear that the benefit was huge, but was the downside huge as well?

Apparently the way scripting is added is the key factor. I think we got the API right, this was already a good start, but there is more than that. A big question is, should you be able to interact with the external world from Redis scripts, like writing files, require libs, and so forth? I think this is not the case. Scripting should just allow users to do client side using EVAL what usually was previously possible only creating specialized commands in C. Also in the previous blog post I explained how the lack of side effects is crucial for the correctness of AOF and Replication in a scripted environment.

So the scripting feature will be limited to what you can do with vanilla Lua, and with the redis() command that lets you interact with Redis of course, and a few standard additions that will be registered by Redis in the Lua interpreter (so you know every Redis instance with scripting support contains this features by default), basically:

Why this additions? Because bit operations are more or less part of every language. In Lua they are into a lib as Lua is minimal by design. But it is very useful. Think for instance about implementing a bloom filter in Redis.

As for Json, a lot of people store Json objects inside Redis. It is handy if it is possible to manipulate this objects from Lua. This means you can ask with a one line script to retrieve or modify just a field of your Json object.

Another commonly useful thing is to take digests. Want to make sure a list is a give version? Take the Sha1 of the elements from Lua and return just that. In general a good hash function is a Swiss Army Knife tool, and it is already part of Redis, so we'll provide a sha1() function that given a string returns the hexadecimal SHA1.

There is always time to add more, but this should be enough for many needs.

News on Lua <-> Redis type conversion

I just pushed a new commit in the scripting branch on github that changes how types are converted from/to Redis protocol types.

A key feature is the ability to return from Lua commands everything is valid in the Redis protocol, so that your Lua scripts can return everything a C coded command is able to return. We where almost there with the old scripting branch, but what was missing was the ability to return tables of tables. Now this is implemented as well.

However there is another issue. What if you want to return a multi bulk reply form Lua, with an element that is a null bulk reply? As we converted the Lua nil type into the Redis null bulk reply, something like that is needed: 
EVAL "return {1,2,3,nil,5,6}" 0
But guess what? This does not work. In Lua there is a single aggregate data type that is used also to represent arrays, but there is no way when you ask for element at position '4' to know if it is nil since the element is not present as the Array only has three elements, or since the element is actually set to the nil value.

My solution was to convert null (multi) bulk types into false, and the other way around, false returned from Ruby is turned as a null bulk reply.

This was the only sane option as it allows to do natural things like: 
if not redis('get','foo') ...


So this is the new full conversion table used by the latest version of the scripting branch:

Redis protocol to Lua type

Lua type to Redis protocol

It's not a one to one map as Redis and Lua type semantic is a bit different, but I think it is good enough and the best we can get.

Needless to say, with after the response from the community and playing with scripting more in this days, I'm pretty sure it will make it into a stable release as soon as possible.

Enjoy scripting!

home