Scripting branch released Monday, 02 May 11

Warning: the scripting API was modified in recent versions of 'scripting' and '2.2-scripting' branch. Now Lua can call Redis commands using Redis.call('get',...) instead of Redis('get',...). Also the EVALSHA command is now available in both branches.


As expected I did not resisted to the temptation of implementing a branch with Lua scripting support for Redis, and this weekend was the right moment to attack the problem, regardless of the fact that in Italy 1th of May is the "International Workers' Day" and everybody is supposed to don't work. But I actually had a lot of fun coding, and managed to attend a concert, to play at bowling, drink some good vodka, and watch the "Source Code" movie, so I'll call this a busy weekend :)

So after some Lua C API crash course, roughly 400 lines of code, and 8 hours of work in two days, this is the result in the scripting branch at github, but most of the code is self contained in the scripting.c file if you want to understand how the implementation works (hint: it is really simple).

IMPORTANT: the fact that now we have a scripting branch does not imply that we'll ever see this in a stable branch. I wrote an actual implementation so that we can collectively test how good is scripting for Redis. However I'll take the branch rebased against the unstable branch (thanks to the fact the implementation is almost completely self-contained in a single file, so that will be trivial) AND I must admit, now that I played with scripting and that I found a suitable API, I'm impressed by the potential.

First steps

Lua scripting is exposed to Redis as a single command called EVAL. 
EVAL <body> <num_keys_in_args> [<arg1> <arg2> ... <arg_N>]
The body argument is a Lua script. It is followed by a variable number of arguments, prefixed by the number of keys that are in those arguments. So for instance if I call eval somescript 3 a b c d e f the arguments a, b, c are considered keys, all the rest of the arguments are considered as normal arguments. Key arguments are received by the Lua script as elements inside the KEYS table, all the other arguments are stored into the ARGV table instead.

You may wonder why there is this distinction between keys and arguments. The reason is, if you respect this semantic, the Eval command will be for Redis not different from any other command. Redis only knows a few things about registered commands: the number of arguments, and what arguments are keys. For example knowing about keys makes Redis Cluster able to forward your requests to the right node. The EVAL command is designed in order to be not different. However this is not enforced. If you want to do strange stuff, you can access all the key space from Lua scripts: we are free to shoot in our own foots :) 
redis> eval "return {KEYS[1],KEYS[2],ARGV[1],ARGV[2]}" 2 key1 key2 arg1 arg2
1) "key1"
2) "key2"
3) "arg1"
4) "arg2"
The above example shows two important points: the first is that our arguments are correctly passed via KEYS and ARGS tables. The second is that if you return a Lua Array (that is actually a Lua table indexed by incrementing integers starting from 1) from a Lua script, it will be returned to the user as a multi bulk reply.

Note that we are sending the same script every time instead of storing a procedure. I discussed this in the previous blog post, but the point is that ending with different instances holding different versions of the scripts is really bad. With our current EVAL semantics every instance is the same, regardless of the fact it was just started or not, and regardless of its redis.conf file. However for the scripting engine to be fast we can't keep compiling the same script again and again, so internally Redis will compile the script only the first time it was seen. If the same body is seen again, the already compiled version is used. This makes EVAL a very fast command as we'll see later.

However if bandwidth will be an issue, I've an exit strategy that I'll discuss later in this post.

Returning Redis data types from Lua

We already saw how returning an array from the Lua script generates a multi bulk reply, but here is the full list of returned types: 
redis> eval "return 10" 0
(integer) 10
redis> eval "return 'foobar'" 0
"foobar"
redis> eval "return {1,2,'a','b'}" 0
1) (integer) 1
2) (integer) 2
3) "a"
4) "b"
redis> eval "return {err='Some Error'}" 0
(error) Some Error
redis> eval "return {ok='This is a status reply'}" 0
This is a status reply
As you can see you can return everything a Redis command can return, including errors and status replies. So:

Accessing Redis from Lua

Our interface to Redis is a single redis() function exposed to the Lua interpreter: 
OK
redis> eval "return redis('get',KEYS[1])" 1 x
"foo"
It is as simple as calling Redis with the command as first argument followed by all the other arguments. The arguments type can be either string or number. Numbers are automatically converted into strings, as Redis commands accept only strings arguments from clients even when the actual meaning of the argument is a number. Now the really interesting part is how the redis() function return values to Lua: using exactly the reverse of the conversion table above. That is, Redis integer replies are passed to Lua as integers, Bulk replies as Lua strings, and so forth. Everything maps to a Lua type and the other way around, so we can return the result of the redis() function as we did in the example above. Lua coders may complain that Lua idiomatic error handling is different, but the point here is that Redis errors are treated as a kind of value. For instance you can do things like: 
ret = redis(... some command ...)
... do more work ...
return ret
Without even caring about what ret was, if a multi bulk reply, an error, or whatever. Btw don't worry, nothing is written on the stone, this is an experimental branch and everything is fluid and can be changed later if needed.

An actual example

One of the reasons I'm positive about integrating scripting into Redis in the near future (but don't take this as a promise!) is that is almost our only salvation from making Redis bloated. For instance yesterday an user wrote in the Redis mailing list writing about how to conditionally decrement a counter, only if the current counter value is greater than zero.

Our current solution is MULTI/EXEC/WATCH, but guess what? It is slow since we are forced to move data from the Redis server to the Redis client, inspect the current state, and finally issue the commands to modify our dataset. Without to mention that like all the optimistic locking approaches does not work when there is too much contention. After all it is so simple to conditionally decrement server side, right? But everybody has a different problem. How much commands should we add? With scripting all this specific problems are solved in a general way without making the Redis server a mess with a big number of commands, and without trying to implement our "little language" that will later turn in an ill conceived real language.

So welcome to our first example, decrementing a value only if the value is greater than a given value we pass as argument.



The above program produces the following output as you could expect. 

ruby cond-decr.rb
3
2
1
0
0

Two more examples

Another common request in the mailing list is to provide "NX" or "EX" versions of Redis commands, that is, commands that are executed only if the target key does not exists or exists. The following example implements INCREX, that is, increment only if the counter already exists.



Executing it will output: 
nil
false
11


And finally a more complex example: List shuffling.



How fast it is?

Small Lua scripts are so fast that it is basically the same as implementing the command in C! For instance a small script like the conditional DECR example takes in my MBA 11" (that is slooow) 31 microseconds compared to 14 microseconds of the real SET command. This timings only consider the execution of the command, without all the client I/O, protocol parsing, dispatching, and so forth. Basically once you add all the overhead what you find is that from the outside you can't say which is faster, if SET or the conditional DECR lua script.

Atomicity

Lua scripts are executed like C implemented commands, in a completely atomic way. This means that you can do everything, but also that you should be aware that you are blocking the server if you execute too slow scripts.

Replication and AOF

Instead of replicating (or writing in the AOF) the executed commands, whole scripts are replicated. This means that if you want to use replication or AOF with scripting you should write scripts without side effects, not using time or any other external events in order to do their work.

In other words a script should always produce the same result if the initial dataset is the same.

It is very important to do it this way instead of replicating all the single commands generated by the script for a simple reason: with scripting you can do things 10 or even 100 times faster than sending commands. If we don't replicate scripts but single commands, the master -> slave link will get saturated in no time, OR, the slave instance will not be able to process things as fast as needed.

Bandwidth is an issue!

I already received many tweets about how bandwidth can become an issue since we have to send the same script again and again. But at the same time it is very important to avoid registering scripts for the reasons exposed, we really don't want to have to deal with versioning of scripts among different instances, or with source code of applications calling some non well specified script inside a Redis instance.

But can we have the best of both world? I think we actually can! If bandwidth will be an issue I'll implement the EVALSHA command. Basically the command will be exactly like EVAL, but instead of the script body it will take the SHA1 of the script body. If the script with such an hash was already seen, it is executed as usually, otherwise an error is returned.

Client libraries will abstract all this from users. The user will always pass the full script, but the client library will try to use the hash. If an error is raised by Redis the client will use EVAL in order to define the script the first time. I don't want to add this right now. Ideas gets better after some time, like red wine.

Have fun

So now I'm back to Redis Cluster for the next weeks, but please if you like Redis scripting play with it and have fun. We need some more experience with it in order to understand what to do with it!

I promise to take the branch rebased against the unstable branch and to fix big issues if you find any.

Feedbacks? You can comment this blog post on Hacker News.

65 comments
home