Scripting branch released
Monday, 02 May 11
Warning: the scripting API was modified in recent versions of 'scripting' and '2.2-scripting' branch. Now Lua can call Redis commands using Redis.call('get',...) instead of Redis('get',...).
Also the EVALSHA command is now available in both branches.
As expected I did not resisted to the temptation of implementing a branch with Lua scripting support for Redis, and this weekend was the right moment to attack the problem, regardless of the fact that in Italy 1th of May is the "International Workers' Day" and everybody is supposed to don't work. But I actually had a lot of fun coding, and managed to attend a concert, to play at bowling, drink some good vodka, and watch the "Source Code" movie, so I'll call this a busy weekend :)
So after some Lua C API crash course, roughly 400 lines of code, and 8 hours of work in two days, this is the result in the scripting branch at github, but most of the code is self contained in the scripting.c file if you want to understand how the implementation works (hint: it is really simple).
IMPORTANT: the fact that now we have a scripting branch does not imply that we'll ever see this in a stable branch. I wrote an actual implementation so that we can collectively test how good is scripting for Redis. However I'll take the branch rebased against the unstable branch (thanks to the fact the implementation is almost completely self-contained in a single file, so that will be trivial) AND I must admit, now that I played with scripting and that I found a suitable API, I'm impressed by the potential.
First steps
Lua scripting is exposed to Redis as a single command called EVAL.EVAL <body> <num_keys_in_args> [<arg1> <arg2> ... <arg_N>]The body argument is a Lua script. It is followed by a variable number of arguments, prefixed by the number of keys that are in those arguments. So for instance if I call eval somescript 3 a b c d e f the arguments a, b, c are considered keys, all the rest of the arguments are considered as normal arguments. Key arguments are received by the Lua script as elements inside the KEYS table, all the other arguments are stored into the ARGV table instead.
You may wonder why there is this distinction between keys and arguments. The reason is, if you respect this semantic, the Eval command will be for Redis not different from any other command. Redis only knows a few things about registered commands: the number of arguments, and what arguments are keys. For example knowing about keys makes Redis Cluster able to forward your requests to the right node. The EVAL command is designed in order to be not different. However this is not enforced. If you want to do strange stuff, you can access all the key space from Lua scripts: we are free to shoot in our own foots :)
redis> eval "return {KEYS[1],KEYS[2],ARGV[1],ARGV[2]}" 2 key1 key2 arg1 arg2
1) "key1"
2) "key2"
3) "arg1"
4) "arg2"
The above example shows two important points: the first is that our arguments are correctly passed via KEYS and ARGS tables. The second is that if you return a Lua Array (that is actually a Lua table indexed by incrementing integers starting from 1) from a Lua script, it will be returned to the user as a multi bulk reply.
Note that we are sending the same script every time instead of storing a procedure. I discussed this in the previous blog post, but the point is that ending with different instances holding different versions of the scripts is really bad. With our current EVAL semantics every instance is the same, regardless of the fact it was just started or not, and regardless of its redis.conf file. However for the scripting engine to be fast we can't keep compiling the same script again and again, so internally Redis will compile the script only the first time it was seen. If the same body is seen again, the already compiled version is used. This makes EVAL a very fast command as we'll see later.
However if bandwidth will be an issue, I've an exit strategy that I'll discuss later in this post.
Returning Redis data types from Lua
We already saw how returning an array from the Lua script generates a multi bulk reply, but here is the full list of returned types:
redis> eval "return 10" 0
(integer) 10
redis> eval "return 'foobar'" 0
"foobar"
redis> eval "return {1,2,'a','b'}" 0
1) (integer) 1
2) (integer) 2
3) "a"
4) "b"
redis> eval "return {err='Some Error'}" 0
(error) Some Error
redis> eval "return {ok='This is a status reply'}" 0
This is a status reply
As you can see you can return everything a Redis command can return, including errors and status replies. So:
- A Lua number is returned as an Redis integer reply.
- A Lua string is returned as a Redis bulk reply.
- A Lua array is returned as a Redis multi bulk reply.
- A Lua table with an 'err' field is returned as a Redis error reply.
- A Lua table with an 'ok' field is returned as a Redis status code reply.
Accessing Redis from Lua
Our interface to Redis is a single redis() function exposed to the Lua interpreter:
OK
redis> eval "return redis('get',KEYS[1])" 1 x
"foo"
It is as simple as calling Redis with the command as first argument followed by all the other arguments.
The arguments type can be either string or number. Numbers are automatically converted into strings, as Redis commands accept only strings arguments from clients even when the actual meaning of the argument is a number.
Now the really interesting part is how the redis() function return values to Lua: using exactly the reverse of the conversion table above. That is, Redis integer replies are passed to Lua as integers, Bulk replies as Lua strings, and so forth. Everything maps to a Lua type and the other way around, so we can return the result of the redis() function as we did in the example above.
Lua coders may complain that Lua idiomatic error handling is different, but the point here is that Redis errors are treated as a kind of value. For instance you can do things like:
ret = redis(... some command ...) ... do more work ... return retWithout even caring about what ret was, if a multi bulk reply, an error, or whatever. Btw don't worry, nothing is written on the stone, this is an experimental branch and everything is fluid and can be changed later if needed.
An actual example
One of the reasons I'm positive about integrating scripting into Redis in the near future (but don't take this as a promise!) is that is almost our only salvation from making Redis bloated. For instance yesterday an user wrote in the Redis mailing list writing about how to conditionally decrement a counter, only if the current counter value is greater than zero.Our current solution is MULTI/EXEC/WATCH, but guess what? It is slow since we are forced to move data from the Redis server to the Redis client, inspect the current state, and finally issue the commands to modify our dataset. Without to mention that like all the optimistic locking approaches does not work when there is too much contention. After all it is so simple to conditionally decrement server side, right? But everybody has a different problem. How much commands should we add? With scripting all this specific problems are solved in a general way without making the Redis server a mess with a big number of commands, and without trying to implement our "little language" that will later turn in an ill conceived real language.
So welcome to our first example, decrementing a value only if the value is greater than a given value we pass as argument.
The above program produces the following output as you could expect.
ruby cond-decr.rb 3 2 1 0 0
Two more examples
Another common request in the mailing list is to provide "NX" or "EX" versions of Redis commands, that is, commands that are executed only if the target key does not exists or exists. The following example implements INCREX, that is, increment only if the counter already exists.Executing it will output:
nil false 11
And finally a more complex example: List shuffling.
How fast it is?
Small Lua scripts are so fast that it is basically the same as implementing the command in C! For instance a small script like the conditional DECR example takes in my MBA 11" (that is slooow) 31 microseconds compared to 14 microseconds of the real SET command. This timings only consider the execution of the command, without all the client I/O, protocol parsing, dispatching, and so forth. Basically once you add all the overhead what you find is that from the outside you can't say which is faster, if SET or the conditional DECR lua script.Atomicity
Lua scripts are executed like C implemented commands, in a completely atomic way. This means that you can do everything, but also that you should be aware that you are blocking the server if you execute too slow scripts.Replication and AOF
Instead of replicating (or writing in the AOF) the executed commands, whole scripts are replicated. This means that if you want to use replication or AOF with scripting you should write scripts without side effects, not using time or any other external events in order to do their work.In other words a script should always produce the same result if the initial dataset is the same.
It is very important to do it this way instead of replicating all the single commands generated by the script for a simple reason: with scripting you can do things 10 or even 100 times faster than sending commands. If we don't replicate scripts but single commands, the master -> slave link will get saturated in no time, OR, the slave instance will not be able to process things as fast as needed.
Bandwidth is an issue!
I already received many tweets about how bandwidth can become an issue since we have to send the same script again and again. But at the same time it is very important to avoid registering scripts for the reasons exposed, we really don't want to have to deal with versioning of scripts among different instances, or with source code of applications calling some non well specified script inside a Redis instance.But can we have the best of both world? I think we actually can! If bandwidth will be an issue I'll implement the EVALSHA command. Basically the command will be exactly like EVAL, but instead of the script body it will take the SHA1 of the script body. If the script with such an hash was already seen, it is executed as usually, otherwise an error is returned.
Client libraries will abstract all this from users. The user will always pass the full script, but the client library will try to use the hash. If an error is raised by Redis the client will use EVAL in order to define the script the first time. I don't want to add this right now. Ideas gets better after some time, like red wine.
Have fun
So now I'm back to Redis Cluster for the next weeks, but please if you like Redis scripting play with it and have fun. We need some more experience with it in order to understand what to do with it!I promise to take the branch rebased against the unstable branch and to fix big issues if you find any.
Feedbacks? You can comment this blog post on Hacker News.
Do you like this article?
Subscribe to the RSS feed of this blog or use the newsletter service in order to receive a notification every time there is something of new to read here.
Note: you'll not see this box again if you are a usual reader.
Subscribe to the RSS feed of this blog or use the newsletter service in order to receive a notification every time there is something of new to read here.
Note: you'll not see this box again if you are a usual reader.
Comments
1
02 May 11, 09:13:28
This is good stuff, but IMHO scripting Redis via JavaScript instead of Lua will be much more helpful to Redis users as most of us know JavaScript well already.
02 May 11, 09:21:25
Baishampayan Ghose, you're going to have to back that comment up with some evidence or it can be asserted that by "most of us" you really mean "I".
02 May 11, 09:41:47
To solve the bandwidth issue maybe just have another command that evaluates precompiled lua bytecode, might save a few bytes but probably not much. I see this more usefull for admin or maintenance tasks as it has nice and easy access to the internals without any c hacking.
02 May 11, 10:03:24
Lua is a mostly algol-like programming language, and you are supposed to write short scripts to implement new commands.
Total time require to master it enough? 5 minutes.
Lua VS Javascript debate is completely useless ;)
Total time require to master it enough? 5 minutes.
Lua VS Javascript debate is completely useless ;)
02 May 11, 10:05:24
its really useful for SNS site,
cause they need lots of different data for one page
and to avoid waste of memory , these data should be atomic , so redis with lua binding solve the problem with
high performance
cause they need lots of different data for one page
and to avoid waste of memory , these data should be atomic , so redis with lua binding solve the problem with
high performance
02 May 11, 10:08:01
Would it not be possible to offer an execkey which will execute the LUA script stored at a key - that way we can deal with versioning if we require (do a set during execution setup and a del during teardown if necessary) and reduce complexity in client libraries?
02 May 11, 10:09:05
@Phil: no, it is very clearly specified in the article (and in the previous one) why this is not a good idea in my opinion. However you can do it already, and easily, using Lua's loadstring() function if you really want, but it is discouraged.
02 May 11, 10:12:33
Whoa ! I just took a look at the Lua function reference, and the skies opened.
I see a redis Bloom filter in my immediate future..
and server side proximity calcs, and.....
I see a redis Bloom filter in my immediate future..
and server side proximity calcs, and.....
02 May 11, 10:13:45
Michael: I don't think I need to qualify my statement that JavaScript is more popular than Lua -- it's quite obvious. By us, I broadly meant (web?) programmers, people like us have better exposure to JS than Lua.
Antirez: Sure, Lua is not very different from C but I hope you don't think all Redis users are as good at C as you are ;)
Antirez: Sure, Lua is not very different from C but I hope you don't think all Redis users are as good at C as you are ;)
02 May 11, 10:15:09
When I read the previous post I was rather skeptical about scripting redis but you've completely won me over with this one. It'll be a good excuse to play with redis and also learn lua.
Also, the EVALSHA command is a terrifically elegant way to solve the bandwidth problem. Well done.
Also, the EVALSHA command is a terrifically elegant way to solve the bandwidth problem. Well done.
02 May 11, 10:26:11
Baishampayan, while I agree with you that the average redis user have more experience with js than with Lua, but to compare Lua to C is really seems like you didn't know much about it.
Tyrael
Tyrael
02 May 11, 10:40:31
@Baishampayan http://www.tiobe.com/index.php/content/paperinfo/t...
JS: 11: Lua: 12 Doesn't look like much of a difference to me.
@Theo That "tables start at 1" is one of the main things that confuse new Lua users, along with "variables are global by default". However, you get used to it quickly, and then it looks more logical imo.
JS: 11: Lua: 12 Doesn't look like much of a difference to me.
@Theo That "tables start at 1" is one of the main things that confuse new Lua users, along with "variables are global by default". However, you get used to it quickly, and then it looks more logical imo.
02 May 11, 10:52:21
I am sorry to say, but no amount of Lua scripting will prevent Redis from being on the deathmarch it is on. No amount of scripting can fix the poorly implemented bloated code base which is Redis.
02 May 11, 10:55:37
This all sounds great!
The only comment I'd have is that it'd probably be easier if all args would be returned in one ARGV table, otherwise if you have 4 keys and then 5 args, you always need to think to offset -4 when accessing ARGV, it's not a big deal, but imo it should just be one list of arguments.
The only comment I'd have is that it'd probably be easier if all args would be returned in one ARGV table, otherwise if you have 4 keys and then 5 args, you always need to think to offset -4 when accessing ARGV, it's not a big deal, but imo it should just be one list of arguments.
02 May 11, 12:21:39
"bloated code base which is Redis"
Wait, if redis is bloated, what are some exmples of projects that you would consider "not bloated" Please, I can't think of projects more slim than redis.
Wait, if redis is bloated, what are some exmples of projects that you would consider "not bloated" Please, I can't think of projects more slim than redis.
02 May 11, 13:00:35
As a guy who's used redis once, and Lua JIT a couple of times I have to admit I'm kinda excited by this. I'm addicted to documentation, and there's plenty of great docs on redis and lua :).
Need a good starter project to make use of this peanut butter and jelly combo.
Need a good starter project to make use of this peanut butter and jelly combo.
02 May 11, 13:25:31
Great!
I think it will pay a lot to keep the implementation this simple. Aside from the 1 indexed tables, I think Lua is really the best language for the purpose, as its a really simple language to learn, REALLY FAST, and once you get into it's metatable & functionnal style programming you can really see the power behind its apparent simplicity... It's really a matter of taste and I know we should not have debate over Lua vs JS, still my taste is that Lua code is more readable than Javascript which is mostly ugly.
One question remains for me... since Lua has no batteries included, people will probably like to use helper libraries in their command scripts, how about including part of stdlib http://luaforge.net/projects/stdlib/ into redis scripting by default? Like the base, tables, list, string and set libs, also having a Lua JSON serializer/deserializer would be nice, since many are serializing data into redis.
Thanks Antirez! Great Work!
I think it will pay a lot to keep the implementation this simple. Aside from the 1 indexed tables, I think Lua is really the best language for the purpose, as its a really simple language to learn, REALLY FAST, and once you get into it's metatable & functionnal style programming you can really see the power behind its apparent simplicity... It's really a matter of taste and I know we should not have debate over Lua vs JS, still my taste is that Lua code is more readable than Javascript which is mostly ugly.
One question remains for me... since Lua has no batteries included, people will probably like to use helper libraries in their command scripts, how about including part of stdlib http://luaforge.net/projects/stdlib/ into redis scripting by default? Like the base, tables, list, string and set libs, also having a Lua JSON serializer/deserializer would be nice, since many are serializing data into redis.
Thanks Antirez! Great Work!
02 May 11, 13:27:19
@loopole thanks, this is a huge amount of good suggestions! Putting part of the std lib sounds like a great idea indeed. Also the JSON idea is awesome :) Thank you.
02 May 11, 13:31:43
@Baishampayan Ghose: JavaScript implementations are heavier, slower, and harder to embed into C code than Lua, and less portable to boot - Lua can run on anything with a C compiler. Also, JavaScript is ugly.
@loopole: Probably the full "stdlib" would be overkill. Some utility functions would definitely be nice, though. I think a higher priority than "battery" libraries right now is sandboxing - as is, you can EVAL "os.exit()" and shut down the Redis process, and scripts can mess around with the environments of other scripts.
@loopole: Probably the full "stdlib" would be overkill. Some utility functions would definitely be nice, though. I think a higher priority than "battery" libraries right now is sandboxing - as is, you can EVAL "os.exit()" and shut down the Redis process, and scripts can mess around with the environments of other scripts.
02 May 11, 14:38:55
Is there a way to address fields within an hash? I need to multiply one hash field by another. Are only regular keys supported?
02 May 11, 14:44:06
I ask because
eval "return redis('hgetall',KEYS[1])" 1 hashname
crashed my server
eval "return redis('hgetall',KEYS[1])" 1 hashname
crashed my server
02 May 11, 15:07:59
i must be doing something wrong..
redis> eval "return {(redis('hget',KEYS[1],KEYS[2])),(redis('hget',KEYS[1],KEYS[3]))}" 3 hq val1 val2
1) "1.48559"
2) "120.669"
redis> eval "return {(redis('hget',KEYS[1],KEYS[2]))*(redis('hget',KEYS[1],KEYS[3]))}" 3 hq val1 val2
1) (integer) 179
redis>
redis> eval "return {(redis('hget',KEYS[1],KEYS[2])),(redis('hget',KEYS[1],KEYS[3]))}" 3 hq val1 val2
1) "1.48559"
2) "120.669"
redis> eval "return {(redis('hget',KEYS[1],KEYS[2]))*(redis('hget',KEYS[1],KEYS[3]))}" 3 hq val1 val2
1) (integer) 179
redis>
02 May 11, 15:17:30
redis> eval "return 3.1415" 0
(integer) 3
is there any way to use real numbers?
(integer) 3
is there any way to use real numbers?
02 May 11, 15:23:30
@Alessandro 1: Using HGET.
@Alessandro 2: This is still very unstable.
@Alessandro 4: The Redis protocol does not support returning doubles directly to clients, only integers. In Lua, the two share a single type - number - so any returned numbers are converted to integers, which drops the fractional parts.
To return a double, you will need to convert it to a string with the tostring function, and then convert it back to a number in the client code.
@Alessandro 2: This is still very unstable.
@Alessandro 4: The Redis protocol does not support returning doubles directly to clients, only integers. In Lua, the two share a single type - number - so any returned numbers are converted to integers, which drops the fractional parts.
To return a double, you will need to convert it to a string with the tostring function, and then convert it back to a number in the client code.
02 May 11, 15:26:36
in my opinion it' s a very good decision to implement scripting using lua, because it's a very powerful language, with very small footprint. it's very easy to learn, too. wow, this all sounds awesome, can't wait to play with it ...
02 May 11, 17:08:22
@Alessandro, @Matthew: now the bug is fixed, thank you. It is now possible to use HGETALL and any other command.
02 May 11, 17:29:54
great implementation Antirez.
I think you nailed ALL of the difficult parts in defining an intuitive/useful/fast API between Lua and Redis, hats off to you :)
My prediction, people start blogging real quickly on very interesting things they have done w/ lua-enabled-redis.
I think you nailed ALL of the difficult parts in defining an intuitive/useful/fast API between Lua and Redis, hats off to you :)
My prediction, people start blogging real quickly on very interesting things they have done w/ lua-enabled-redis.
02 May 11, 17:31:50
@jaksprats thank you! I really hope your prediction will turn true, it will be very interesting and instructive to see how people can use this to solve real problems.
02 May 11, 19:03:26
This looks really cool! I do have a question though. You show how you can use an array to return multiple values, but is there any reason you're not taking advantage of Lua's in-built support for multiple return values? In other words, is there any reason "return {1, 2, 3}" is better than "return 1, 2, 3"?
02 May 11, 19:08:15
Also, if you were to store a script in a Redis key, I believe you could use loadstring() to eval the script. Using that technique you could run a script at a given key using a simple line:
return loadstring(redis("get", KEYS[1]))()
EDIT: After looking at the code, I guess you're actually wrapping the script in a function and executing it. I'm curious why you're doing that. Couldn't you just set the ARGS/KEYS/redis/etc variables into _G and run the script as-is?
return loadstring(redis("get", KEYS[1]))()
EDIT: After looking at the code, I guess you're actually wrapping the script in a function and executing it. I'm curious why you're doing that. Couldn't you just set the ARGS/KEYS/redis/etc variables into _G and run the script as-is?
02 May 11, 19:19:13
@Jonathan: hello. about your first question, we need to provide a coherent Redis -> Lua, Lua -> Redis conversion, so the same data type Lua gets when calling a Redis command returning a multi bulk should be used when returning a value to Redis. The table is the type that semantically resembles more closely the bulk reply.
About loadstring() we are aware this can be done and will not deny it, but it is discouraged for the reasons explained in this blog post and in the previous one.
Thanks for your comment!
About loadstring() we are aware this can be done and will not deny it, but it is discouraged for the reasons explained in this blog post and in the previous one.
Thanks for your comment!
02 May 11, 19:51:41
@antirez: Thanks for your response! I also just discovered another (probably easier) way to store scripts:
EVAL "myscript = function() return 42 end" 0
Then later on:
EVAL "return myscript()" 0
EVAL "myscript = function() return 42 end" 0
Then later on:
EVAL "return myscript()" 0
02 May 11, 20:01:33
@Jonathan: indeed, the same Lua interpreter is used again and again, but warning we plain a "SCRIPT FLUSH" command or alike that flushes and creates the Lua interpreter for scratch :) This will mostly be used when the Lua interpreter gets polluted on in cloud environments when a spare Redis instance needs to get completely wiped.
03 May 11, 04:36:53
The EVALSHA approach is elegant indeed. I came up with something similar for a communication protocol for talking to clusters of servers - the idea is that the cluster is identified by a public key and a list of IP addresses / hostnames of the members. A client uses that identifier to try cluster members until it gets one that works. It then embeds a hash of the list of members in its request, which is encrypted with the cluster's public key so that only actual cluster members can reply correctly. If the cluster has changed (nodes added/removed/moved), then the hash of the list of members will have changed, so the server includes an up to date list in its (signed) response.
Voila - clients can follow a moving cluster (unless it moves TOO fast!) at the cost of only including an extra hash in requests, and then having a new list of members in responses only when it's needed.
tl;dr: HASHES ARE MAGIC AND SOLVE DIFFICULT PROBLEMS IN SURPRISING WAYS.
Voila - clients can follow a moving cluster (unless it moves TOO fast!) at the cost of only including an extra hash in requests, and then having a new list of members in responses only when it's needed.
tl;dr: HASHES ARE MAGIC AND SOLVE DIFFICULT PROBLEMS IN SURPRISING WAYS.
03 May 11, 05:53:24
if you plan on implementing a script flush please do add the possibility of having a lua script autoloaded when it happens -- I too am going to be using lua functions
03 May 11, 06:16:04
also, any idea why i cant seem to access to os.currentdir()? it is in the lua 5.1 spec..
redis> eval "return {ok=os.date()}" 0
Tue May 3 12:14:44 2011
redis> eval "return {ok=_VERSION}" 0
Lua 5.1
redis> eval "return {ok=os.currentdir()}" 0
(error) ERR Error running script (call to f_3cba57088cac1f60388c6085385a464eef74f42c): [string "func definition"]:2: attempt to call field 'currentdir' (a nil value)
redis> eval "return {ok=os.date()}" 0
Tue May 3 12:14:44 2011
redis> eval "return {ok=_VERSION}" 0
Lua 5.1
redis> eval "return {ok=os.currentdir()}" 0
(error) ERR Error running script (call to f_3cba57088cac1f60388c6085385a464eef74f42c): [string "func definition"]:2: attempt to call field 'currentdir' (a nil value)
03 May 11, 07:39:40
I am loving how redis has solved so many of our previous problems and the enhancements with scripting is going to clean up some of our php classes enormously.
03 May 11, 07:42:59
@Alessandro: Redis binds with Lua with "ansi" target. So POSIX stuff are not supported. In general we'll try to sandbox it even more probably...
@Lloyd: thanks, cool to know!
@Lloyd: thanks, cool to know!
03 May 11, 08:09:44
Cool.. I like the idea of sending the script in the command and Lua seems easy enough to understand. If this were to make it into stable I'm sure I would use it at some point or another.
03 May 11, 08:13:09
i suppose loadfile() is out of the questionthen .. editing a 1K function on the cli is not much fun and none of my clients are written in ruby unfortunately
03 May 11, 08:31:46
@Alessandro: if you are talking about scripts development:
Edit your script with your preferred editor into /tmp/myscript.lua
To execute it use:
redis-cli eval "$(cat /tmp/myscript.lua)" 0
Salvatore
Edit your script with your preferred editor into /tmp/myscript.lua
To execute it use:
redis-cli eval "$(cat /tmp/myscript.lua)" 0
Salvatore
03 May 11, 14:10:35
03 May 11, 17:19:34
@antirez I have a feeling that redis would be the perfect sandbox for a lisp based dialect. So I am wondering why you took the decision to implement LUA for instance instead of Lisp or any other very simple langage from the syntax perspective, especially since you´ve been playing with scheme as you stated in your previous post.
This is just some idea i am throwing, I am in my 30s so i never used LISP in real life. I learned basic and assembly langage when i was a kid (I wish somebody showed me lisp at that time) and I now spend most of my time coding in python.
Lisp looks to me like a lost langage, the atlantide of programming :) Like we had a past civilisation that was so great that we yet can't do all they used to make. The pyramid builder of the programming era :)
I think we can make a great basic lisp using the already developed object of redis (list, string, hashes, sets..)
This would probably be a redis specific lisp though..
Kader
This is just some idea i am throwing, I am in my 30s so i never used LISP in real life. I learned basic and assembly langage when i was a kid (I wish somebody showed me lisp at that time) and I now spend most of my time coding in python.
Lisp looks to me like a lost langage, the atlantide of programming :) Like we had a past civilisation that was so great that we yet can't do all they used to make. The pyramid builder of the programming era :)
I think we can make a great basic lisp using the already developed object of redis (list, string, hashes, sets..)
This would probably be a redis specific lisp though..
Kader
03 May 11, 21:51:37
@Alessandro: Because it is not. It is part of the LuaFileSystem extension module. Nothing related to directories is included in stock Lua.
03 May 11, 21:57:59
@aallamaa: Because he's not *implementing* a complete Lua interpreter, he's just grabbing the language and compiling it in. As amazing as everyone says that Lisp is, Lua is easier to integrate with software and also easier for people who do not already know what Lisp is to learn.
04 May 11, 12:05:49
@Matthew sorry, i didnt know, i only use Lua as embedded in a music making app which does support file access. Anyway Salvatore's tip will do for now
04 May 11, 21:54:30
@Alessandro: If you would like to know more about Lua, the best source is the reference manual at http://www.lua.org/manual/5.1/
06 May 11, 04:22:14
@Matthew you are certainly right, LUA is much more easier. But we could have an exo-lisp (check this blog post http://blog.fogus.me/2011/05/03/the-german-school-... for details about the various lisp implementation). And I think this would provide a very very powerful scripting implementation given that any object defined from lisp could actually be a redis object and vice-versa, we already got lists, sets, etc. It could be a very very minimalist lisp.
Anyway now i need to learn LUA :)
Anyway now i need to learn LUA :)
06 May 11, 07:44:50
@aallamaa: The first thing to learn is that it is spelled "Lua" and not "LUA". Why is that so hard for people to understand?
07 May 11, 02:20:30
This is fantatstic. We deployed it in production (where nothing mission critical run). Has anyone a patched version of redis-py with the eval function implemented?
12 May 11, 18:55:43
We have bunch of additional commands implemented in
https://github.com/petrohi/redis/wiki
We use them to express complex queries as dataflows between temporary keys, which are executed in pipelined fashion -- all to avoid roundtrips with intermediate results.
I rewrote then all with Lua scripting in a matter of day and it all runs smoothly on stock 2.2.107. Would love to see this feature in stable 2.2. Good job!
https://github.com/petrohi/redis/wiki
We use them to express complex queries as dataflows between temporary keys, which are executed in pipelined fashion -- all to avoid roundtrips with intermediate results.
I rewrote then all with Lua scripting in a matter of day and it all runs smoothly on stock 2.2.107. Would love to see this feature in stable 2.2. Good job!
12 May 11, 18:58:45
@Peter: this is awesome! Really a on the road test of this feature. Can I ask you something? I and Pieter are very biased for providing scripting without too much Lua additional libraries and contact with the external world (OS and so forth), are you using scripting in this way? That is, just to put logic server side and gain speed and latency? I guess so but I would love to have a confirmation.
This is the right way to use it, and not to write file on disks or talk with socket and so forth. So we want to stress this in the design.
This is the right way to use it, and not to write file on disks or talk with socket and so forth. So we want to stress this in the design.
12 May 11, 19:15:14
Yes, basically our use of scripting in very contained, or in functional terms has no side effects and does not retain any state -- this way it is a) much easier to reason about consistency especially in replication case b) performance implications are very transparent.
So additional libraries if they in turn have no side effects can be useful, but not required in our case. E.g. SHA hash calculation or similar things.
Libraries that go to OS I would consider harmful because they are most likely to provoke side effects and negatively affect performance when people do not realize Redis internals such as single threaded access to data structures. Solving these issues may complicate design enormously that in turn will affect simple and clean use case.
So additional libraries if they in turn have no side effects can be useful, but not required in our case. E.g. SHA hash calculation or similar things.
Libraries that go to OS I would consider harmful because they are most likely to provoke side effects and negatively affect performance when people do not realize Redis internals such as single threaded access to data structures. Solving these issues may complicate design enormously that in turn will affect simple and clean use case.
13 May 11, 03:42:39
@Peter: Thanks Peter, your contribution makes me even more confident about the fact this is the best direction.
13 May 11, 04:29:41
To echo what @Peter said: Moodstocks API v2 is going out live next week, along with a full rewrite of our image search engine that uses Redis instead of Tokyo Cabinet. This version is already in production in our in-house iPhone application Moodstocks Notes. It used to depend on a forked version of Redis with three custom commands (MHDEL, MHLEN, MHSET). I have re-written them with EVAL and now I can use vanilla Redis (with scripting) instead.
I have other code that uses custom commands that I will port later but that was the most critical and I am extremely happy with the results so far.
I don't need to communicate with the OS or anything and I do not think I will ever need it. Parsing binary (eg. MsgPack) or text (eg. JSON) data, though, is something I might need sometimes.
I have other code that uses custom commands that I will port later but that was the most critical and I am extremely happy with the results so far.
I don't need to communicate with the OS or anything and I do not think I will ever need it. Parsing binary (eg. MsgPack) or text (eg. JSON) data, though, is something I might need sometimes.