Comments for post Scripting branch released

catwell writes: To echo what @Peter said: Moodstocks API v2 is going out live next week, along with a full rewrite of our image search engine that uses Redis instead of Tokyo Cabinet. This version is already in production in our in-house iPhone application Moodstocks Notes. It used to depend on a forked version of Redis with three custom commands (MHDEL, MHLEN, MHSET). I have re-written them with EVAL and now I can use vanilla Redis (with scripting) instead. I have other code that uses custom commands that I will port later but that was the most critical and I am extremely happy with the results so far. I don't need to communicate with the OS or anything and I do not think I will ever need it. Parsing binary (eg. MsgPack) or text (eg. JSON) data, though, is something I might need sometimes.
antirez writes: @Peter: Thanks Peter, your contribution makes me even more confident about the fact this is the best direction.
Peter Hizalev writes: Yes, basically our use of scripting in very contained, or in functional terms has no side effects and does not retain any state -- this way it is a) much easier to reason about consistency especially in replication case b) performance implications are very transparent. So additional libraries if they in turn have no side effects can be useful, but not required in our case. E.g. SHA hash calculation or similar things. Libraries that go to OS I would consider harmful because they are most likely to provoke side effects and negatively affect performance when people do not realize Redis internals such as single threaded access to data structures. Solving these issues may complicate design enormously that in turn will affect simple and clean use case.
antirez writes: @Peter: this is awesome! Really a on the road test of this feature. Can I ask you something? I and Pieter are very biased for providing scripting without too much Lua additional libraries and contact with the external world (OS and so forth), are you using scripting in this way? That is, just to put logic server side and gain speed and latency? I guess so but I would love to have a confirmation. This is the right way to use it, and not to write file on disks or talk with socket and so forth. So we want to stress this in the design.
Peter Hizalev writes: We have bunch of additional commands implemented in https://github.com/petrohi/redis/wiki We use them to express complex queries as dataflows between temporary keys, which are executed in pipelined fashion -- all to avoid roundtrips with intermediate results. I rewrote then all with Lua scripting in a matter of day and it all runs smoothly on stock 2.2.107. Would love to see this feature in stable 2.2. Good job!
goz writes: greatest thing, i feel like am addicted to it
Laurent writes: This is fantatstic. We deployed it in production (where nothing mission critical run). Has anyone a patched version of redis-py with the eval function implemented?
aallamaa writes: Matthew honestly, we dont give a damn
Matthew Frazier writes: @aallamaa: The first thing to learn is that it is spelled "Lua" and not "LUA". Why is that so hard for people to understand?
aallamaa writes: @Matthew you are certainly right, LUA is much more easier. But we could have an exo-lisp (check this blog post http://blog.fogus.me/2011/05/03/the-german-school-of-lisp-2/ for details about the various lisp implementation). And I think this would provide a very very powerful scripting implementation given that any object defined from lisp could actually be a redis object and vice-versa, we already got lists, sets, etc. It could be a very very minimalist lisp. Anyway now i need to learn LUA :)
Matthew Frazier writes: @Alessandro: If you would like to know more about Lua, the best source is the reference manual at http://www.lua.org/manual/5.1/
Alessandro writes: @Matthew sorry, i didnt know, i only use Lua as embedded in a music making app which does support file access. Anyway Salvatore's tip will do for now
Alessandro writes: @antirez, thanks for the redis-cli eval "$(cat /tmp/myscript.lua)" 0 tip!
Matthew Frazier writes: @aallamaa: Because he's not *implementing* a complete Lua interpreter, he's just grabbing the language and compiling it in. As amazing as everyone says that Lisp is, Lua is easier to integrate with software and also easier for people who do not already know what Lisp is to learn.
Matthew Frazier writes: @Alessandro: Because it is not. It is part of the LuaFileSystem extension module. Nothing related to directories is included in stock Lua.
aallamaa writes: @antirez I have a feeling that redis would be the perfect sandbox for a lisp based dialect. So I am wondering why you took the decision to implement LUA for instance instead of Lisp or any other very simple langage from the syntax perspective, especially since you´ve been playing with scheme as you stated in your previous post. This is just some idea i am throwing, I am in my 30s so i never used LISP in real life. I learned basic and assembly langage when i was a kid (I wish somebody showed me lisp at that time) and I now spend most of my time coding in python. Lisp looks to me like a lost langage, the atlantide of programming :) Like we had a past civilisation that was so great that we yet can't do all they used to make. The pyramid builder of the programming era :) I think we can make a great basic lisp using the already developed object of redis (list, string, hashes, sets..) This would probably be a redis specific lisp though.. Kader
Navigatore Anonimo writes:
antirez writes: @Alessandro: if you are talking about scripts development: Edit your script with your preferred editor into /tmp/myscript.lua To execute it use: redis-cli eval "$(cat /tmp/myscript.lua)" 0 Salvatore
Alessandro writes: i suppose loadfile() is out of the questionthen .. editing a 1K function on the cli is not much fun and none of my clients are written in ruby unfortunately
Martijn writes: Cool.. I like the idea of sending the script in the command and Lua seems easy enough to understand. If this were to make it into stable I'm sure I would use it at some point or another.
antirez writes: @Alessandro: Redis binds with Lua with "ansi" target. So POSIX stuff are not supported. In general we'll try to sandbox it even more probably... @Lloyd: thanks, cool to know!
Lloyd Moore writes: I am loving how redis has solved so many of our previous problems and the enhancements with scripting is going to clean up some of our php classes enormously.
Alessandro writes: also, any idea why i cant seem to access to os.currentdir()? it is in the lua 5.1 spec.. redis> eval "return {ok=os.date()}" 0 Tue May 3 12:14:44 2011 redis> eval "return {ok=_VERSION}" 0 Lua 5.1 redis> eval "return {ok=os.currentdir()}" 0 (error) ERR Error running script (call to f_3cba57088cac1f60388c6085385a464eef74f42c): [string "func definition"]:2: attempt to call field 'currentdir' (a nil value)
Alessandro writes: if you plan on implementing a script flush please do add the possibility of having a lua script autoloaded when it happens -- I too am going to be using lua functions
Alaric Snell-Pym writes: The EVALSHA approach is elegant indeed. I came up with something similar for a communication protocol for talking to clusters of servers - the idea is that the cluster is identified by a public key and a list of IP addresses / hostnames of the members. A client uses that identifier to try cluster members until it gets one that works. It then embeds a hash of the list of members in its request, which is encrypted with the cluster's public key so that only actual cluster members can reply correctly. If the cluster has changed (nodes added/removed/moved), then the hash of the list of members will have changed, so the server includes an up to date list in its (signed) response. Voila - clients can follow a moving cluster (unless it moves TOO fast!) at the cost of only including an extra hash in requests, and then having a new list of members in responses only when it's needed. tl;dr: HASHES ARE MAGIC AND SOLVE DIFFICULT PROBLEMS IN SURPRISING WAYS.
antirez writes: @Jonathan: indeed, the same Lua interpreter is used again and again, but warning we plain a "SCRIPT FLUSH" command or alike that flushes and creates the Lua interpreter for scratch :) This will mostly be used when the Lua interpreter gets polluted on in cloud environments when a spare Redis instance needs to get completely wiped.
Jonathan Castello writes: @antirez: Thanks for your response! I also just discovered another (probably easier) way to store scripts: EVAL "myscript = function() return 42 end" 0 Then later on: EVAL "return myscript()" 0
antirez writes: @Jonathan: hello. about your first question, we need to provide a coherent Redis -> Lua, Lua -> Redis conversion, so the same data type Lua gets when calling a Redis command returning a multi bulk should be used when returning a value to Redis. The table is the type that semantically resembles more closely the bulk reply. About loadstring() we are aware this can be done and will not deny it, but it is discouraged for the reasons explained in this blog post and in the previous one. Thanks for your comment!
Jonathan Castello writes: Also, if you were to store a script in a Redis key, I believe you could use loadstring() to eval the script. Using that technique you could run a script at a given key using a simple line: return loadstring(redis("get", KEYS[1]))() EDIT: After looking at the code, I guess you're actually wrapping the script in a function and executing it. I'm curious why you're doing that. Couldn't you just set the ARGS/KEYS/redis/etc variables into _G and run the script as-is?
Jonathan Castello writes: This looks really cool! I do have a question though. You show how you can use an array to return multiple values, but is there any reason you're not taking advantage of Lua's in-built support for multiple return values? In other words, is there any reason "return {1, 2, 3}" is better than "return 1, 2, 3"?
antirez writes: @jaksprats thank you! I really hope your prediction will turn true, it will be very interesting and instructive to see how people can use this to solve real problems.
jaksprats writes: great implementation Antirez. I think you nailed ALL of the difficult parts in defining an intuitive/useful/fast API between Lua and Redis, hats off to you :) My prediction, people start blogging real quickly on very interesting things they have done w/ lua-enabled-redis.
antirez writes: @Alessandro, @Matthew: now the bug is fixed, thank you. It is now possible to use HGETALL and any other command.
Alessandro writes: Thanks for all the info. Great work.
harald writes: in my opinion it' s a very good decision to implement scripting using lua, because it's a very powerful language, with very small footprint. it's very easy to learn, too. wow, this all sounds awesome, can't wait to play with it ...
Matthew Frazier writes: @Alessandro 1: Using HGET. @Alessandro 2: This is still very unstable. @Alessandro 4: The Redis protocol does not support returning doubles directly to clients, only integers. In Lua, the two share a single type - number - so any returned numbers are converted to integers, which drops the fractional parts. To return a double, you will need to convert it to a string with the tostring function, and then convert it back to a number in the client code.
Alessandro writes: redis> eval "return 3.1415" 0 (integer) 3 is there any way to use real numbers?
Alessandro writes: i must be doing something wrong.. redis> eval "return {(redis('hget',KEYS[1],KEYS[2])),(redis('hget',KEYS[1],KEYS[3]))}" 3 hq val1 val2 1) "1.48559" 2) "120.669" redis> eval "return {(redis('hget',KEYS[1],KEYS[2]))*(redis('hget',KEYS[1],KEYS[3]))}" 3 hq val1 val2 1) (integer) 179 redis>
Alessandro writes: I ask because eval "return redis('hgetall',KEYS[1])" 1 hashname crashed my server
Alessandro writes: Is there a way to address fields within an hash? I need to multiply one hash field by another. Are only regular keys supported?
Matthew Frazier writes: @Baishampayan Ghose: JavaScript implementations are heavier, slower, and harder to embed into C code than Lua, and less portable to boot - Lua can run on anything with a C compiler. Also, JavaScript is ugly. @loopole: Probably the full "stdlib" would be overkill. Some utility functions would definitely be nice, though. I think a higher priority than "battery" libraries right now is sandboxing - as is, you can EVAL "os.exit()" and shut down the Redis process, and scripts can mess around with the environments of other scripts.
antirez writes: @loopole thanks, this is a huge amount of good suggestions! Putting part of the std lib sounds like a great idea indeed. Also the JSON idea is awesome :) Thank you.
@loopole writes: Great! I think it will pay a lot to keep the implementation this simple. Aside from the 1 indexed tables, I think Lua is really the best language for the purpose, as its a really simple language to learn, REALLY FAST, and once you get into it's metatable & functionnal style programming you can really see the power behind its apparent simplicity... It's really a matter of taste and I know we should not have debate over Lua vs JS, still my taste is that Lua code is more readable than Javascript which is mostly ugly. One question remains for me... since Lua has no batteries included, people will probably like to use helper libraries in their command scripts, how about including part of stdlib http://luaforge.net/projects/stdlib/ into redis scripting by default? Like the base, tables, list, string and set libs, also having a Lua JSON serializer/deserializer would be nice, since many are serializing data into redis. Thanks Antirez! Great Work!
Mark Essel writes: As a guy who's used redis once, and Lua JIT a couple of times I have to admit I'm kinda excited by this. I'm addicted to documentation, and there's plenty of great docs on redis and lua :). Need a good starter project to make use of this peanut butter and jelly combo.
antirez writes: @Ann: I think our friend is just playing troll :) Thanks btw.
Ann E. Mouse writes: "bloated code base which is Redis" Wait, if redis is bloated, what are some exmples of projects that you would consider "not bloated" Please, I can't think of projects more slim than redis.
Daniel Huckstep writes: This is awesome. Do want! This makes a CAS operation trivial.
antirez writes: @Sunnil Mahotra: but are you the actor? http://www.imdb.com/name/nm0539503/
Jordi Boggiano writes: This all sounds great! The only comment I'd have is that it'd probably be easier if all args would be returned in one ARGV table, otherwise if you have 4 keys and then 5 args, you always need to think to offset -4 when accessing ARGV, it's not a big deal, but imo it should just be one list of arguments.
rch writes: Lua is an excellent fit. Many thanks.
Sunnil Mahotra writes: I am sorry to say, but no amount of Lua scripting will prevent Redis from being on the deathmarch it is on. No amount of scripting can fix the poorly implemented bloated code base which is Redis.
catwell writes: @Baishampayan http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html JS: 11: Lua: 12 Doesn't look like much of a difference to me. @Theo That "tables start at 1" is one of the main things that confuse new Lua users, along with "variables are global by default". However, you get used to it quickly, and then it looks more logical imo.
Theo writes: That Lua uses 1 for the first element in a list will cause much grief.
Tyrael writes: Baishampayan, while I agree with you that the average redis user have more experience with js than with Lua, but to compare Lua to C is really seems like you didn't know much about it. Tyrael
brad dunbar writes: When I read the previous post I was rather skeptical about scripting redis but you've completely won me over with this one. It'll be a good excuse to play with redis and also learn lua. Also, the EVALSHA command is a terrifically elegant way to solve the bandwidth problem. Well done.
Baishampayan Ghose writes: Michael: I don't think I need to qualify my statement that JavaScript is more popular than Lua -- it's quite obvious. By us, I broadly meant (web?) programmers, people like us have better exposure to JS than Lua. Antirez: Sure, Lua is not very different from C but I hope you don't think all Redis users are as good at C as you are ;)
jbland writes: Whoa ! I just took a look at the Lua function reference, and the skies opened. I see a redis Bloom filter in my immediate future.. and server side proximity calcs, and.....
antirez writes: @Phil: no, it is very clearly specified in the article (and in the previous one) why this is not a good idea in my opinion. However you can do it already, and easily, using Lua's loadstring() function if you really want, but it is discouraged.
Phil writes: Would it not be possible to offer an execkey which will execute the LUA script stored at a key - that way we can deal with versioning if we require (do a set during execution setup and a del during teardown if necessary) and reduce complexity in client libraries?
geek42 writes: its really useful for SNS site, cause they need lots of different data for one page and to avoid waste of memory , these data should be atomic , so redis with lua binding solve the problem with high performance
antirez writes: Lua is a mostly algol-like programming language, and you are supposed to write short scripts to implement new commands. Total time require to master it enough? 5 minutes. Lua VS Javascript debate is completely useless ;)
Joo writes: To solve the bandwidth issue maybe just have another command that evaluates precompiled lua bytecode, might save a few bytes but probably not much. I see this more usefull for admin or maintenance tasks as it has nice and easy access to the internals without any c hacking.
Paolo Possanzini writes: Very good stuff
Michael writes: Baishampayan Ghose, you're going to have to back that comment up with some evidence or it can be asserted that by "most of us" you really mean "I".
Baishampayan Ghose writes: This is good stuff, but IMHO scripting Redis via JavaScript instead of Lua will be much more helpful to Redis users as most of us know JavaScript well already.
home