Comnts for post Redis and scripting

Comments for post Redis and scripting

Kader writes: @antirez you´ve played with scheme/lisp so why not implement a redis lisp dialect?

Marco Rogers writes: @Felix Gallo, it seems to me that if the problem domain is loosely defined, then expanding the redis API with a bunch more commands is the exact wrong approach. If things don't work out, you have to decide what to do with people who are using this expanded API. Or you have to cut them loose. Neither of these is desirable. Adding one command that enables scripting is more future proof as the whole thing is "siloed" and can be easily abandoned or re-imagined if necessary. Embedding a scripting language and monitoring use cases is the best way to explore the problem IMO.

Carl Zulauf writes: Looking forward to taking advantage of scripting support in Redis. Sounds like you are taking a really well-thought-out approach and I'm just hoping to get my hands on it soon :)

Felix Gallo writes: Salvatore, I humbly suggest that Lua is a premature optimization to a loosely defined problem domain. If the problem is a question of expressivity/richness, then I'd suggest that implementing the common relational algebraic and grouping operators in plain Redis command language would go 99.95% of the way there, while maintaining the design paradigm. For the other .05%, let them fork redis and write their own C.

dubek writes: The problem of storing functions/scripts in database keys is the problem of mixing data and code. Say you choose to revert your database to snapshot from a week ago; now not only your data is week old, but also some of the Lua functions you stored as keys in the database. In this sense, antirez's idea of sending the entire script each time avoids this problem. But, as people mention, it would start to be cumbersome to use bigger scripts (with functions defined in them and so on). I'm not sure how UDFs are stored in MySQL - maybe there's an idea there than you can use.

Matthew Frazier writes: @Pixy Misa: This would be fairly trivial to do in Lua. You could load the script with something like `db.set("SCRIPTNAME", string.dump(--[[ define a function in here ]]))` and run it with `assert(loadstring(db.get("SCRIPTNAME")), "script not present")(...)`.

Ethan writes: Definitely a good addition to Redis, a thing that has been missing. How's about to begin with, adding very basic support like test-and-set -- read a hash key, check against a given value, if different set to a new value, else, say touch the TTL or delete. This will help in replacing several RMW operations on redis from a client with a single call. Would be nice if we can have similar crisp commands, which can be quickly released with redis as well.

Pixy Misa writes: Lua is brilliant, LuaJIT is exceptionally fast, and Redis+Lua will be awesome. :) I'd definitely be interested in being able to store the script on the server side rather than sending it every time. If we use hashes for this I'd suggest sending a script name/identifier as well as the hash so that users can easily which script they're running.

Michael writes: LuaJIT is way faster than V8 and has lower memory footprint. There was some information at "Language shootout" site but they removed V8 & LuaJIT benchmarks :( Another very important thing is that Lua was created with embedability in mind, and it is really easy to use in cases like this one with Redis.

geek42 writes: how about forth?

jaksprats writes: luajit is faster than V8. additionally lua embeds itself so well into C, that lua can be called w/o invoking the lua interpreter. also lua can call C, so lua scripts can call redis' commands at basically C speed. lua can also load files, w/ function definitions in them and the functions are stored in lua's memory, no need for redis to store functions in a hash, no need to pass function definitions over the wire. Lua embeds itself very nicely into redis, and luajit2 makes it near post JITed Java fast .... Ive been playing around w/ lua in redis for 9 months, and it kicks ass, I expect people to love it, and I also expect them to misuse it initially, but thats life

weepy writes: out of interest. where does lua beat v8 ?

Piotr Sikora writes: @antirez: Hehe, that's true :) Anyway, good luck with the scripting stuff! Picking hash is probably least of the problems, so I'll end it here.

antirez writes: @piotr: with a good hash that is collision resistant, and where flipping a single bit in the output will on the average get 50% of probability of every output bit to flip, you can consider same hash the same script, as it is much more probable than everything wrong else happens than a random collision.

Piotr Sikora writes: @antirez: I can't say that I agree with you. Same-length hashes (good hashes) should have pretty much the same collision rate regardless of their crypto/non-crypto properties. And you should not assume that same hash == same source (although in this exact case you most likely can).

antirez writes: @Piotr: if you use SHA1 or any other crypto-level hash function you can avoid checking at all for the script when you do the hashing. If it matches, it is the same script.

Piotr Sikora writes: Why do people insist on using cryptographic hashes for non-cryptographic purposes? :P @antirez: Please consider using Murmur, Jenkins, etc. Algorithm is a lot simpler, a lot faster and implementation is usually in the public domain.

Marcus writes: The part I'd want to reduce most is the sending of the body of the script over the network each time. I know the hashing will be quick, but if you have long scripts, personally I'd rather fetch them from another server only when needed. Having two interfaces, one with hashes and one with just the scripts I think would be fairly simple.

antirez writes: @Marcus: I think we can directly hash the script, SHA1 is really fast, millions of times per second, so probably it is better to have a simpler API. Otherwise I can use the *whole* script as a key into the hash table if it is short enough, and use the SHA1 only for larger scripts.

Marcus writes: On top of a command to send the script, you could also have a command where you send a hash (or some other kind of reference) of the script along with the script itself. Something like EVAL_WITH_KEY [hash] [script] [args] The purpose of this would be to help reduce script parsing / compilation time. When Redis receives the request, it first checks to see if it has the script given by the hash in its compiled form. If that's the case, then it just runs the script. If it doesn't, then it compiles the script and adds it to is tree of compiled scripts. The key would be defined by the client, so it would need to be consistent for all servers. You could also have a different command, where instead of sending the script, you just send the hash/key of the script. If the Redis server doesn't have the script pre-compiled, then it sends a request to another server (Redis, HTTP or whatever) to fetch the script (obviously the config for this would need to be set up on each server, but it could include redundant servers). Something like EVAL_KEY [hash] [args] This method has a couple of key advantages: - you avoid needing to send the script over the network for each request - you don't have to parse the script code on every request (which you'd need to do if you don't use the hash method above or if you put the script before the arguments) - it would be consistent across a cluster (so long as you made sure that the hashes/keys of the scripts were different for each script, which would be easy), so you don't have the different copies issues you mention

antirez writes: @yesso this idea was also expored, but you end with the same problems as with registering scripts. Possibly different instances with different versions of the script, and so forth...

yesso writes: Perhaps you could have another data type, "script", in addition to the existing data types. Then you can set a script into a key in the same way that you currently set a string or a hash into a key. And then, instead of specifying the script again and again on every command, you can specify the key of the script that was set into Redis at an earlier time. This saves on bandwidth and allows you to precompile the script.

antirez writes: The time spend inside Lua will be very small compared to command dispatch I think, but there is something interesting we can do to make scripting trivial to implement and at the same time have decent Lua performances... more soon :)

Jonatas Esteves writes: @Tobias Lua actually has a JIT compiler which is way faster than V8. It's called LuaJIT by Mike Pall [http://luajit.org/]. But even it's pure interpreter implementation is more than fast enough for this use case.

P-A -> @pastjean writes: if you forget Ruby , js & python , personally Io is little more sane than Lua http://iolanguage.com/ Having a stack based interpreter on a pointer based language is kind of harsh, you always have to rebase the stack. http://julien.danjou.info/blog/2011.html#Why_not_Lua Why not have pluggable scripting engines ? complexity in implementation? Tough I like the IDEA!

Jonatas Esteves writes: Just like what JakSprats did with Alchemy Database on top of Redis, but without all the Relational nonsense. I just LOVE this idea! Love the way you plan to implement it and love the choice of programing language too. What's not to love about it!? Kudos for building the best DB ever! I'll definitely try it.

Tobias writes: How about background compilation of LUA-Scripts into plain-c for maximum efficency? Or using Googles V8 with it's internal compilations, it beats LUA-Performance for miles.

Marc writes: The Lua integration reminds me of Tokyo Tyrant :) Love the idea!

home