Why speed matters?

• Raw speed is bound to queries per watt. Energy is a serious problem, not just for the environment, but also cost-wise. More computers running to serve your 1000 requests per second, the bigger your monthly bill.
• Large systems are hard to handle, and hardware is not for free. If I can handle the same workload with 10 servers instead of 100 servers, everything will be cheaper and simpler to handle.
• 95% of all the web innovation today is driven by startups. Startups start very very small, they don't have budget for large powerful computers. A software system that is able to handle a reasonable amount of load with a cheap Linux box is going to win a lot of acceptance in the most fertile of the environments for new technology toys, that is, startups.
• Scalable systems many times can work better with a smaller number of nodes, so even if there is the ability to scale it's better to run a 10 nodes cluster than a 100 nodes cluster. So the single node of a scalable system should be as fast as possible.

Systems for the toilet

• All the tests are run using a single client into a busy loop. I love '80s busy loops but... well, let's be serious. I left a comment in the blog about this problem, and the author replied that in the real world things are different and threading is hard. Sorry but I was not thinking about multi thread programming: a DB under load receives many concurrent queries by different instances of the web server(s) (that can be threads, but much more easily different Apache processes). So it is not required that your web dude starts writing concurrent software to have N queries at the same time against your DB. If your DB is under serious load, you can bet that you are serving many requests at the same time. Sorry if this is obvious for most of the readers.
• Not just the above, when you run single clients benchmarks what you are metering actually is, also: the round trip time between the client and the server, and all the other kind of latencies involved, and of course, the speed of the client library implementation.
• The test was performed with very different client libraries. For instance libmemcached accepted as input a buffer pointer and a length, while credis, the library used in the mentioned benchmark, was calling strlen() for every key. So in one benchmark there was a table of pre-computed lengths passed to the library, and in the other the lib was calling strlen() itself. This is just an example, there are much bigger parsing, buffering, and other implementation details affecting the outcome especially if you are using a single-client approach, that is perfect to meter this things instead of the actual server performance.

Redis VS Memcached

Apple to Apple, with the same mouth

@@ -213,8 +206,15 @@
     char *tail = strstr(p,"\r\n");
     if (tail == NULL)
         return 0;
-    *tail = '\0';
-    *len = atoi(p+1);
+
+    if (p[0] == 'E') { /* END -- key not present */
+        *len = -1;
+    } else {
+        char *x = tail;
+        while (*x != ' ') x--;
+        *tail = '\0';
+        *len = atoi(x)+5; /* We add 5 bytes for the final "END\r\n" */
+    }
     return tail+2-p;
 }

Methodology

• I used Redis master, the "betterparsing" branch to run the test. The result of the benchmark should be pretty similar to what you'll get with Redis 2.0, excluding the MGET test, that I'm not including now since it's not part of redis-benchmark currently.
• I used the latest memcached stable release, that is, memcached-1.4.5.
• Both systems were compiled with the same GCC, same optimization level (-O2).
• I disabled persistence in Redis to avoid overhead due to the forking child in the benchmark.
• I ran memcached with -m 5000 in order to avoid any kind of expire of old keys. This way both Redis and memcached were loading the whole dataset in memory.
• The test was performed against real hardware, an oldish Linux box completely idle.
• Every test was ran three times, the higher figure obtained was used.
• The size of payload was 32 bytes. Sizes of 64, 128, 1024 bytes were tested without big differences in the results. Once you start to go out of the bounds of the MTU, performances will degradate a lot in both systems.

#!/bin/bash

# BIN=./redis-benchmark
BIN=./mc-benchmark
payload=32
iterations=100000
keyspace=100000

for clients in 1 5 10 20 30 40 50 60 70 80 90 100 200 300
do
    SPEED=0
    for dummy in 0 1 2
    do
        S=$($BIN -n $iterations -r $keyspace -d $payload -c $clients | grep 'per second' | tail -1 | cut -f 1 -d'.')
        if [ $(($S > $SPEED)) != "0" ]
        then
            SPEED=$S
        fi
    done
    echo "$clients $SPEED"
done

Results

• memcache: user 8.640 seconds, system 19.370 seconds
• redis: user 8.95 seconds, system 20.59 seconds

Atomic operations can be traded for speed

Conclusions

Disclaimer