Cheeky Python: A Redis CLI
Posted July 21, 2010
Recently, at work, we started using MiniRedis as a lightweight store for some job queuing on Kiln's backend. We had originally planned on using the full Redis, but because we have to deploy licensed Kiln on Windows, we had to come up with our own solution.
So far, it's been working very well for us. The Redis command set is pretty small and very straightforward, which makes it easy to clone. The only annoyance we've run into is that the stable command line interface (CLI) for Redis only speaks the 1.x protocol, while MiniRedis only speaks the 2.0 version. The redis CLI also does not run on Windows. This is a bit of a problem, since we're still working on our queuing system and we need to do testing.
To get around not having a client to use, Ben would telnet in to the MiniRedis port and type out the commands manually. It ended up looking a bit like this:
I, on the other hand, would fire up Python and, using the redis- py library (which is a very nice client library), issue commands directly from there. Neither option was very convenient.
So one day, tired of having to do all the imports and set up the connection, I decided to put together a CLI using Python.
import cmd
class RedisCli(cmd.Cmd, object):
pass
if __name__ == '__main__':
RedisCli().cmdloop()
The basic CLI is very simple. Python's cmd
module takes care of all the hard
parts for you. If you run this script, you'll get a prompt (Cmd)
that
supports one command: help
. Unfortunately, that's all you have. You can't
even exit gracefully. So that's the first thing I added:
class RedisCli(cmd.Cmd, object):
def do_exit(self, line):
return True
do_EOF = do_exit
The cmd
module lets you define commands by writing a do_foo()
method which
takes the line the user typed in. The above code gives you the exit
command,
and also makes EOF
(Unix: Ctrl-D, Windows: Ctrl-Z) exit for you. That's
helpful, but it doesn't really add much in terms of actual functionality. For
that, we import the Redis client libraries
from redis import Redis
initialize the connection
class RedisCli(cmd.Cmd, object):
def __init__(self, host, port):
self.redis = Redis(host=host, port=port)
and add some of the Redis commands
def do_get(self, line):
print self.redis.get(line)
def do_set(self, line):
key, value = line.split()
print self.redis.set(key, value)
This is nice, because help
will now list get
and set
. But Redis has
many more commands
available. One option would be to continue adding all the Redis commands until
you had the full set specified and properly parsing the command line. That's
pretty time consuming, brittle, and just plain boring. Personally, I don't
have the patience.
Fortunately, the cmd
module has a default()
method which is called for any
unrecognized functions. That means we can get rid of do_get
and do_set
and
replace them with:
def default(self, line):
parts = line.split()
print getattr(self.redis, parts[0])(_parts[1:])
Huh? Let me explain: getattr()
takes an object and an
attribute name, and returns that attribute if it exists. To illustrate,
calling getattr(obj, 'foo')
is the same as calling obj.foo
. In this case,
we're assuming the user typed in one of the functions defined in the Redis
object. We use getattr
to get that function, and then we pass the rest of
the arguments to it using Python's _args
syntax. This sidesteps the problem
of not knowing how many arguments the commands take.
Unfortunately, this approach is prone to errors. For example, the user can
type in __init__
, which will call the Redis
object's
constructor again, overwriting our connection. Someone could also try to call
foobarbazbat
, which does not exist in the Redis
object and will throw an
error. Lastly, we've also lost our list of commands when you type help
.
To fix this, we're going to have to do some spelunking in the Redis
object.
Fortunately, Python's dir()
function returns all of the Redis object's
attributes. We can then iterate over them, filter out any that start with an
underscore (Python's convention for private attributes) and make sure they're
callable. We then use setattr
to create a function in our own class that
calls into the Redis
object and prints the result.
def __init__(self, host, port):
super(RedisCli, self).__init__()
self.redis = Redis(host=host, port=port)
for name in dir(self.redis):
if not name.startswith('_'):
attr = getattr(self.redis, name)
if callable(attr):
setattr(self.__class__, 'do_' + name, self._make_cmd(name))
@staticmethod
def _make_cmd(name):
def handler(self, line):
parts = line.split()
print getattr(self.redis, name)(*parts)
return handler
Notice here that _make_cmd
is creating a new function inside of it, and
returning that function so we can set the do_foo
of our own class to the
function that calls self.redis.foo()
. Likewise for any callable function in
the Redis
class.
Now if we type help
on our command line, we'll get a list of all functions
in the Redis
object. Also, if we try to access anything private, like
__init__
, we'll be told that syntax is unknown. This also means that our
default()
method is no longer necessary, since we've already enumerated
everything that we could possibly call on the Redis
object.
You'll notice, however, that help lists all of the functions as
"Undocumented". It would be really nice if we could also get documentation for
each of these commands. Now that we can easily list all of the available
commands, we could write the documentation ourselves by specifying a
help_foo()
function for each command. However, this is boring and like I
said before, I don't have that kind of patience. It also turns out that
writing our own documentation would be redundant, as the authors of our Redis
client library have done a good job documenting each function in the form of
docstrings:
def get(self, name):
"""
Return the value at key name, or None of the key doesn't exist
"""
Python takes these docstrings pretty seriously. In fact, they become an
attribute of the function itself, called __doc__
. This is
great for us, because it means we can pull those docstrings into our CLI and
make them documentation for our commands. We use the same method as before to
dynamically add help_foo()
methods to our own class for every function that
has a docstring:
doc = (getattr(func, '__doc__', '') or '').strip()
if doc: # Not everything has a docstring
setattr(self.__class__, 'help_' + name, self._make_help(doc))
@staticmethod
def _make_help(self, doc):
def help(self):
print doc
return help
So now if we type help
, we'll get a list of "Documented" and "Undocumented"
commands. If we type help get
, it'll tell us "Return the value at key
name
, or None of the key doesn't exist." Awesome!
This is getting to be a useful little CLI. In addition to having a documented
list of all commands available, we also get tab completion for our commands,
so if we type l<TAB>
, we get the list of all list commands. (On Windows,
you'll need the pyreadline module
installed for this to work.) But what if we could also autocomplete our keys?
Some of our keys get pretty long, and typing them out is a pain. We could
define a complete_foo()
method for each of our functions, but all we're ever
going to be completing are the keys, so we can just use the completedefault
,
which is a catchall completion, to grab our keys for us.
def completedefault(self, text, line, start, end):
return self.redis.keys(text + '*').split()
Once we've added this, we can type get bar<TAB>
and we'll get all of the
keys that start with bar
.
We're in the home stretch now. Just to make things a little nicer, let's modify the prompt and the intro message so the user knows they're in the Redis CLI:
def __init__(self, host, port):
...
self.prompt = '(Redis) '
self.intro = 'nConnected to Redis on %s:%d' % (host, port)
And finally, because this is a tool we want to be able to use with different
servers, let's add the ability to specify a host (-h
or --host
) and port
(-p
or --port
).
import getopt
if __name__ == '__main__':
opts = dict(getopt.getopt(sys.argv[1:], 'h:p:', ['host=', 'port='])[0])
host = opts.get('-h', None) or opts.get('--host', 'localhost')
port = int(opts.get('-p', None) or opts.get('--port', 12345))
RedisCli(host=host, port=port).cmdloop()
To finish, we just add some error checking so we don't get bailed out with an exception if we happen to make a typo. Here's the final script:
import cmd
import getopt
import sys
from redis import Redis
from redis.exceptions import ConnectionError
class RedisCli(cmd.Cmd, object):
def __init__(self, host, port):
super(RedisCli, self).__init__()
self.redis = Redis(host=host, port=port)
self.prompt = '(Redis) '
self.intro = 'nConnected to Redis on %s:%d' % (host, port)
for name in dir(self.redis):
if not name.startswith('_'):
attr = getattr(self.redis, name)
if callable(attr):
setattr(self.__class__, 'do_' + name, self._make_cmd(name))
doc = (getattr(attr, '__doc__', '') or '').strip()
if doc:
doc = (' ' * 8) + doc # Fix up the indentation
setattr(self.__class__, 'help_' + name, self._make_help(doc))
try:
# Test the connection. It doesn't matter if 'a' exists or not.
self.redis.get('a')
except ConnectionError, e:
print e
sys.exit(1)
@staticmethod
def _make_cmd(name):
def handler(self, line):
parts = line.split()
try:
print getattr(self.redis, name)(*parts)
except Exception, e:
print 'Error:', e
return handler
@staticmethod
def _make_help(doc):
def help(self):
print doc
return help
def completedefault(self, text, line, start, end):
return self.redis.keys(text + '*').split()
def do_exit(self, line):
return True
do_EOF = do_exit
def emptyline(self):
pass # By default, cmd repeats the command. We don't want to do that.
if __name__ == '__main__':
opts = dict(getopt.getopt(sys.argv[1:], 'h:p:', ['host=', 'port='])[0])
host = opts.get('-h', None) or opts.get('--host', 'localhost')
port = int(opts.get('-p', None) or opts.get('--port', 12345))
RedisCli(host=host, port=port).cmdloop()
And there we have it. A full-featured, robust, well-documented Redis CLI in about 60 lines of code. For comparison, the C version is over 500 lines of code, and has no help documentation or code completion.
The best part, though, is that this doesn't really know anything about Redis
at all. The only parts that are aware of Redis are __init__()
, which sets up
the Redis
object, and completedefault()
, which gets our keys. That means
that you could easily adapt this script to be a CLI on top of any client
library you have.