Basic X Window keyboard and mouse input blocking

Here’s a small Python script using python-xlib that will block the keyboard and mouse input and then display random-colored, random-sized rectangles whenever an event happens. I tested it in Xubuntu 14.10, not sure if it is possible to run on non-Linux platforms. Obviously, you’ll need to install python-xlib. I use virtualenv to do that, but there’s also a “regular” Debian package available.

from Xlib import Xatom, Xutil
from Xlib.display import Display, X
import sys
import signal 
import random

class bunch(dict):
  __getattr__ = dict.__getitem__
  __setattr__ = dict.__setitem__

def check_for_magic_keys(state, event):
  keys = state['keys']
  if event.type == X.KeyPress:
    keys[event.detail] = True
  elif event.type == X.KeyRelease:
    keys[event.detail] = False

  keycode_alt = 64
  keycode_1 = 10
  keycode_delete = 119

  magic_keys = keys.get(keycode_alt) and keys.get(keycode_1) and keys.get(keycode_delete)
  if magic_keys:
    print("Magic keys pressed, exiting")
    return True

  return False

def random_color(screen):
  red = random.randrange(0, 65536)
  green = random.randrange(0, 65536)
  blue = random.randrange(0, 65536)

  return screen.default_colormap.alloc_color(red, green, blue).pixel

def random_rectangle(screen, window):
  x = random.randrange(0, screen.width_in_pixels)
  y = random.randrange(0, screen.height_in_pixels)
  width = random.randrange(0, screen.width_in_pixels - x)
  height = random.randrange(0, screen.height_in_pixels - y)

  window.window.fill_rectangle(
    gc = window.gc,
    x = x,
    y = y,
    width  = width,
    height = height,
  )

def draw(state, event):
  screen = state.display.screen()
  foreground = random_color(screen)
  background = random_color(screen)

  state.window.gc.change(
    foreground = foreground,
    background = background,
  )

  random_rectangle(screen, state.window)

def handle_event(state, event):
  debug = False
  if debug:
    print(event)
    return True

  if check_for_magic_keys(state, event):
    return False

  draw(state, event)
  return True

def grab_keyboard_and_mouse(root):
  root.grab_keyboard(
    owner_events = True,
    pointer_mode = X.GrabModeAsync,
    keyboard_mode = X.GrabModeAsync,
    time = X.CurrentTime
  )
  
  root.grab_pointer(
    owner_events = True,
    event_mask = X.ButtonPressMask | X.ButtonReleaseMask | X.PointerMotionMask,
    pointer_mode = X.GrabModeAsync,
    keyboard_mode = X.GrabModeAsync,
    confine_to = 0,
    cursor = 0,
    time = X.CurrentTime
  )

def create_window(display, root):
  screen = display.screen()

  window = root.create_window(
    x = 0,
    y = 0,
    width = screen.width_in_pixels,
    height = screen.height_in_pixels,
    border_width = 0,
    depth = screen.root_depth)
  
  atom_net_wm_state = display.intern_atom('_NET_WM_STATE', True)
  atom_net_wm_state_fullscreen = display.intern_atom('_NET_WM_STATE_FULLSCREEN', True)

  window.change_property(
    property = atom_net_wm_state,
    type = Xatom.ATOM,
    format = 32,
    data = [atom_net_wm_state_fullscreen], 
  )

  window.set_wm_normal_hints(
    flags = Xutil.PPosition | Xutil.PSize | Xutil.PMinSize,
    min_width = screen.width_in_pixels,
    min_height = screen.height_in_pixels,
  )

  gc = window.create_gc(
    foreground = screen.black_pixel,
    background = screen.white_pixel,
  )

  return bunch(window = window, gc = gc)

def event_loop(state):
  display = state.display

  while True:
    event = display.next_event()
    display.allow_events(
      mode = X.AsyncBoth,
      time = X.CurrentTime)            

    if not handle_event(state, event):
      break

def main():
  display = Display()
  root = display.screen().root

  grab_keyboard_and_mouse(root)
  window = create_window(display, root)
  window.window.map()

  state = bunch(
    display = display,
    root = root,
    window = window,
    keys = bunch())

  # Comment these out after you have confirmed the magic key works
  signal.signal(signal.SIGALRM, lambda a, b: sys.exit(1))
  signal.alarm(4)

  event_loop(state)

if __name__ == '__main__':
  main()

How it works:

  • It creates a window that covers the whole desktop
  • It then grabs both the key and the mouse input
  • Whenever an event happens, it will first check for a magic key
  • If it is a magic key, it will exit
  • If it is not a magic key, it will display a random rectangle and continue to run

The magic key is currently set to Alt + 1 + Del, pressed at the same time. There’s also a small debug section in the event loop that can be enabled by setting debug = True. Printing events like this allows for reconfiguration if needed, e.g. to change the magic key.

The above version has the signal.alarm call – I left this there, so that it exits after 4 seconds while testing. Otherwise, if it doesn’t work, you have no input. Well, actually you do – I found that Alt + F1 works, so you can escape to a console and kill the python process, but while testing I found it easier to just do a timed exit. In order to be used for longer periods, the signal lines should be commented out.


Matplotlib – plotting stock prices

I was playing the other day with Matplotlib. I was amazed how easy it is to plot graphs, all with the builtin browser.

Say you want to plot some stock data. You can download a sample provided by EOD data site. They give a sample for AAPL prices, which can be downloaded from here. If you unzip, you’ll get a NASDAQ_AAPL.txt file, which actually is a csv.

Installation

Installing matplotlib is simple with pip:

$ pip install matplotlib

Plotting

Here’s a very short python code to read and plot it:

import matplotlib
matplotlib.use('webagg')

import matplotlib.pyplot as plt
import csv

with open('etc/NASDAQ_AAPL.txt') as inf:
  csvr = csv.reader(inf)
  csvr.next()
  prices = [float(row[2]) for row in csvr]
  
plt.title('AAPL prices')
plt.xlabel('Time')
plt.ylabel('Price')

plt.plot(prices)
plt.grid(True)
plt.show()

Walkthrough:

import matplotlib
matplotlib.use('webagg')

Imports matplotlib and sets the default backend used for plotting to WebAgg.

import matplotlib.pyplot as plt
import csv

with open('etc/NASDAQ_AAPL.txt') as inf:
  csvr = csv.reader(inf)
  csvr.next()
  prices = [float(row[2]) for row in csvr]

Import plotting part of matplotlib and the standard Python csv library. Read the file, skip the header and pick open prices (3rd column in the CSV file).

plt.title('AAPL prices')
plt.xlabel('Time')
plt.ylabel('Price')
plt.grid(True)

plt.plot(prices)
plt.show()

Set the title, X and Y axis labels of the plot and specify that plot should have the grid shown. Actually perform plotting and show it.

There’s a minor issue for which I filed a report here, but after manually updating the file as per the ticket, all seems to be working just fine.

After doing:

$ python matplotlib_prb.py

It will start a server and open your browser pointing to http://127.0.0.1:8988/.

Final plot

Here’s how it looks for the above file:

plot


Python – mix lists, preserving order

Say you have the following lists:

  • lower_case – A list of all lower-case letters
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',
'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']
  • upper_case – A list of all upper-case letters
['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M',
'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z']
  • digits – A list of digits
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

and a list of the above, i.e.:

list = [lower_case, upper_case, digits]
 

Now, you want to make a single list with the following properties:

  • All elements of the given lists need to be present
  • The order of elements of each of the individual lists needs to be preserved
  • The final list needs to be randomized

In our case, the final list would be a list of all lower-case letters, upper-case letters and digits, where all individual categories are still ordered (e.g. ‘a’ needs to come before ‘b’ in the final list), but randomized. Here’s one way to do it:

  • Start from the starting list
[['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',
'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'],
['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M',
'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]
  • Zip the lists to get the ids (which are conveniently list indices in the starting list) [
(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',
'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'], 0),
(['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M',
'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'], 1),
([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 2)]
  • For each list, create the list which has the same length, but whose every element is the index of that list. Make a master index list that contains these
[[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[2, 2, 2, 2, 2, 2, 2, 2, 2, 2]]
  • Flatten the master index list
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2]
  • Shuffle the master index list
[2, 2, 1, 0, 1, 1, 1, 0, 2, 0, 1, 1, 1,
2, 1, 0, 0, 0, 1, 0, 1, 2, 1, 0, 1, 0,
0, 1, 1, 1, 2, 0, 0, 0, 1, 0, 2, 1, 1,
1, 2, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0,
0, 0, 2, 0, 1, 0, 0, 0, 2, 0]
  • Create iterators for each of the starting lists. Put them in the iterators list
  • Go through the master index list and pick the iterator from the iterators list based on the index which is given by the current element
[0, 1, 'A', 'a', 'B', 'C', 'D', 'b', 2, 'c',
'E', 'F', 'G', 3, 'H', 'd', 'e', 'f', 'I',
'g', 'J', 4, 'K', 'h', 'L', 'i', 'j', 'M',
'N', 'O', 5, 'k', 'l', 'm', 'P', 'n', 6,
'Q', 'R', 'S', 7, 'T', 'o', 'p', 'U', 'q',
'r', 'V', 'W', 'X', 'Y', 's', 't', 'u', 8,
'v', 'Z', 'w', 'x', 'y', 9, 'z']

In something readable :), i.e. Python:

#!/usr/bin/python
from __future__ import print_function
import random
import itertools
import string

def mix_lists(lists):
  id_pairs = zip(lists, range(len(lists))) 
  print(id_pairs)
  id_lists = [[id] * len(list) for (list, id) in id_pairs]
  print(id_lists)
  id_list = [id for sub_list in id_lists for id in sub_list]
  print(id_list)
  random.shuffle(id_list)
  print(id_list)
  iterators = [iter(list) for list in lists]
  mixed_lists = [iterators[id].next() for id in id_list]
  return mixed_lists

lower_case = 
upper_case = 
digits = range(10)
lists = [lower_case, upper_case, digits]

print(lists)

for i in range(1):
  print(mix_lists(lists))

Flask/Gevent/Gunicorn on Xubuntu with Upstart

Flask

Flask is a Python web micro-framework. I’ll not go into details too much, you can read more in the docs if you are interested. What I’m going to use it for is to make a small Flask-based (and thus Python-based) test bed.

Flask installation

Here are the steps needed to get Flask up and running on Xubuntu 12.04. It should be straightforward on other systems, too, but bear in mind there might be differences.

On Xubuntu systems, Python is installed by default – on 12.04 you have this version:

python 2.7.3-0ubuntu2

If you don’t have it for some reason:

sudo apt-get install python

I prefer to use virtualenv to manage python environments. A handy extension is called virtualenvwrapper, so let’s install it:

$ sudo apt-get install virtualenvwrapper 

You’ll notice when you run the above that you also install some other packages:

The following extra packages will be installed:
  libjs-sphinxdoc libjs-underscore python-pip python-setuptools python-virtualenv

The important ones for our story are the python-* ones:

  • pip is a python package manager
  • setuptools is used by pip
  • virtualenv is of course the virtual environment manager I mentioned above

Now, one note for Xubuntu. When you install virtualenvwrapper package, it installs some docs here:

/usr/share/doc/virtualenvwrapper/en/html/index.html

which is similar to this online doc page. In both of these, it says that you should do something like this:

$ pip install virtualenvwrapper
...
$ export WORKON_HOME=~/Envs
$ mkdir -p $WORKON_HOME
$ source /usr/local/bin/virtualenvwrapper.sh
$ mkvirtualenv env1
Installing
...

This actually will not work in Xubuntu, at least in the current version that I have installed, which is 2.11.1-2. The thing is, there’s a small file here:

/usr/share/doc/virtualenvwrapper/README.Debian

that explains Debian differences. What is happening here is that Debian package actually puts the file in this location:

/etc/bash_completion.d/virtualenvwrapper

and is intended to be used with bash-completion package. This works fine for me, but if it doesn’t work for you, you can always manually source this file as explained in the docs (and also the above README.Debian file):

$ source /etc/bash_completion.d/virtualenvwrapper

and you are all set to go.

Note that in either case you need to set up virtualenvwrapper – this is explained in /etc/bash_completion.d/virtualenvwrapper, here’s how I have it configured:

$ mkdir $HOME/.virtualenvs
$ echo 'export WORKON_HOME=$HOME/.virtualenvs' >> ~/.bashrc

You need to start a new bash session to pick up the .bashrc config – it will do something like this:

$ bash
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/initialize
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/premkvirtualenv
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/postmkvirtualenv
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/prermvirtualenv
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/postrmvirtualenv
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/predeactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/postdeactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/preactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/postactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/get_env_details
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/premkproject
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/postmkproject
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/prermproject
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/postrmproject

and you should be good to go.

Let’s start a sample python environment:

$ mkvirtualenv default
New python executable in default/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/default/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/default/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/default/bin/preactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/default/bin/postactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/default/bin/get_env_details

Note that whenever you have a fresh bash run, no virtualenv will be initialized. Do this if so:

$ workon default

This will initialized the virtualenv we aptly named “default”.

Finally, let’s install Flask and its dependencies:

$ pip install flask
Downloading/unpacking flask
  Downloading Flask-0.9.tar.gz (481Kb): 481Kb downloaded
  Running setup.py egg_info for package flask
...
Cleaning up...

Let’s do a quick test that all is OK. Following [this guide] from Flask itself, do this:

$ cat >flask_hello_world.py
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello World!'

if __name__ == '__main__':
    app.run()
^D
$ python flask_hello_world.py 
 * Running on http://127.0.0.1:5000/

(Note for those not familiar – ^D above means press Ctrl+D to send EOF to cat). Browse to http://localhost:5000 and you should see “Hello World!” written to confirm Flask is installed and all OK.

Flask + gevent + gunicorn

Whenever you restart the machine you are working on, to run a Flask application, you need to:

  • Start your terminal (usually a bash session these days)
  • Initialize a Python virtualenv (“default” in our case) by running “workon default”
  • Run “python application-name.py”

This can be scripted, but you at least need to do something manually. If you are developing an application that should be running all the time, you obviously don’t want to have manual steps. This can be solved in different ways, such as:

  • Run the server on startup (e.g. using upstart)
  • Run it when the user logs in (e.g. Xubuntu has a convenient Session and Startup app in Session Manager)
  • Use xinetd
  • Use a standalone server, such as the ones described in Flask’s deployment options page

I opted for the last one. It requires some sort of communication between the server and the Flask application. When it’s for Python applications, WSGI is typically used for this purpose. I found this page describing different WSGI servers and measuring their performance in depth. It’s a couple of years old, but it’s a great write up nonetheless. The conclusion was that gevent is the best option. Obviously, far from that I need it in this case, but it’s a good exercise for the future.

Flask’s deployment page also has a gevent setup part. To make a gevent front-end for our hello world application, put this in a file called flask_hello_world_gevent.py in the same folder where flask_hello_world.py lies:

from gevent.wsgi import WSGIServer
from flask_hello_world import app

http_server = WSGIServer(('', 5000), app)
http_server.serve_forever()

For the above to work, you’ll have to install libevent-dev and gevent:

$ sudo apt-get install libevent-dev
...
$ pip install gevent
...

Now you can run it:

$ python flask_hello_world_gevent.py 
127.0.0.1 - - [2012-07-14 16:41:40] "GET / HTTP/1.1" 200 12 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1"

The above shows the first request, too. You should get the familiar “Hello World!” text at this point by going to http://localhost:5000/.

The problem with the above is that it’s not really a server – you still need to run it via python. Not that this cannot be scripted, but there’s a better option – use Gunicorn with gevent. So install it:

$ pip install gunicorn

and start it:

$ gunicorn flask_hello_world:app -kgevent
2012-07-14 17:46:12 [17983] [INFO] Starting gunicorn 0.14.5
2012-07-14 17:46:12 [17983] [INFO] Listening at: http://127.0.0.1:8000 (17983)
2012-07-14 17:46:12 [17983] [INFO] Using worker: gevent
2012-07-14 17:46:12 [17986] [INFO] Booting worker with pid: 17986

Again, you should get the overly familiar “Hello World!” text, only now by going to http://localhost:8000/. Type “gunicorn -h” for a full page of options gunicorn supports, some of which can be used for controlling things such as the number of worker processes, bind address / port, logging files, etc.

Starting up applications with upstart

Upstart is useful when you have a need to manage process lifecycle, i.e. start / stop on boot or some events. It’s a good solution for having our Gunicorn server auto-started when the machine starts. As of version 1.3, Upstart supports user-level services and as of version 1.4 it supports setuid / setgid, which will be useful here to set the user / group. Let’s create a service called gunicorn:

$ cat|sudo tee /etc/init/gunicorn.conf
setuid icyrock.com
setgid icyrock.com
script
  export HOME=/home/icyrock.com
  . $HOME/.virtualenvs/default/bin/activate
  cd $HOME/web/flask
  gunicorn flask_hello_world:app -kgevent
end script
^D

Now we can start and stop the server easily with start / stop commands and get its status with status:

$ sudo start gunicorn
gunicorn start/running, process 18790
$ status gunicorn
gunicorn start/running, process 18790
$ sudo stop gunicorn
gunicorn stop/waiting
$ status gunicorn
gunicorn stop/waiting

After “start unicorn”, you should get now annoyingly familiar “Hello World!” text at http://localhost:8000/.


Sample CouchDB curl session

CouchDB has a lot of documentation in its wiki. A good overview of how to access it directly through HTTP is at this wiki page. When starting learning CouchDB I found it quite succinct. This is great after you are “in business” and just need to confirm something or refresh your memory, but I think it’s hard for the first couple of sessions.

The below sample is a sample interaction that shows:

  • Using pure curl access – the basics
  • The following operations:
    • Creating a database
    • Deleting a database
    • Creating documents
    • Creating design documents
    • Which implies updating documents, as design documents are nothing but regular documents with a special name
    • Creating views
    • Updating views
    • Getting view results
  • Python curl is used for simplicity – this could have been done through command line curl or using some other programming language
  • Actual HTTP calls if enabled via
    • conn.set_debuglevel(1) setting (see code)
  • Most importantly – how much easier is to use some of the libraries (such as these) that wrap all this in simpler calls, after you grasp the basics

Some important notes:

  • Be sure to change the parameters at the start of the script if they are different
  • Important: this is destructive – it will ruin the given database, so be careful
  • For the above reason, I put the ‘raise Exception’ statement – comment it out to run the code

Here’s the code:

#!/usr/bin/python

# License: MIT (http://www.opensource.org/licenses/mit-license.php)
#
# Copyright (c) 2011. icyrock.com
# 
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation files
# (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
# 
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
# BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

# --------------------------------------------
# Sample plain HTTP Python <-> CouchDB session
# --------------------------------------------
#
# This sample python script is a small curl-based interaction with
# CouchDB. When you run it, you should be able to see what you can
# actually do using low-level curl interface that CouchDB supports. If
# you uncomment the following line:
#
# #conn.set_debuglevel(1)
#
# you will see pure HTTP calls that are actually being made. Looking at
# these you should get a better understanding about how CouchDB works
# if you take out all the libraries that you might use and which cover
# the details of what actually happens beneath.
#
# Of course, after you have this all figured out, go and use the
# library - it's easier that way. Take a look at this page for more
# info: http://wiki.apache.org/couchdb/Getting_started_with_Python

import httplib
import urllib2
import json
from textwrap import dedent

# Some defaults, change to match your system
# !!! Note the database dbname will be destroyed !!!
base_url = 'localhost:5984'
dbname = 'robocouch'
docprefix = 'rcd'
raise Exception('Comment out after confirming the above parameters')

# Some helpers
class PropertyDict(dict):
  def __init__(self, **items):
    self.__dict__.update(items)
    for k in items:
      self[k] = items[k]

conn = httplib.HTTPConnection(base_url)
# If you want to see the actual HTTP calls made by httplib, uncomment the below
# Very useful to grasp the low level stuff
#conn.set_debuglevel(1)

def couchdb_http_action(method, url, body=None):
  if body != None:
    json_send = json.dumps(body)
    headers = {'Content-Type': 'application/json'}
    conn.request(method, '/%s' % url, json_send, headers)
  else:
    conn.request(method, '/%s' % url)
  resp = conn.getresponse()
  raw = resp.read()
  info = PropertyDict(
    status=resp.status,
    raw=raw,
    data=json.loads(raw),
  )
  return info

# Helper methods - these basically do a HTTP call as given the name 
# For example, hget = HTTP GET
def hget(url):
  return couchdb_http_action('GET', url)

def hpost(url, data):
  return couchdb_http_action('POST', url, data)

def hdelete(url):
  return couchdb_http_action('DELETE', url)

def hput(url, data=None):
  return couchdb_http_action('PUT', url, data)

def pretty(data):
#  return json.dumps(data, sort_keys=True, indent=2)
  return json.dumps(data, indent=2)

#
# Let's get to work 
#
print('Welcome!')

# Create our database, delete if currently existing
print('Our database: %s' % dbname)
alldbs = hget('_all_dbs').data
print('All databases: %s' % alldbs)

if dbname in alldbs:
  print('Our database %s is present, deleting' % dbname)
  hdelete(dbname)
  print('Deleted %s, all databases: %s' % (dbname, hget('_all_dbs').data))

hput(dbname)
print('Created database %s, all databases: %s' % (dbname, hget('_all_dbs').data))

# Create some documents
for i in range(1, 6):
  docid = '%s%d' % (docprefix, i) 
  print('Creating document %s' % docid)
  hput('%s/%s' % (dbname, docid), dict(number=i*33))
#print('All document IDs in %s: %s' % (dbname, [row['id'] for row in hget('%s/_all_docs' %dbname).data['rows']]))
print('All document IDs in %s: %s' % (dbname, hget('%s/_all_docs' %dbname).raw))

# Create a design document with the same name as our database (defined by dbname variable)
# It will contain a single view called 'all_numbers'
# This view will have only a map part and will return all documents with 'number' attribute
hput('%s/_design/%s' % (dbname, dbname), dict(
    language='javascript',
    views=dict(
      all_numbers=dict(
        map=dedent('''
               function(doc) {
                 if(doc.number) {
                   emit(null, doc)
                 }
               }
            ''')
      ),
    )
))

# Actually run the view
print("All documents with 'number' attribute:")
#print(pretty(hget('%s/_design/%s/_view/all_numbers' % (dbname, dbname)).data))
print(hget('%s/_design/%s/_view/all_numbers' % (dbname, dbname)).raw)
 
# Update the design document by adding another view called count_sum_numbers
# Since this is an update, we need to fetch to get the current revision of the document
# The revision is part of our JSON outbound request
# This one will sum all documents which have 'number' attribute and give us their count
# Note that the design document is a whole, thus our previous view needs to be preserved
rev = hget('%s/_design/%s' % (dbname, dbname)).data['_rev']
print('Updating revision %s' % rev)
hput('%s/_design/%s' % (dbname, dbname), dict(
    _rev=rev,
    language='javascript',
    views=dict(
      all_numbers=dict(
        map='function(doc) { if(doc.number) emit(null, doc) }',
      ),
      count_sum_numbers=dict(
        map='function(doc) { if(doc.number) emit(null, doc.number) }',
        reduce=dedent('''
                  function(key, values, rereduce) {
                    var sum = 0
                    var count = 0
                    for(var i = 0; i < values.length; i++) {
                      sum += values[i]
                      count += rereduce ? values.count : 1
                    }
                    return {count:count, sum:sum}
                  }
               ''')
      ),
    )
))

# Get the sum of all numbers
# Rows contains one row per key. Since we had only one (null) key, we have only one row
res = hget('%s/_design/%s/_view/count_sum_numbers' % (dbname, dbname)).data['rows'][0]['value']
print('Sum of all %d available numbers: %d' % (res['count'], res['sum']))

Python list comprehensions

Python has a succinct way of representing lists:

>>> a = [1, 2, 3]

is a list of first three natural numbers. It is often necessary to do something with all elements of the list, to transform them in some way. This can be done, among other ways, by using list comprehensions ([[http://docs.python.org/tutorial/datastructures.html|docs]]):

>>> [z * z for z in a]
[1, 4, 9]

What we have done with the above is create a list of squares of the elements in the original list.

[z * z for z in a]
is saying: make a new list by going through all the elements of list a and making a new element of the new list by doing z * z for each element z of the original list. Very useful.

You can do even better then this:

>>> a = [90, 80, 70]
>>> b = [1, 2, 3]
>>> [x + y for x in a for y in b]
[91, 92, 93, 81, 82, 83, 71, 72, 73]

What the above does is does a double list comprehension – it adds the numbers in the list a with each of the numbers of the list b, again creating a new list.

What is important about the above is the order of operations. You can see that it first goes with the outermost list (

for y in b
) and then with the innermost list (
for x in a
), something that you might find counterintuitive (at least I did – I thought it would go through the lists left-to-right). So, it first says – OK, let’s take one element from the innermost list (list a) and make x be that element. Then, go to the next list to the right (in our case, list b) and work in the same way with it. That is, bind it’s first element to y. Since there are no more lists, process the instruction given (in our case, x + y) and go to the next element of the outermost list – so, bind the second element of list b to y. Continue until there are no more elements in y. Thus, in the first step, we will have the following:

||x||y||
|90|1|
|90|2|
|90|3|

Since there are no more elements in list b, go back one comprehension to the left (i.e. go to the previous inner comprehension), change its binding variable (x) to the next from its list (80) and repeat by going to the next outer comprehension (for y in b). The second step would be:

||x||y||
|80|1|
|80|2|
|80|3|

Repeat this again and again, until all lists are exhausted. The whole picture would be:

||x||y||
|90|1|
|90|2|
|90|3|
|80|1|
|80|2|
|80|3|
|70|1|
|70|2|
|70|3|

A couple weeks ago I ran into the above construct which has been used along these lines:

>>> a = [(1, 2), (3, 4), (5, 6)]
>>> [b for c in a for b in c]

In this example, the list comprehensions are being shared – for b in c is actually using c from it’s previous comprehension for c in a. I say previous based on the above logic – previous means some of the comprehensions that are inner-bound. Given the above, here’s what would happen:

||c||b||
|(1, 2)|1|
|(1, 2)|2|
|(3, 4)|3|
|(3, 4)|4|
|(5, 6)|5|
|(5, 6)|6|

Again, c is the inner comprehension, so for each of the values in it (and remember – c is just looping through all the elements in a) you will go through the whole cycle for all outer comprehensions. In our case, there’s only one comprehension – for b in c. So, for each c bound to each of the elements in the list a, you will go through all the elements in c, bind them to b and do something to them – in our case, just copy them.

The output of the above would be:

>>> a = [(1, 2), (3, 4), (5, 6)]
>>> [b for c in a for b in c]
[1, 2, 3, 4, 5, 6]

This is a very neat way to “unwrap” the list of lists or, as in this case, list of tuples.

Of course, you can go deeper then that. If you remember that strings are also iterable, you can do this:

>>> a = [("abc", "def"), ("geh", "ijk")]
>>> [d for b in a for c in b for d in c]
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'e', 'h', 'i', 'j', 'k']