Sample CouchDB curl session

CouchDB has a lot of documentation in its wiki. A good overview of how to access it directly through HTTP is at this wiki page. When starting learning CouchDB I found it quite succinct. This is great after you are “in business” and just need to confirm something or refresh your memory, but I think it’s hard for the first couple of sessions.

The below sample is a sample interaction that shows:

  • Using pure curl access – the basics
  • The following operations:
    • Creating a database
    • Deleting a database
    • Creating documents
    • Creating design documents
    • Which implies updating documents, as design documents are nothing but regular documents with a special name
    • Creating views
    • Updating views
    • Getting view results
  • Python curl is used for simplicity – this could have been done through command line curl or using some other programming language
  • Actual HTTP calls if enabled via
    • conn.set_debuglevel(1) setting (see code)
  • Most importantly – how much easier is to use some of the libraries (such as these) that wrap all this in simpler calls, after you grasp the basics

Some important notes:

  • Be sure to change the parameters at the start of the script if they are different
  • Important: this is destructive – it will ruin the given database, so be careful
  • For the above reason, I put the ‘raise Exception’ statement – comment it out to run the code

Here’s the code:

#!/usr/bin/python

# License: MIT (http://www.opensource.org/licenses/mit-license.php)
#
# Copyright (c) 2011. icyrock.com
# 
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation files
# (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
# 
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
# BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

# --------------------------------------------
# Sample plain HTTP Python <-> CouchDB session
# --------------------------------------------
#
# This sample python script is a small curl-based interaction with
# CouchDB. When you run it, you should be able to see what you can
# actually do using low-level curl interface that CouchDB supports. If
# you uncomment the following line:
#
# #conn.set_debuglevel(1)
#
# you will see pure HTTP calls that are actually being made. Looking at
# these you should get a better understanding about how CouchDB works
# if you take out all the libraries that you might use and which cover
# the details of what actually happens beneath.
#
# Of course, after you have this all figured out, go and use the
# library - it's easier that way. Take a look at this page for more
# info: http://wiki.apache.org/couchdb/Getting_started_with_Python

import httplib
import urllib2
import json
from textwrap import dedent

# Some defaults, change to match your system
# !!! Note the database dbname will be destroyed !!!
base_url = 'localhost:5984'
dbname = 'robocouch'
docprefix = 'rcd'
raise Exception('Comment out after confirming the above parameters')

# Some helpers
class PropertyDict(dict):
  def __init__(self, **items):
    self.__dict__.update(items)
    for k in items:
      self[k] = items[k]

conn = httplib.HTTPConnection(base_url)
# If you want to see the actual HTTP calls made by httplib, uncomment the below
# Very useful to grasp the low level stuff
#conn.set_debuglevel(1)

def couchdb_http_action(method, url, body=None):
  if body != None:
    json_send = json.dumps(body)
    headers = {'Content-Type': 'application/json'}
    conn.request(method, '/%s' % url, json_send, headers)
  else:
    conn.request(method, '/%s' % url)
  resp = conn.getresponse()
  raw = resp.read()
  info = PropertyDict(
    status=resp.status,
    raw=raw,
    data=json.loads(raw),
  )
  return info

# Helper methods - these basically do a HTTP call as given the name 
# For example, hget = HTTP GET
def hget(url):
  return couchdb_http_action('GET', url)

def hpost(url, data):
  return couchdb_http_action('POST', url, data)

def hdelete(url):
  return couchdb_http_action('DELETE', url)

def hput(url, data=None):
  return couchdb_http_action('PUT', url, data)

def pretty(data):
#  return json.dumps(data, sort_keys=True, indent=2)
  return json.dumps(data, indent=2)

#
# Let's get to work 
#
print('Welcome!')

# Create our database, delete if currently existing
print('Our database: %s' % dbname)
alldbs = hget('_all_dbs').data
print('All databases: %s' % alldbs)

if dbname in alldbs:
  print('Our database %s is present, deleting' % dbname)
  hdelete(dbname)
  print('Deleted %s, all databases: %s' % (dbname, hget('_all_dbs').data))

hput(dbname)
print('Created database %s, all databases: %s' % (dbname, hget('_all_dbs').data))

# Create some documents
for i in range(1, 6):
  docid = '%s%d' % (docprefix, i) 
  print('Creating document %s' % docid)
  hput('%s/%s' % (dbname, docid), dict(number=i*33))
#print('All document IDs in %s: %s' % (dbname, [row['id'] for row in hget('%s/_all_docs' %dbname).data['rows']]))
print('All document IDs in %s: %s' % (dbname, hget('%s/_all_docs' %dbname).raw))

# Create a design document with the same name as our database (defined by dbname variable)
# It will contain a single view called 'all_numbers'
# This view will have only a map part and will return all documents with 'number' attribute
hput('%s/_design/%s' % (dbname, dbname), dict(
    language='javascript',
    views=dict(
      all_numbers=dict(
        map=dedent('''
               function(doc) {
                 if(doc.number) {
                   emit(null, doc)
                 }
               }
            ''')
      ),
    )
))

# Actually run the view
print("All documents with 'number' attribute:")
#print(pretty(hget('%s/_design/%s/_view/all_numbers' % (dbname, dbname)).data))
print(hget('%s/_design/%s/_view/all_numbers' % (dbname, dbname)).raw)
 
# Update the design document by adding another view called count_sum_numbers
# Since this is an update, we need to fetch to get the current revision of the document
# The revision is part of our JSON outbound request
# This one will sum all documents which have 'number' attribute and give us their count
# Note that the design document is a whole, thus our previous view needs to be preserved
rev = hget('%s/_design/%s' % (dbname, dbname)).data['_rev']
print('Updating revision %s' % rev)
hput('%s/_design/%s' % (dbname, dbname), dict(
    _rev=rev,
    language='javascript',
    views=dict(
      all_numbers=dict(
        map='function(doc) { if(doc.number) emit(null, doc) }',
      ),
      count_sum_numbers=dict(
        map='function(doc) { if(doc.number) emit(null, doc.number) }',
        reduce=dedent('''
                  function(key, values, rereduce) {
                    var sum = 0
                    var count = 0
                    for(var i = 0; i < values.length; i++) {
                      sum += values[i]
                      count += rereduce ? values.count : 1
                    }
                    return {count:count, sum:sum}
                  }
               ''')
      ),
    )
))

# Get the sum of all numbers
# Rows contains one row per key. Since we had only one (null) key, we have only one row
res = hget('%s/_design/%s/_view/count_sum_numbers' % (dbname, dbname)).data['rows'][0]['value']
print('Sum of all %d available numbers: %d' % (res['count'], res['sum']))

CouchDB and Kanso

To see how CouchDB and Kanso work together, I’m going to build a very simple application that allows you to list and show capitals of countries in the world.

For the impatient

If you just want to play or first see the app and then determine whether it’s worth reading at all, you can find the code at GitHub.

Installation

You can read about installing CouchDB here, for Ubuntu:

The important thing is that you need CouchDB 1.1+, as that’s what Kanso requires. You can use the following ppa for that:

like this:

$ sudo apt-add-repository ppa:kmpm/ppa
$ sudo apt-get update
$ sudo apt-get install couchdb

You should verify you have CouchDB 1.1+ now:

$ curl http://localhost:5984/
{"couchdb":"Welcome","version":"1.2.0a-00c37a8-git"}

Installing Kanso is outlined in Kanso tutorial, but basically boils down to:

  • Installing node.js
  • Installing node.js package manager npm
  • Installing Kanso with sudo npm install -g kanso.

For Ubuntu, you can follow the following to-the-point tutorial by Kip Lawrence for node.js and npm:

If you want to use ppas, you can check this:

I used the last method, like this:

$ sudo apt-add-repository ppa:mdz/locker
$ sudo apt-get update
$ sudo apt-get install nodejs npm
$ sudo npm install -g kanso

and a quick check that everything is in place:

$ node -v
v0.4.9
$ npm -v
1.0.18
$ kanso -v
0.0.7 

Getting started with the application

To start a new Kanso application:

$ kanso create country-capitals
OK: /home/icyrock.com/web/couchdb/kanso/country-capitals

If you list the files:

$ find -type f
./templates/welcome.html
./templates/base.html
./templates/404.html
./static/js/jquery-1.5.2.min.js
./static/js/json2.js
./kanso.json
./lib/app.js
./lib/views.js
./lib/rewrites.js
./lib/lists.js
./lib/validate.js
./lib/shows.js
./lib/updates.js
./lib/types.js
./lib/events.js
./lib/filters.js

you will see there are several important folder and files. Most of these are outlined in the Kanso tutorial, here are my comments on this:

  • You can see that jQuery version is 1.5.2 and minified. Current jQuery is 1.6.2, so this is a bit outdated and if you run into some problems where you actually need to debug, you’d probably want to substitute a non-minified version. Obviously, Kanso was designed for “ready to push to prod” kind of attitude, so this is expected
  • You already have some basic templates there – edit these to suit your needs
  • There are a lot of .js files in lib folder – this is the logic piece that should be edited by the application creator to describe what the app should actually do
  • There are a lot of CommonJS modules included – if on Ubuntu and installed as I did, take a look at /usr/lib/node_modules/kanso, especially commonjs/kanso subfolder

Data

As my data source, I used list of national capitals from Wikipedia. Kanso supports data transformations and uploads just as we need. Here’s what we are going to do:

  • For starters, I put the list from Wikipedia into a CSV file. Note the csv will have three columns: the expected two are country and capital, but the third one is a requirement of Kanso type system that will assign the type to a document. We’ll see this later, but for now just remember the type is countryCapital
  • Kanso can transform CSV into JSON with kanso transform
  • After that, we will use kanso transform add-ids to populate the IDs for each of the documents, to prevent duplication from multiple pushes
  • The next step is to create the database – curl will be just enough
  • We’ll then use kanso pushdata to push the data into CouchDB

So, do the following within your app’s root folder:

$ mkdir data
$ cd data
$ wget <file>
$ kanso transform csv country-list.csv country-list.json
$ kanso transform add-ids country-list.json country-list-with-ids.json
$ curl -X PUT http://localhost:5984/country-capitals
$ kanso pushdata http://localhost:5984/country-capitals country-list-with-ids.json

If you glance your local Futon, specifically country-capitals database, you should see all the data in there. Awesome, let’s go to the application.

Basic test

Let’s push this to our local CouchDB and test whether it works:

$ kanso push http://localhost:5984/country-capitals
Uploading...
OK: http://localhost:5984/country-capitals/_design/country-capitals/_rewrite

The url after OK is where you should point your browser after the above.

If during running the above you get the message such as:

{"error":"os_process_error","reason":"OS process timed out."}

then check that you have CouchDB 1.1+. Otherwise, you should get a greeting and a confirmation you are running a Kanso-powered app.

After confirming Kanso works as expected, let’s build our app.

Types

Kanso has a notion of types which allow for a simple mapping from CouchDB database objects to plain Javascript objects. We have only one database, country-capitals, and the objects in it have country and capital fields, both strings. Here’s how we’d map this with Kanso’s types:

var Type = require('kanso/types').Type,
    fields = require('kanso/fields');

exports.countryCapital = new Type('countryCapital', {
    fields: {
        country: fields.string(),
        capital: fields.string(),
    }
});

Let’s run through the above quickly. The first part requires the necessary CommonJS modules from Kanso – Type and fields. These are used later to define the document contents. Now we use exports to export a newly defined type – country-capitals, that references CouchDB documents in country-capitals database. Each of these have two string fields, country and capital.

The above and all code related to defining types goes to lib/types.js file. Let’s put it there, push the app again, then push the admin app like this:

$ kanso pushadmin http://localhost:5984/country-capitals
Uploading...
OK: http://localhost:5984/country-capitals/_design/admin/_rewrite/

and go to the above URL. You should see the admin interface which allows you to drill down to actual CountryCapitals instances. You should see all the instances we put previously.

Views

Views are used by CouchDB to do transformation of data, mainly to do searches and display. Views transform all documents into a list of (key, value) pairs that should suit your particular needs in solving a problem. They are indexed, the index is created on the first view use and will be updated quickly on document insert/update/delete operations. One important thing is that they are static, i.e. you cannot pass any dynamic parameters into the view. That’s why they are easy to index. The only thing that’s dynamic is the keyset restriction you can optionally add. This can greatly reduce the number of documents returned. To compare with SQL – they are a pandan of SQL views, but it’s important to realize they are not similar to SQL queries.

In our case, we’ll make two views. The first one will transform key all country/capital pairs by their first letter. Here are a few random rows from country-capitals database:

"Aruba","Oranjestad","countryCapital"
"Comoros","Moroni","countryCapital"
"Marshall Islands","Majuro","countryCapital"
"Myanmar","Naypyidaw","countryCapital"

which our view will transform into:

{key:"a", country:"Aruba",            capital:"Oranjestad"}
{key:"c", country:"Comoros",          capital:"Moroni"}
{key:"m", country:"Marshall Islands", capital:"Majuro"}
{key:"m", country:"Myanmar",          capital:"Naypyidaw"}

The other way is symmetric one, but for the capitals – here’s the other view for the above data:

{key:"m", country:"Comoros",          capital:"Moroni"}
{key:"m", country:"Marshall Islands", capital:"Majuro"}
{key:"n", country:"Myanmar",          capital:"Naypyidaw"}
{key:"o", country:"Aruba",            capital:"Oranjestad"}

The code for these two should go into lib/views.js file and would be:

exports.countriesByFirstLetter = {
    map: function(doc) {
        if(doc.type = 'countryCapital') {
            emit(doc.country.charAt(0).toLowerCase(), {
                country: doc.country,
                capital: doc.capital
            })
        }
    }
}

exports.capitalsByFirstLetter = {
    map: function(doc) {
        if(doc.type = 'countryCapital') {
            emit(doc.capital.charAt(0).toLowerCase(), {
                country: doc.country,
                capital: doc.capital
            })
        }
    }
}

You can now kanso push http://localhost:5984/country-capitals and go to:

to see these.

Templates and rewrite rules

Kanso supports Dust templates directly. We’ll need to make a template to show the results – here’s what we’ll use (this is in templates/ccindex.htm):

<h1>Country/capital index</h1>
Countries by first letter:
<ul id="countriesByFirstLetter">
  {#letters}
  <li>
    <a href="{baseURL}/countriesByFirstLetter?key=&quot;{letter}&quot;">{display}</a>
  </li>
  {/letters}
</ul>

Capitals by first letter:
<ul id="capitalsByFirstLetter">
  {#letters}
  <li>
    <a href="{baseURL}/capitalsByFirstLetter?key=&quot;{letter}&quot;">{display}</a>
  </li>
  {/letters}
</ul>

<h1>Results</h1>

<table id="results">
  <tr>
    <th>Country</th>
    <th>Capital</th>
  </tr>
  {#rows}
  <tr>
    <td>{country}</td>
    <td>{capital}</td>
  </tr>
  {/rows}
</table>

Here are a few comments about the above:

  • base_template attribute in kanso.json file in your application root defines which is the base template. By default it’s base.html and all other templates are embedded into it by supplying content attribute. It also has title attribute that you can use to set the title of the current page. I added the CSS (base.css), so the file looks like:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> 
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> 
  <head> 
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> 
    <link rel="stylesheet" href="static/css/base.css" />
    <title>{title}</title> 
  </head> 
  <body> 
    <div id="content">
      {content|s}
    </div>
    <script src="{baseURL}/static/js/jquery-1.5.2.min.js"></script>
    <script src="{baseURL}/static/js/json2.js"></script>
    <script src="{baseURL}/kanso.js"></script>
  </body>
</html>

Note that CSS is references as static/css/base.css – this is done by the rewrite rules, that’s discussed after this

  • The above template uses {#list} and {/list} to iterate over the array named list – this is used for letters and rows above. When doing this, everything in between can use the objects in the array directly. For example, the following:
    <a href="{baseURL}/countriesByFirstLetter?key=&quot;{letter}&quot;">{display}</a>

will take the current letter (from letters array) and, from the object representing the letter, extract two properties – letter and display. – Similar to the above example, there are some “global” variables, such as {baseURL}, which denotes the base of the design document rewrite root

Let’s add a bit of CSS to the above just to be nice (this is not functionally required) – this goes to static/css/base.css:

#countriesByFirstLetter li,
#capitalsByFirstLetter li {
  display: inline;
}

#countriesByFirstLetter li a,
#capitalsByFirstLetter li a {
  text-decoration: none;
  display: inline-block;
  width: 1em;
  height: 1em;
  padding: .2em;
  background: #ffee50;
  border: 1px solid #009999;
  color: black;
}

#results {
  border-collapse: collapse;
}
#results,
#results td,
#results th {
  background: #ffcc30;
  border: 1px solid #009999;
  padding: 0em 1em 0em .3em;
  width: 60em;
}
#results td {
  background: #ffee50;
}

Templates are internal Kanso’s thing and without it they are not useful. To make use of them, you make either list or show functions – these are regular CouchDB show/list functions, but are extended by Kanso to do client-side rendering if possible (== Javascript enabled). Let’s make a list function for our case – these go into lib/lists.js:

exports.ccindex = function (head, req) {
    start({code: 200, headers: {'Content-Type': 'text/html'}});

    var row, rows = [];
    while(row = getRow()) {
        rows.push(row.value)
    }

    var letters = [];
    _.each("abcdefghijlkmnopqrstuvwxyz", function(e) {
        letters.push({
            letter: e,
            display: e.toUpperCase(),
        })
    })
 
    var content = templates.render('ccindex.htm', req, {
        letters: letters,
        rows: rows,
    });
    return {title: 'Country/capital index', content: content};
};

Here’s a commentary about the above code:

  • The above exports a function called ccindex that is stored as a list function. This is exported as a regular CouchDB list function, so you can call it with e.g.:
$ curl http://localhost:5984/country-capitals/_design/country-capitals/_list/ccindex/countriesByFirstLetter?key=%22a%22

to get a list of countries whose name starts with letter a. Note that you will not be able to run within browser if Javascript is enabled, as Kanso interferes with this. The result will be a HTML document that renders ccindex.htm template

  • getRows function will get all the rows from the specified view. List functions always take a view – if you look at the above URL, it has _list/ccindex/countriesByFirstLetter. This means “use ccindex list function to display the results of the countriesByFirstLetter view”. Thus, getRows in this case would be the rows returned by countriesByFirstLetter view.

  • Variable letters is just a dummy letter holder – this could have been done to actually go through the keys and extract only the ones present. This variable is used in the template to populate the filter sections.

The last part we need to do is rewrites. These are used to tell CouchDB what to actually do. Here’s our rewrite:

module.exports = [
    {from: '/static/*', to: 'static/*'},
//    {from: '/', to: '_show/welcome'},
    {from: '/', to: '_list/ccindex/countriesByFirstLetter', query: {key: ''}},
    {from: '/countriesByFirstLetter', to: '_list/ccindex/countriesByFirstLetter'},
    {from: '/capitalsByFirstLetter', to: '_list/ccindex/capitalsByFirstLetter'},
    {from: '*', to: '_show/not_found'}
];

The above says:

  • You can see that there’s a mention of static. This is used to reference our static/css/base.css file
  • I commented out the default root that GTE’s to _show/welcome and instead remapped this to actually call the ccindex list function with a countriesByFirstLetter view. As I supplied key that has a value of an empty string (”), nothing will match, so we will get no results. This is just for defaulting purposes – I could very well have done something else, e.g. default to letter a
  • There are two more redirects to our two views through ccindex list

Recreating

I put a small script called recreate.sh that you can always use to scratch everything, upload the data and the application that you currently have. This might help you in the beginning if you get stuck and don’t know whether the issue is with the app itself or some file that you left dangling somewhere.

Conclusion

After all this, you should be able to go to

and see our small app. Again, the code is here. All is left is to relax