Flask/Gevent/Gunicorn on Xubuntu with Upstart

Flask

Flask is a Python web micro-framework. I’ll not go into details too much, you can read more in the docs if you are interested. What I’m going to use it for is to make a small Flask-based (and thus Python-based) test bed.

Flask installation

Here are the steps needed to get Flask up and running on Xubuntu 12.04. It should be straightforward on other systems, too, but bear in mind there might be differences.

On Xubuntu systems, Python is installed by default – on 12.04 you have this version:

python 2.7.3-0ubuntu2

If you don’t have it for some reason:

sudo apt-get install python

I prefer to use virtualenv to manage python environments. A handy extension is called virtualenvwrapper, so let’s install it:

$ sudo apt-get install virtualenvwrapper 

You’ll notice when you run the above that you also install some other packages:

The following extra packages will be installed:
  libjs-sphinxdoc libjs-underscore python-pip python-setuptools python-virtualenv

The important ones for our story are the python-* ones:

  • pip is a python package manager
  • setuptools is used by pip
  • virtualenv is of course the virtual environment manager I mentioned above

Now, one note for Xubuntu. When you install virtualenvwrapper package, it installs some docs here:

/usr/share/doc/virtualenvwrapper/en/html/index.html

which is similar to this online doc page. In both of these, it says that you should do something like this:

$ pip install virtualenvwrapper
...
$ export WORKON_HOME=~/Envs
$ mkdir -p $WORKON_HOME
$ source /usr/local/bin/virtualenvwrapper.sh
$ mkvirtualenv env1
Installing
...

This actually will not work in Xubuntu, at least in the current version that I have installed, which is 2.11.1-2. The thing is, there’s a small file here:

/usr/share/doc/virtualenvwrapper/README.Debian

that explains Debian differences. What is happening here is that Debian package actually puts the file in this location:

/etc/bash_completion.d/virtualenvwrapper

and is intended to be used with bash-completion package. This works fine for me, but if it doesn’t work for you, you can always manually source this file as explained in the docs (and also the above README.Debian file):

$ source /etc/bash_completion.d/virtualenvwrapper

and you are all set to go.

Note that in either case you need to set up virtualenvwrapper – this is explained in /etc/bash_completion.d/virtualenvwrapper, here’s how I have it configured:

$ mkdir $HOME/.virtualenvs
$ echo 'export WORKON_HOME=$HOME/.virtualenvs' >> ~/.bashrc

You need to start a new bash session to pick up the .bashrc config – it will do something like this:

$ bash
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/initialize
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/premkvirtualenv
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/postmkvirtualenv
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/prermvirtualenv
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/postrmvirtualenv
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/predeactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/postdeactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/preactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/postactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/get_env_details
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/premkproject
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/postmkproject
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/prermproject
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/postrmproject

and you should be good to go.

Let’s start a sample python environment:

$ mkvirtualenv default
New python executable in default/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/default/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/default/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/default/bin/preactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/default/bin/postactivate
virtualenvwrapper.user_scripts creating /home/icyrock.com/.virtualenvs/default/bin/get_env_details

Note that whenever you have a fresh bash run, no virtualenv will be initialized. Do this if so:

$ workon default

This will initialized the virtualenv we aptly named “default”.

Finally, let’s install Flask and its dependencies:

$ pip install flask
Downloading/unpacking flask
  Downloading Flask-0.9.tar.gz (481Kb): 481Kb downloaded
  Running setup.py egg_info for package flask
...
Cleaning up...

Let’s do a quick test that all is OK. Following [this guide] from Flask itself, do this:

$ cat >flask_hello_world.py
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello World!'

if __name__ == '__main__':
    app.run()
^D
$ python flask_hello_world.py 
 * Running on http://127.0.0.1:5000/

(Note for those not familiar – ^D above means press Ctrl+D to send EOF to cat). Browse to http://localhost:5000 and you should see “Hello World!” written to confirm Flask is installed and all OK.

Flask + gevent + gunicorn

Whenever you restart the machine you are working on, to run a Flask application, you need to:

  • Start your terminal (usually a bash session these days)
  • Initialize a Python virtualenv (“default” in our case) by running “workon default”
  • Run “python application-name.py”

This can be scripted, but you at least need to do something manually. If you are developing an application that should be running all the time, you obviously don’t want to have manual steps. This can be solved in different ways, such as:

  • Run the server on startup (e.g. using upstart)
  • Run it when the user logs in (e.g. Xubuntu has a convenient Session and Startup app in Session Manager)
  • Use xinetd
  • Use a standalone server, such as the ones described in Flask’s deployment options page

I opted for the last one. It requires some sort of communication between the server and the Flask application. When it’s for Python applications, WSGI is typically used for this purpose. I found this page describing different WSGI servers and measuring their performance in depth. It’s a couple of years old, but it’s a great write up nonetheless. The conclusion was that gevent is the best option. Obviously, far from that I need it in this case, but it’s a good exercise for the future.

Flask’s deployment page also has a gevent setup part. To make a gevent front-end for our hello world application, put this in a file called flask_hello_world_gevent.py in the same folder where flask_hello_world.py lies:

from gevent.wsgi import WSGIServer
from flask_hello_world import app

http_server = WSGIServer(('', 5000), app)
http_server.serve_forever()

For the above to work, you’ll have to install libevent-dev and gevent:

$ sudo apt-get install libevent-dev
...
$ pip install gevent
...

Now you can run it:

$ python flask_hello_world_gevent.py 
127.0.0.1 - - [2012-07-14 16:41:40] "GET / HTTP/1.1" 200 12 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1"

The above shows the first request, too. You should get the familiar “Hello World!” text at this point by going to http://localhost:5000/.

The problem with the above is that it’s not really a server – you still need to run it via python. Not that this cannot be scripted, but there’s a better option – use Gunicorn with gevent. So install it:

$ pip install gunicorn

and start it:

$ gunicorn flask_hello_world:app -kgevent
2012-07-14 17:46:12 [17983] [INFO] Starting gunicorn 0.14.5
2012-07-14 17:46:12 [17983] [INFO] Listening at: http://127.0.0.1:8000 (17983)
2012-07-14 17:46:12 [17983] [INFO] Using worker: gevent
2012-07-14 17:46:12 [17986] [INFO] Booting worker with pid: 17986

Again, you should get the overly familiar “Hello World!” text, only now by going to http://localhost:8000/. Type “gunicorn -h” for a full page of options gunicorn supports, some of which can be used for controlling things such as the number of worker processes, bind address / port, logging files, etc.

Starting up applications with upstart

Upstart is useful when you have a need to manage process lifecycle, i.e. start / stop on boot or some events. It’s a good solution for having our Gunicorn server auto-started when the machine starts. As of version 1.3, Upstart supports user-level services and as of version 1.4 it supports setuid / setgid, which will be useful here to set the user / group. Let’s create a service called gunicorn:

$ cat|sudo tee /etc/init/gunicorn.conf
setuid icyrock.com
setgid icyrock.com
script
  export HOME=/home/icyrock.com
  . $HOME/.virtualenvs/default/bin/activate
  cd $HOME/web/flask
  gunicorn flask_hello_world:app -kgevent
end script
^D

Now we can start and stop the server easily with start / stop commands and get its status with status:

$ sudo start gunicorn
gunicorn start/running, process 18790
$ status gunicorn
gunicorn start/running, process 18790
$ sudo stop gunicorn
gunicorn stop/waiting
$ status gunicorn
gunicorn stop/waiting

After “start unicorn”, you should get now annoyingly familiar “Hello World!” text at http://localhost:8000/.


Building Emacs from source on Xubuntu 12.04

Emacs is an extensible text editor written mostly in Emacs Lisp (or elisp for short). It’s quite old – the first version was released in 1976. according to Wikipedia article about Emacs. It’s, however, still regularly updated – the last stable version as of this writing is 23.4 released on January 29th, 2012.

One problem with Emacs is exactly its extensibility and customizability. It’s updated very often, so the last stable release is, well, unusable. All the extensions and testing is usually done on the development branch – currently, the preview release is 24.1-rc, released just yesterday, June 1st, 2012. In the current Xubuntu repository, the current version is present:

Version: 23.3+1-1ubuntu9

In order to compensate for this, I (and I suppose many other people) install directly from source. Here’s how to easy do this using the git mirror repository.

Steps for building Emacs

  • Build packages

In order to build emacs, you’ll need some packages installed that don’t come pre-installed on a fresh Xubuntu install. In case you did not, install these:

$ sudo apt-get install autoconf automake build-essential libjpeg-dev libgif-dev libncursesw5-dev libpng-dev libtiff4-dev texinfo
  • Clone the git mirror of Emacs

On this Emacs Wiki page – called Emacs From Git – you have several git repositories listed. Pick one, e.g. the one on gnu.org, and clone it somewhere:

$ git clone git://git.savannah.gnu.org/emacs.git

This is going to take a while, so be patient.

  • Build Emacs

Emacs uses the standard Linux configure scripts and make command. To build it, enter the newly cloned git repository and issue configure / make:

$ cd emacs
$ ./autogen.sh
$ ./configure
$ make

The above assumes that you want to install emacs in /usr/local – change that if you have other preferences here. To do that, run configure like this instead:

$ ./configure --prefix=/my/emacs/path

The build step will likely take even more then git cloning, so grab a few coffees or something.

  • Installing

After compiling Emacs, you can now install it:

$ sudo make install

For for a custom folder setup:

$ sudo make install --prefix=/my/emacs/path
  • Test run

Run emacs to confirm the version:

$ emacs --version
GNU Emacs 24.1.50.1
Copyright (C) 2012 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.

All set.


Xubuntu – moving windows between monitors

I just installed Xubuntu 12.04. I like it so far and am getting used to differences (was on Ubuntu 11.10 before). One thing that I had with Ubuntu by default was Compiz. It has a lot of plugins, one of which I used a lot was Compiz Put plugin. I could not find a similar thing in Xfce – there are shortcuts to move between workspaces, but not between the monitors that belong to one workspace.

Now I searched over the net and found some useful things – one of them is xdotool. It’s a nice utility that allows you to perform different operations on available X windows, such as move, hide, maximize, etc. Xfce has keyboard shortcuts readily available – Main menu / Settings / Settings Manager, Keyboard, Application Shortcuts tab. These allow you to bind a keyboard shortcut and run any application, including bash scripts, with it. Exactly what I need here – make a xdotool script that moves the window left or right, depending on the current position.

Searching for a window

To do something to a window, you first search for it. You can do this using the following command:

wid=`xdotool search --name Calculator|head -1`

The above will yield the window ID of the first found window that has “Calculator” in the default set of search categories (window class, window class name or window name / title).

Displaying the names of all windows

In case you don’t know the name of the window – for example, if you turned off the window decorations – you can use this script to show them:

wids=`xdotool search --name ""`
for wid in $wids; do
  wname=`xdotool getwindowname $wid`
  if [ -n "$wname" ]; then
    echo $wid --- $wname
  fi
done

which gives the output such as:

16777220 --- xfce4-panel
50331652 --- xfce4-verve-plugin
54525956 --- xfce4-cpugraph-plugin
37748740 --- xfce4-netload-plugin
52428804 --- xfce4-systemload-plugin
58720260 --- xfce4-xkb-plugin
56623108 --- wrapper
...

One thing you’ll notice is that it’s quite slow – that’s because it needs to spawn a shell and a xdotool instance for each of the window IDs it found in the first line.

Moving windows

Let’s say we wanted to move our calculator instance 10px to the left, just for fun. What we need to do is:

  • Find the window ID of the calculator
  • Find it’s geometry (position and size)
  • Calculate the new value for position
  • Move the window there

Here’s a script that does these steps:

wid=`xdotool search --name Calculator|head -1`
eval `xdotool getwindowgeometry --shell $wid`
X=$(($X - 10))
xdotool windowmove $wid $X $Y

You can test this easily – just run this in a terminal while you have a calculator instance running.

Maximized windows

Now, there’s a slight problem with the above – it doesn’t work with maximized windows. Although that’s not usually a problem (i.e. maximized windows are considered stationary for regular purposes), for the purpose of moving to the other monitor this will be a problem. There’s another utility – wmctrl – that can be used for this job:

wid=`xdotool search --name Calculator|head -1`
wmctrl -ir $wid -b remove,maximized_vert,maximized_horz

Similarly, to maximize:

wid=`xdotool search --name Calculator|head -1`
wmctrl -ir $wid -b add,maximized_vert,maximized_horz

You can even toggle:

wid=`xdotool search --name Calculator|head -1`
wmctrl -ir $wid -b toggle,maximized_vert,maximized_horz

Now, we can use the third tool – xprop – to actually get the state:

wid=`xdotool search --name Calculator|head -1`
xprop -id $wid _NET_WM_STATE

This should print something like this:

$ xprop -id $wid _NET_WM_STATE
_NET_WM_STATE(ATOM) = 
$ xprop -id $wid _NET_WM_STATE
_NET_WM_STATE(ATOM) = _NET_WM_STATE_MAXIMIZED_HORZ, _NET_WM_STATE_MAXIMIZED_VERT

In the first case, the window is not maximized, in the second case it is (both vertically and horizontally).

Hiding windows

Another issue with this is – if we are to move the windows, it’s going to actually be very flickery. This is especially true if the difference between maximized and minimized versions of the window are great. So, we need to hide / show them before we do all the things. xdotool can do that:

wid=`xdotool search --name Calculator|head -1`
xdotool windowunmap $wid

hides the window. To show it:

wid=`xdotool search --name Calculator|head -1`
xdotool windowmap $wid

Unfortunately, this is not going to help due to the issue – when the window is shown (mapped), all the changes to its position / size seem to be lost.

Getting the current window

As this is going to be a keyboard shortcut, it needs to work on the current window – xdotool allows getting the current window ID like this:

wid=`xdotool getactivewindow`

Raising the window

When moved to the other monitor, the window may actually be below the other windows, depending on how their creation / switching happened, so we need to raise it – this can be done like this:

wid=`xdotool getactivewindow`
xdotool windowraise $wid

Moving the active window to the other monitor (finally!)

Here’s what we need to do:

  • Find the active window
  • Get its maximized state and remember it
  • Remove maximization
  • Get its geometry
  • Calculate the new position
  • Move it
  • Maximize based on the previous state
  • Raise it

Here’s a script that does that:

wid=`xdotool getactivewindow`
max_state=`xprop -id $wid _NET_WM_STATE`

wmctrl -ir $wid -b remove,maximized_vert,maximized_horz
eval `xdotool getwindowgeometry --shell $wid`

new_x=1600
if [[ "$X" -ge "$new_x" ]]; then
  new_x=0
fi

xdotool windowmove $wid $new_x $Y
if [ -z "${max_state/*_NET_WM_STATE_MAXIMIZED_*/}" ]; then
  wmctrl -ir $wid -b add,maximized_vert,maximized_horz
fi

xdotool windowraise $wid

Some problems

There are some problems with the above that I’ve noticed:

  • Obviously, as made works only with two horizontally located monitors side by side
  • It’s a bit slow as it spawns a lot of shells / runs a lot of commands
  • Although it’s not hard to get one, hard-coding the new_x to the half of your monitor setup speeds this up – a trade-off here
  • When you have a panel on one monitor only, moving the windows slides below the panel if they are in the same position on the other monitor