Multiple Django and Flask Sites with Nginx and uWSGI Emperor

Posted May 16, 2012

I like Python, a lot. I'm one of those people who would use it all the time if they could, especially for web development. Which is convenient, since I'm currently working on my own company.

One of the drawbacks of using Python, though, has always been getting it to play nicely with web servers. I've never been completely happy with the options out there: mod_python, mod_wsgi, FastCGI, and uWSGI. They've all had some instability, performance issue, or other problem.

But this past weekend, while doing some server upgrades, I came across a relatively new feature of uWSGI that seems to be the ideal solution for hosting Python sites: Emperor mode.

When you start uWSGI in Emperor mode, you give it a directory to watch that contains all of your uWSGI config files. If a new file is added, the emperor process reads it and spins up loyal worker processes to handle requests. If a config file is touched or modified, the emperor will gracefully restart the associated worker processes. And if a config file is removed, the emperor will kill the relevant workers.

Installation

First, you'll need to install the latest versions of Nginx and uWSGI. I'll assume an Ubuntu install, but you should be able to modify this if needed.

$ sudo apt-get install nginx uwsgi

If you're on a recent version of Ubuntu, this should get you Nginx 1.2 and uWSGI 1.04. If not, you might need to add the PPAs:

$ sudo add-apt-repository ppa:nginx/stable
$ sudo add-apt-repository ppa:uwsgi/release
$ sudo apt-get update
$ sudo apt-get install nginx uwsgi

Upstart and uWSGI

Next, you'll need an Upstart script for uWSGI:

/etc/init/uwsgi.conf

description "uWSGI"
start on runlevel [2345]
stop on runlevel [06]
respawn

env UWSGI=/usr/bin/uwsgi
env LOGTO=/var/log/uwsgi/emperor.log

exec $UWSGI --master --emperor /etc/uwsgi/apps-enabled --die-on-term --uid www-data --gid www-data --logto $LOGTO

You may need to create the appropriate directories:

$ sudo mkdir -p /var/log/uwsgi
$ sudo mkdir -p /etc/uwsgi/apps-available
$ sudo mkdir -p /etc/uwsgi/apps-enabled

Now you should be able to get uWSGI started:

sudo service uwsgi start

The emperor process's logs will go to /var/log/uwsgi/emperor.log, so if you run into any errors, check there.

Getting Flask Running

Now we need to set up the configuration for one of our sites. First we'll do a Flask site, since it covers most of what we need to know for Django, as well.

Let's assume the directory structure looks something like this:

/
  var/
    www/
      flaskapp/
        static/
        templates/
        venv/
        app.py
        models.py
        settings.py

(Where venv/ is the virtualenv for this app.)

Let's start with the uWSGI config.

/etc/uwsgi/apps-available/flaskapp.ini

[uwsgi]
# Variables
base = /var/www/flaskapp
app = app
# Generic Config
plugins = http,python
home = %(base)/venv
pythonpath = %(base)
socket = /var/www/run/%n.sock
module = %(app)
logto = /var/log/uwsgi/%n.log

Conveniently, uWSGI allows for variables in its config files. In this case, we're using three variables, %(base), %(app), and %n. We define base to be the root directory of our app, app to be the module name where the WSGI application is defined (uWSGI will expect it to be called application. If it's called something else you should specify callable = foo in the config.), and %n is already defined for us as the basename of the config file, in this case, flaskapp.

We'll enable that config in a bit, but before we do that, let's get the Nginx config ready.

/etc/nginx/sites-available/flaskapp

server {
    listen 80;
    server_name mygreatstartup.com;
    server_name www.mygreatstartup.com;
    root /var/www/flaskapp;

    location /static/ {
        alias /var/www/flaskapp/static/;
        include /etc/nginx/gzip.conf;
        expires 30d;
        access_log off;
    }

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/var/www/run/flaskapp.sock;
    }
}

That should be all we need to get set up. uWSGI and Nginx are already running, so let's enable the app and the site.

$ sudo ln -s /etc/uwsgi/apps-available/flaskapp.ini /etc/uwsgi/apps-enabled/flaskapp.ini

The uWSGI Emperor will see the new file (a symlink to the real config file), spin up new processes, and point it at that config.

Next, we enable the Nginx site.

$ sudo ln -s /etc/nginx/sites-available/flaskapp /etc/nginx/sites-enabled/flaskapp
$ sudo service nginx restart

(Nginx requires a reload to pick up config changes.)

If the DNS entries are properly configured, we should start serving our new site.

Getting Django Running

Fortunately, getting Django to run under WSGI is very easy, so we just need a couple of tweaks. First, the uWSGI config.

/etc/uwsgi/apps-available/djangoapp.ini

[uwsgi]
# Variables
base = /var/www/djangoapp
app = wsgi
# Generic Config
plugins = http,python
home = %(base)/venv
pythonpath = %(base)
socket = /var/www/run/%n.sock
module = %(app)
logto = /var/log/uwsgi/%n.log

You'll notice that the only real change is that we've called our app wsgi. This is just a convention that seems to be prevalent with Django apps; the WSGI app is put in a file called wsgi.py inside of the project directory. A simple version would look like this.

/var/www/djangoapp/wsgi.py

import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

Aside from replacing flaskapp with djangoapp in the rest of the config files above, everything else should work as specified.

Restarting Apps

If you ever need to restart a single WSGI app (Flask, Django, or anything else), which is usually necessary after any changes to the code, just run the following:

$ sudo touch /etc/uwsgi/apps-enabled/flaskapp

The emperor process will notice that the file has been touched and gracefully restart the process serving flaskapp.

That's it!

Comments are starting over at Hacker News

Update: Reader Jason emailed with a recipe for nested WSGI sites. In case you want to set something like that up, he's posted it on his blog.

You should follow me on Twitter. If you have any questions, shoot me an email.

Pull Hard

Posted May 02, 2012

As Jeff mentioned, he and I applied to Y Combinator for this summer in the hopes of getting a bit of a jump start on the new venture he and I are working on (which I alluded to in my previous post).

Back at the beginning of March, we filled out our application and made a short, introductory video about ourselves and our idea. Then we waited. The application deadline wasn't for another week or so, and then it'd be another two weeks until they got back to us.

In the meantime, I gave my two weeks notice, and notified YC that I'd left my full time job to pursue the new venture.

Three days after I finished my last day at work, I was in Arches National Park as a warmup for a bit of burnout prevention. Most of the park had spotty cell reception, but somewhere along one of my many short hikes, I got enough connection to download my email. One of them caught my eye:

XXXXXXXX@ycombinator.com
Y Combinator application
Your application looks promising and we'd like to meet you in person...

I didn't even open the email. I immediately called Jeff and told him the good news. Unfortunately, cell service was still spotty, so it took a few repetitions before the message was clearly communicated. That night, we booked our flights to Mountain View.

A week and a half later, we met in the SFO airport, got a car, and drove down to Palo Alto. We spent the afternoon preparing for our interview in the Starbucks he and I both frequented while we were students at Stanford. The next day we spent more time working in a coffee shop in Mountain View and then tried to kill an hour or so walking the length of Castro Street, asking each other questions we thought might come up during the interview.

We arrived at the Y Combinator offices about half an hour early for the interview, but we decided to wait in the car. We quizzed each other some more, and nervously joked about the clown car we had rented (a Fiat 500). We could see other teams milling about outside, also killing time before their interviews.

With fifteen minutes to go, we grabbed our stuff and went into the YC headquarters. We checked in, grabbed a bottle of water, and got our demo set up on the laptops. A few YC alums came by and introduced themselves, asking if we had any questions about the interview. Even if we did, there wasn't enough time to really talk about them. So we thanked the alums and got on with waiting.

Right at our designated interview time, one of the receptionists came and got us and brought us over to the kitchen, next to the office where we would be interviewing. We were told it'd be just a minute. Then Paul Graham came out and made a smoothie for himself, Trevor Blackwell , and Robert Morris; our interviewers.

I noticed Paul wasn't wearing any shoes.

Once they'd finished their smoothies, we were called into the office, and the interview began. The allotted ten minutes flew by quickly. When the timer went off, we hadn't even shown our demo yet. Fortunately, Paul asked if we had one, so we got a few extra minutes to show off what we'd done so far.

We left feeling like we'd been well prepared for the interview, but a bit beat up regardless. We knew we'd be getting a call (if we got in) or an email (if we didn't) in a few hours, so we went to a movie to kill some time, then to a bar to kill some brain cells, then to dinner with a few friends. The email came in the middle of dinner.

Paul Graham
yc interview
I'm sorry to say we decided not to fund you guys. You're clearly smart hackers but...

He went on to describe the reasoning behind their decision. Fortunately, it was something we knew might be an issue. What's more, he encouraged us to continue working on the project:

...because you will probably be able to make yourselves a lot of money doing it.

We were both disappointed. I'm sure I was probably a lousy dinner companion.

But at least we know our path forward. The company can be bootstrapped and we have enough runway to figure out if this thing is going to fly.

So what are we going to do?

We're going to grab our bootstraps and pull hard.

Update: Snaposit's fate is now determined. Read about it.

The Year of Change

Posted April 10, 2012

In August, I moved halfway across the country, swept out of New York by a hurricane, and landed in Denver three days later with everything I owned in the back of a truck. Over the next few months, I worked off all of the weight that I'd slowly been gaining, and more, replacing it with muscle I've never had before. In February, I became single again.

And last week, I initiated the final "big" change by giving work my two weeks notice. Friday will be my last day of gainful employment.

So what comes next?

I'm starting a new venture with a good friend. But that's for another post.

Welcome Back, Internet

Posted January 19, 2012

Internet blackout day is over. Thank goodness! We can all get back to reading our favorite news sites.

By the way, did you contact your senators and house representative to tell them that PIPA and SOPA are bad ideas?

No? Oh dear. You can't go read the internet then.

But it's okay. We still have some time. At the very least, you should gosign a petition. Even better, you can write a letter to your senators and representativetel ling them not to support SOPA/PIPA. (Don't worry if you're not sure what to say, the site I linked you to even helps you write the letter.)

Still not sure what all the hoopla is about? Is "stopping piracy" really bad? I know the names can be a bit confusing. That's okay. Salman Khan of KhanAcademy.com has a great explanation:

Stop SOPA & PIPA

Posted January 18, 2012

It's here, today is the SOPA/PIPA Blackout Strike.

While I won't be blacking out my site today, I would like to ask you to contact your members of congress today and let them know that you do not want them to pass legislation that would allow much broader censorship of the internet.

Here are some resources for learning more about SOPA/PIPA and contacting your representatives: