My Adventures in Coding

January 13, 2011

PyMongo-Frisk – Now supports Replica Sets

Filed under: MongoDB,PyMongo,Python — Brian @ 12:35 am
Tags: , , , , ,

Yep, we have updated the open source project PyMongo-Frisk (0.0.4) to now support MongoDB Replica Sets! Our library extends the PyMongo Connection class and adds an additional method called “check_health()”. This method when called returns who is the master, a list of all slave nodes, verifies read and write connectivity with the master, and finally verifies it can read from ALL slaves in the Replica Set. The intended usage for this connection class extension would be an internal monitoring page that is pinged regularly by an automated monitor and sets off an alarm if anything fails.

We created this project because we wanted to ensure that our app server had the ability to reach all failover slave nodes. We did not want to wait until a failure of the master occurred to find out the app server could not reach one of our slaves!

To use our extension of the connection class:

from pymongo_frisk import FriskConnection as Connection
connection = Connection(['host1','host2','host3'])
results = connection.check_health()

When called the method returns a dictionary containing info about the health of ALL nodes in a Replica Set:
{'db_master_host': 'host1:27017',
'db_slave_hosts': ['host2:27018','host3:27019'],
'db_master_can_write': True,
'db_master_can_read': True,
'db_slaves_can_read': [('host2',True),('host3',True)] }

Note, the library still supports Replica Pairs using authentication. We have moved on to using version 1.6+ of MongoDB, so we decided to make the jump recently from Replica Pairs to Replica Sets. Replica Pairs will be removed in MongoDB version 1.8.

Advertisements

January 2, 2011

Creating a REST API in Python using Bottle and MongoDB

Filed under: MongoDB,Python,REST — Brian @ 10:15 pm
Tags: , , , , , ,

I have been using Bottle and MongoDB for a REST API project for almost a year now. I frequently get asked the question “Bottle, what is Bottle, I have never heard of it” and “Ok, I have read the Bottle tutorial, it is simple, but how do I use Bottle with MongoDB?”. So I decided to write up a simple “Getting Started” tutorial for my friends curious about these technologies.

Prerequisites: Python 2.7 or higher

Setup MongoDB

MongoDB is a document store, a schema less database. What makes MongoDB unique (and why we decided to use it) is that MongoDB is a hybrid document store. What this means is that MongoDB offers the freedom of storing data in a schema less fashion, while still providing a flexible query syntax that will make anyone familiar with SQL feel comfortable.

You will need to download the latest version of MongoDB from the downloads page. Let’s download MongoDB and get it up and running.

First, setup the data folder. By default MongoDB stores all data files in the folder /data/db, so let’s create the folder:

sudo mkdir -m 777 -p /data/db

Download MongoDB:

wget http://fastdl.mongodb.org/osx/mongodb-osx-x86_64-1.6.5.tgz
tar -zxvf mongodb-osx-x86_64-1.6.5.tgz

Now let’s start MongoDB:

cd mongodb-osx-x86_64-1.6.5/bin
./mongod

NOTE: We have not created a database, or even a collection (you can think of a collection as a schema free table). When a document is being saved, MongoDB will lazily create the collection if it does not exist, and if the database does not exist, it will also create the database. It’s just that simple!

If you would like a more detailed introduction to using MongoDB, the query syntax, etc, please refer to my previous post: Getting Started with MongoDB

Setup Bottle

Bottle is a very simple Python web framework that is contained in a single Python file (bottle.py). We have been using Bottle for creating REST APIs and have been very impressed with just how easy it is to use.

Install the Bottle package into your Python site-packages folder:

pip install bottle

Setup PyMongo

PyMongo is a library for interacting with MongoDB from a Python application. This package bridges the gap between Bottle and MongoDB.

Install the PyMongo package into your Python site-packages folder:

pip install pymongo

Putting it all together: A Simple REST API

Now, on to the fun stuff! Let’s write a simple Python REST API, with a GET and a PUT, using Bottle, that allows us to save and retrieve documents to and from MongoDB using PyMongo.

Create a new file:

touch myrestapi.py

Cut and paste the following code:

import json
import bottle
from bottle import route, run, request, abort
from pymongo import Connection

connection = Connection('localhost', 27017)
db = connection.mydatabase

@route('/documents', method='PUT')
def put_document():
	data = request.body.readline()
	if not data:
		abort(400, 'No data received')
	entity = json.loads(data)
	if not entity.has_key('_id'):
		abort(400, 'No _id specified')
	try:
		db['documents'].save(entity)
	except ValidationError as ve:
		abort(400, str(ve))
	
@route('/documents/:id', method='GET')
def get_document(id):
	entity = db['documents'].find_one({'_id':id})
	if not entity:
		abort(404, 'No document with id %s' % id)
	return entity

run(host='localhost', port=8080)

Start the application:

python myrestapi.py

Using our new REST API

Now let’s try saving and retrieving some documents. If you do not have a preferred REST client handy and are using the Google Chrome browser, you should try the extension Simple REST Client.

Save a document
  • Open up your favorite REST client and point it at the following URL:
  • Now select PUT and enter the following data:
    • {"_id": "doc1", "name": "Test Document 1"}

NOTE: The primary key of a document in a MongoDB collection is always the field “_id”. If you specify an “_id” in your document it will be used, if you do not, MongoDB will generate a unique id.

Retrieve a document
  • In your REST client enter the URL for retrieving the document we just saved:
  • Now select GET and you should see the same document returned:
    • {"_id": "doc1", "name": "Test Document 1"}

That is it! You now have a basic, working, REST API written in Python using Bottle and MongoDB!

Create a free website or blog at WordPress.com.