Python – Upgrading Python with easy_install, pip, and virtualenv on a Mac

For this tutorial I am upgrading from python 2.6 to python 2.7 on my Mac. Every time I have to upgrade Python versions, it always takes me a few minutes to remember the steps since I do this task so rarely. So here is a simple set of instructions that seem to work for me!

Install python 2.7

• Download the Python Mac dmg file from the Python Downloads page
  Just open the dmg and follow the installer as you would with any application
• You should now have Python 2.7 installed in:
  /Library/Frameworks/Python.framework/Versions/2.7/bin/python

NOTE: Even though we have installed python 2.7, your /usr/bin/python is still pointing to an older version of Python, so for example if you run easy_install, if will run from the older version of Python, and install packages into the library of the old Python installation.

So, on to step two of the install:
Let’s switch the /usr/bin/python link to point to the newest version of Python, and at the same time let’s create a soft link in the Python install folder that always points to the current install of Python. So the next time we upgrade, we will only need to change the “Current” version link.

#Create a soft link in /Library/Frameworks/Python.framework/Versions/
cd /Library/Frameworks/Python.framework/Versions/
ln -s 2.7 Current

#Switch the /usr/bin/python link to point to current python link
cd /usr/bin
rm -f python
ln -s /Library/Frameworks/Python.framework/Versions/Current/bin/python python

Install Setup Tools (this includes easy_install)

Download the egg file for your version of python (for this example, 2.7) from the Setup Tools Downloads page. To install setup tools just run the egg file like any other shell script:

sudo sh setuptools-0.6c11-py2.7.egg

Install pip (Using easy_install)

Open a new Terminal window session. You will need a new terminal session for easy_install to show up on the path.

sudo easy_install pip

Now we can install any python packages we want using pip!

Install Virtualenv (Using pip)

Now that we have pip, lets install a very important package called virtualenv!

pip install virtualenv

Thats it!

Now you will have Python 2.7 on your Mac, with easy_install, pip, and virtualenv ready to go.

Python – Getting Started with Python Mock

These examples all use the Python Mock library. This post is just a quick “Getting Started” for someone who is new to using the Python Mock library, once you get started refer to the Python Mock Documentation.

First you will need to install the mock library:

pip install -U mock

Your test file will need the following import statement:

from mock import patch, Mock

To mock a method in a class to return a specific value use @patch.object.

@patch.object(my_module.MyClass, 'my_method')
def test_my_method_shouldReturnTrue_whenMyMethodReturnsSomeValue(self, mock_my_method):
    mock_my_method.return_value=True
    some_other_class =  SomeOtherClassThatUsesMyClass()
    result = some_other_class.method_under_test()
    self.assertTrue(result)

To mock a method in a class with @patch.object but return a different value each time it is called, use side_effect. Side effect allows you to define a custom method and have that method called each time your mock method is called. The value returned from this method will be used as the return value your mock method.

@patch.object(my_module.MyClass, 'my_method')
def test_my_method_shouldReturnMultipleValues_whenMyMethodReturnsSomeValue(self, mock_my_method):
    list_of_return_values= [True,False,False]
    def side_effect():
        return list_of_return_values.pop()
    mock_my_method.side_effect = side_effect
    some_other_class =  SomeOtherClassThatUsesMyClass()
    self.assertFalse(some_other_class.method_under_test())
    self.assertFalse(some_other_class.method_under_test())
    self.assertTrue(some_other_class.method_under_test())

To mock an entire class to test interactions with that class use @patch.

@patch('my_module.MyClass')
def test_my_method_shouldCallMyClassMethodMyMethod_whenSomeOtherClassMethodIsCalled(self, mock_my_class):
    some_other_class =  SomeOtherClassThatUsesMyClass()
    some_other_class.method_under_test()
    self.assertTrue(mock_my_class.called)

To mock an entire class with @patch and still set the return value of a method in that class, grab the instance of the mock object’s return value and set the method’s return value on the instance. There is a section on the patch page explaining how to do this.

@patch('my_module.MyClass')
def test_my_method_shouldReturnTrue_whenSomeOtherClassMethodIsCalledAndAReturnValueIsSet(self, mock_my_class):
    mc = mock_my_class.return_value
    mc.my_method.return_value = True
    some_other_class =  SomeOtherClassThatUsesMyClass()
    result = some_other_class.method_under_test()
    self.assertTrue(result)

Finally, here is the simple code I used as my “code under test” to write these example tests:

class MyClass():
    def my_method(self):
        pass

class SomeOtherClassThatUsesMyClass():
    def method_under_test(self):
        myclass = MyClass()
        return myclass.my_method()

There are many other features in the Python Mock library, but these are the basics that will help you get started!

PyMongo-Frisk – Now supports Replica Sets

Yep, we have updated the open source project PyMongo-Frisk (0.0.4) to now support MongoDB Replica Sets! Our library extends the PyMongo Connection class and adds an additional method called “check_health()”. This method when called returns who is the master, a list of all slave nodes, verifies read and write connectivity with the master, and finally verifies it can read from ALL slaves in the Replica Set. The intended usage for this connection class extension would be an internal monitoring page that is pinged regularly by an automated monitor and sets off an alarm if anything fails.

We created this project because we wanted to ensure that our app server had the ability to reach all failover slave nodes. We did not want to wait until a failure of the master occurred to find out the app server could not reach one of our slaves!

To use our extension of the connection class:

from pymongo_frisk import FriskConnection as Connection
connection = Connection(['host1','host2','host3'])
results = connection.check_health()

When called the method returns a dictionary containing info about the health of ALL nodes in a Replica Set:
{'db_master_host': 'host1:27017',
'db_slave_hosts': ['host2:27018','host3:27019'],
'db_master_can_write': True,
'db_master_can_read': True,
'db_slaves_can_read': [('host2',True),('host3',True)] }

Note, the library still supports Replica Pairs using authentication. We have moved on to using version 1.6+ of MongoDB, so we decided to make the jump recently from Replica Pairs to Replica Sets. Replica Pairs will be removed in MongoDB version 1.8.

Creating a REST API in Python using Bottle and MongoDB

I have been using Bottle and MongoDB for a REST API project for almost a year now. I frequently get asked the question “Bottle, what is Bottle, I have never heard of it” and “Ok, I have read the Bottle tutorial, it is simple, but how do I use Bottle with MongoDB?”. So I decided to write up a simple “Getting Started” tutorial for my friends curious about these technologies.

Prerequisites: Python 2.7 or higher

Setup MongoDB

MongoDB is a document store, a schema less database. What makes MongoDB unique (and why we decided to use it) is that MongoDB is a hybrid document store. What this means is that MongoDB offers the freedom of storing data in a schema less fashion, while still providing a flexible query syntax that will make anyone familiar with SQL feel comfortable.

You will need to download the latest version of MongoDB from the downloads page. Let’s download MongoDB and get it up and running.

First, setup the data folder. By default MongoDB stores all data files in the folder /data/db, so let’s create the folder:

sudo mkdir -m 777 -p /data/db

Download MongoDB:

wget http://fastdl.mongodb.org/osx/mongodb-osx-x86_64-1.6.5.tgz
tar -zxvf mongodb-osx-x86_64-1.6.5.tgz

Now let’s start MongoDB:

cd mongodb-osx-x86_64-1.6.5/bin
./mongod

NOTE: We have not created a database, or even a collection (you can think of a collection as a schema free table). When a document is being saved, MongoDB will lazily create the collection if it does not exist, and if the database does not exist, it will also create the database. It’s just that simple!

If you would like a more detailed introduction to using MongoDB, the query syntax, etc, please refer to my previous post: Getting Started with MongoDB

Setup Bottle

Bottle is a very simple Python web framework that is contained in a single Python file (bottle.py). We have been using Bottle for creating REST APIs and have been very impressed with just how easy it is to use.

Install the Bottle package into your Python site-packages folder:

pip install bottle

Setup PyMongo

PyMongo is a library for interacting with MongoDB from a Python application. This package bridges the gap between Bottle and MongoDB.

Install the PyMongo package into your Python site-packages folder:

pip install pymongo

Putting it all together: A Simple REST API

Now, on to the fun stuff! Let’s write a simple Python REST API, with a GET and a PUT, using Bottle, that allows us to save and retrieve documents to and from MongoDB using PyMongo.

Create a new file:

touch myrestapi.py

Cut and paste the following code:

import json
import bottle
from bottle import route, run, request, abort
from pymongo import Connection

connection = Connection('localhost', 27017)
db = connection.mydatabase

@route('/documents', method='PUT')
def put_document():
	data = request.body.readline()
	if not data:
		abort(400, 'No data received')
	entity = json.loads(data)
	if not entity.has_key('_id'):
		abort(400, 'No _id specified')
	try:
		db['documents'].save(entity)
	except ValidationError as ve:
		abort(400, str(ve))
	
@route('/documents/:id', method='GET')
def get_document(id):
	entity = db['documents'].find_one({'_id':id})
	if not entity:
		abort(404, 'No document with id %s' % id)
	return entity

run(host='localhost', port=8080)

Start the application:

python myrestapi.py

Using our new REST API

Now let’s try saving and retrieving some documents. If you do not have a preferred REST client handy and are using the Google Chrome browser, you should try the extension Simple REST Client.

Save a document

• Open up your favorite REST client and point it at the following URL:
• http://localhost:8080/documents
• Now select PUT and enter the following data:
• {"_id": "doc1", "name": "Test Document 1"}

NOTE: The primary key of a document in a MongoDB collection is always the field “_id”. If you specify an “_id” in your document it will be used, if you do not, MongoDB will generate a unique id.

Retrieve a document

• In your REST client enter the URL for retrieving the document we just saved:
• http://localhost:8080/documents/doc1
• Now select GET and you should see the same document returned:
• {"_id": "doc1", "name": "Test Document 1"}

That is it! You now have a basic, working, REST API written in Python using Bottle and MongoDB!