My Adventures in Coding

September 11, 2011

Python – Upgrading Python with easy_install, pip, and virtualenv on a Mac

Filed under: Mac,Python — Brian @ 1:45 pm
Tags: , ,

For this tutorial I am upgrading from python 2.6 to python 2.7 on my Mac. Every time I have to upgrade Python versions, it always takes me a few minutes to remember the steps since I do this task so rarely. So here is a simple set of instructions that seem to work for me!

Install python 2.7

  • Download the Python Mac dmg file from the Python Downloads page
    Just open the dmg and follow the installer as you would with any application
  • You should now have Python 2.7 installed in:
    /Library/Frameworks/Python.framework/Versions/2.7/bin/python

NOTE: Even though we have installed python 2.7, your /usr/bin/python is still pointing to an older version of Python, so for example if you run easy_install, if will run from the older version of Python, and install packages into the library of the old Python installation.

So, on to step two of the install:
Let’s switch the /usr/bin/python link to point to the newest version of Python, and at the same time let’s create a soft link in the Python install folder that always points to the current install of Python. So the next time we upgrade, we will only need to change the “Current” version link.

#Create a soft link in /Library/Frameworks/Python.framework/Versions/
cd /Library/Frameworks/Python.framework/Versions/
ln -s 2.7 Current

#Switch the /usr/bin/python link to point to current python link
cd /usr/bin
rm -f python
ln -s /Library/Frameworks/Python.framework/Versions/Current/bin/python python

Install Setup Tools (this includes easy_install)

Download the egg file for your version of python (for this example, 2.7) from the Setup Tools Downloads page. To install setup tools just run the egg file like any other shell script:

sudo sh setuptools-0.6c11-py2.7.egg

Install pip (Using easy_install)

Open a new Terminal window session. You will need a new terminal session for easy_install to show up on the path.

sudo easy_install pip

Now we can install any python packages we want using pip!

Install Virtualenv (Using pip)

Now that we have pip, lets install a very important package called virtualenv!

pip install virtualenv

Thats it!

Now you will have Python 2.7 on your Mac, with easy_install, pip, and virtualenv ready to go.

February 26, 2011

Python – Getting Started with Python Mock

Filed under: Python,Testing — Brian @ 12:00 am
Tags: , , ,

These examples all use the Python Mock library. This post is just a quick “Getting Started” for someone who is new to using the Python Mock library, once you get started refer to the Python Mock Documentation.

First you will need to install the mock library:

pip install -U mock

Your test file will need the following import statement:

from mock import patch, Mock

To mock a method in a class to return a specific value use @patch.object.

@patch.object(my_module.MyClass, 'my_method')
def test_my_method_shouldReturnTrue_whenMyMethodReturnsSomeValue(self, mock_my_method):
    mock_my_method.return_value=True
    some_other_class =  SomeOtherClassThatUsesMyClass()
    result = some_other_class.method_under_test()
    self.assertTrue(result)

To mock a method in a class with @patch.object but return a different value each time it is called, use side_effect. Side effect allows you to define a custom method and have that method called each time your mock method is called. The value returned from this method will be used as the return value your mock method.

@patch.object(my_module.MyClass, 'my_method')
def test_my_method_shouldReturnMultipleValues_whenMyMethodReturnsSomeValue(self, mock_my_method):
    list_of_return_values= [True,False,False]
    def side_effect():
        return list_of_return_values.pop()
    mock_my_method.side_effect = side_effect
    some_other_class =  SomeOtherClassThatUsesMyClass()
    self.assertFalse(some_other_class.method_under_test())
    self.assertFalse(some_other_class.method_under_test())
    self.assertTrue(some_other_class.method_under_test())

To mock an entire class to test interactions with that class use @patch.

@patch('my_module.MyClass')
def test_my_method_shouldCallMyClassMethodMyMethod_whenSomeOtherClassMethodIsCalled(self, mock_my_class):
    some_other_class =  SomeOtherClassThatUsesMyClass()
    some_other_class.method_under_test()
    self.assertTrue(mock_my_class.called)

To mock an entire class with @patch and still set the return value of a method in that class, grab the instance of the mock object’s return value and set the method’s return value on the instance. There is a section on the patch page explaining how to do this.

@patch('my_module.MyClass')
def test_my_method_shouldReturnTrue_whenSomeOtherClassMethodIsCalledAndAReturnValueIsSet(self, mock_my_class):
    mc = mock_my_class.return_value
    mc.my_method.return_value = True
    some_other_class =  SomeOtherClassThatUsesMyClass()
    result = some_other_class.method_under_test()
    self.assertTrue(result)

Finally, here is the simple code I used as my “code under test” to write these example tests:

class MyClass():
    def my_method(self):
        pass

class SomeOtherClassThatUsesMyClass():
    def method_under_test(self):
        myclass = MyClass()
        return myclass.my_method()

There are many other features in the Python Mock library, but these are the basics that will help you get started!

January 13, 2011

PyMongo-Frisk – Now supports Replica Sets

Filed under: MongoDB,PyMongo,Python — Brian @ 12:35 am
Tags: , , , , ,

Yep, we have updated the open source project PyMongo-Frisk (0.0.4) to now support MongoDB Replica Sets! Our library extends the PyMongo Connection class and adds an additional method called “check_health()”. This method when called returns who is the master, a list of all slave nodes, verifies read and write connectivity with the master, and finally verifies it can read from ALL slaves in the Replica Set. The intended usage for this connection class extension would be an internal monitoring page that is pinged regularly by an automated monitor and sets off an alarm if anything fails.

We created this project because we wanted to ensure that our app server had the ability to reach all failover slave nodes. We did not want to wait until a failure of the master occurred to find out the app server could not reach one of our slaves!

To use our extension of the connection class:

from pymongo_frisk import FriskConnection as Connection
connection = Connection(['host1','host2','host3'])
results = connection.check_health()

When called the method returns a dictionary containing info about the health of ALL nodes in a Replica Set:
{'db_master_host': 'host1:27017',
'db_slave_hosts': ['host2:27018','host3:27019'],
'db_master_can_write': True,
'db_master_can_read': True,
'db_slaves_can_read': [('host2',True),('host3',True)] }

Note, the library still supports Replica Pairs using authentication. We have moved on to using version 1.6+ of MongoDB, so we decided to make the jump recently from Replica Pairs to Replica Sets. Replica Pairs will be removed in MongoDB version 1.8.

January 2, 2011

Creating a REST API in Python using Bottle and MongoDB

Filed under: MongoDB,Python,REST — Brian @ 10:15 pm
Tags: , , , , , ,

I have been using Bottle and MongoDB for a REST API project for almost a year now. I frequently get asked the question “Bottle, what is Bottle, I have never heard of it” and “Ok, I have read the Bottle tutorial, it is simple, but how do I use Bottle with MongoDB?”. So I decided to write up a simple “Getting Started” tutorial for my friends curious about these technologies.

Prerequisites: Python 2.7 or higher

Setup MongoDB

MongoDB is a document store, a schema less database. What makes MongoDB unique (and why we decided to use it) is that MongoDB is a hybrid document store. What this means is that MongoDB offers the freedom of storing data in a schema less fashion, while still providing a flexible query syntax that will make anyone familiar with SQL feel comfortable.

You will need to download the latest version of MongoDB from the downloads page. Let’s download MongoDB and get it up and running.

First, setup the data folder. By default MongoDB stores all data files in the folder /data/db, so let’s create the folder:

sudo mkdir -m 777 -p /data/db

Download MongoDB:

wget http://fastdl.mongodb.org/osx/mongodb-osx-x86_64-1.6.5.tgz
tar -zxvf mongodb-osx-x86_64-1.6.5.tgz

Now let’s start MongoDB:

cd mongodb-osx-x86_64-1.6.5/bin
./mongod

NOTE: We have not created a database, or even a collection (you can think of a collection as a schema free table). When a document is being saved, MongoDB will lazily create the collection if it does not exist, and if the database does not exist, it will also create the database. It’s just that simple!

If you would like a more detailed introduction to using MongoDB, the query syntax, etc, please refer to my previous post: Getting Started with MongoDB

Setup Bottle

Bottle is a very simple Python web framework that is contained in a single Python file (bottle.py). We have been using Bottle for creating REST APIs and have been very impressed with just how easy it is to use.

Install the Bottle package into your Python site-packages folder:

pip install bottle

Setup PyMongo

PyMongo is a library for interacting with MongoDB from a Python application. This package bridges the gap between Bottle and MongoDB.

Install the PyMongo package into your Python site-packages folder:

pip install pymongo

Putting it all together: A Simple REST API

Now, on to the fun stuff! Let’s write a simple Python REST API, with a GET and a PUT, using Bottle, that allows us to save and retrieve documents to and from MongoDB using PyMongo.

Create a new file:

touch myrestapi.py

Cut and paste the following code:

import json
import bottle
from bottle import route, run, request, abort
from pymongo import Connection

connection = Connection('localhost', 27017)
db = connection.mydatabase

@route('/documents', method='PUT')
def put_document():
	data = request.body.readline()
	if not data:
		abort(400, 'No data received')
	entity = json.loads(data)
	if not entity.has_key('_id'):
		abort(400, 'No _id specified')
	try:
		db['documents'].save(entity)
	except ValidationError as ve:
		abort(400, str(ve))
	
@route('/documents/:id', method='GET')
def get_document(id):
	entity = db['documents'].find_one({'_id':id})
	if not entity:
		abort(404, 'No document with id %s' % id)
	return entity

run(host='localhost', port=8080)

Start the application:

python myrestapi.py

Using our new REST API

Now let’s try saving and retrieving some documents. If you do not have a preferred REST client handy and are using the Google Chrome browser, you should try the extension Simple REST Client.

Save a document
  • Open up your favorite REST client and point it at the following URL:
  • Now select PUT and enter the following data:
    • {"_id": "doc1", "name": "Test Document 1"}

NOTE: The primary key of a document in a MongoDB collection is always the field “_id”. If you specify an “_id” in your document it will be used, if you do not, MongoDB will generate a unique id.

Retrieve a document
  • In your REST client enter the URL for retrieving the document we just saved:
  • Now select GET and you should see the same document returned:
    • {"_id": "doc1", "name": "Test Document 1"}

That is it! You now have a basic, working, REST API written in Python using Bottle and MongoDB!

November 2, 2010

PyMongo – Checking the health of Replica Pair connections

Filed under: MongoDB,PyMongo,Python — Brian @ 9:13 pm

We are currently using MongoDB setup with Replica Pairs (We are waiting until version 1.8.0 to switch to Replica Sets). Our application that uses MongoDB is a Python REST API using PyMongo. Once we started development on this project the first thing we wanted to do was sort out how we were going to monitor the application. We ended up deciding to add a monitoring page to our REST API and used it to display information showing the health of the application. We setup monitors to verify each instance of MongoDB was up and running, monitors to verify that both instances of MongoDB were pairing correctly, and we setup monitors to call our new monitoring page to verify our application had read and write connectivity to the Master in our Replica Pair. However, there was a very important piece not being verified by our monitors:

In the event of a failure of the Master in our Replica Pair, would our application be able to successfully connect to the Slave instance when it took over as the Master?

The issue is that when PyMongo connects to a Replica Pair, if it is able to connect to the first MongoDB instance in the pair, it just connects and never verifies that it can also connect to the Slave instance. So for example if there were some firewall rule that would prevent the application from connecting to the second instance of MongoDB in the pair, we would prefer to find this out BEFORE a failure of the Master occurs!

So our solution was to extend the PyMongo connection class to add an additional method called “check_health()”. This method verifies that it can connect to both MongoDB instances in the Replica Pair, verifies it can write to the Master, read from the Master, and can read from the Slave.

My co-worker James and have created a small open source project called PyMongo-Frisk which you can use to get our additional connection functionality. You use it just like the connection class in PyMongo:

import pymongo_frisk as pymongo
connection = pymongo.connection.Connection.from_uri("mongo://username:password@host1,host2/database")
results = connection.check_health()

The check_health() call returns the results as a dictionary:
{'db_master_url': 'host2', 'db_slave_url': 'host1', 'db_master_can_write': True, 'db_slave_can_read': True, 'db_master_can_read': True}

You can get a copy of the PyMongo-Frisk code from any of the following sources:

Softpedia
pypi
Github

October 8, 2010

Python – Extending a class with a class method acting as an instance method

Filed under: Python — Brian @ 11:17 pm

Recently I had a problem where I wanted to extend a class in an external library to add an additional method. I wanted to ensure that the usage of the extended version of the class would not require any code changes, just changing of an import statement. The new method we were adding to the class required that some information be retained in the object from when it was constructed. The problem was that the method in the existing library that is used to create an instance of the object is a class method (@classmethod) so it is called on the class and returns the object. Unfortunately any parameters used in that method, which included the parameter we wanted to store, could not be stored for later use in the object.

In order to make this work we would need to add an __init__ method to our new extended version of the class and use that init method rather than the existing class method to create the object. However, if we did this we would not be maintaining the existing API of the class so using our modified class would not be seamless.

It turns out there is a simple way to do this in Python (Thanks to Chris for pointing out this pattern!).

For this example let’s say we have a class called “carslib” that contains “Car” that we want to extend with a new method called “get_top_speed”:

New Wrapper Class for carslib called new_carslib

So what happens here is when the new version of carslib is used and the class method some_class_method is called it calls the __init__ method where we then store which then stores our parameter. Create the following Python file called “new_carslib.py”.

from carslib.car import Car as mycartoextend

Class new_carslib(object)
      """"Warpper for carslib to allow for a new method to be added called get_top_speed()""""
      def __init__(self):
           self._topspeed = topspeed
           self._car = mycartoextend.some_class_method(topspeed, **kw)

      @classmethod
      def some_class_method(cls, topspeed, **kw):
           return cls(uri, **kw)

      def get_top_speed(self):
           return self._topspeed

Car = new_carslib

Usage of the old class

    import carslib
    car = carslib.Car.some_class_method(180)

Usage of the new class

    import new_carslib as carslib
    car = carslib.Car.some_class_method(180)
    results = car.get_top_speed()

Notice that the only change we make here, is updating the import statement, since we took the extra care to maintain the existing api of carslib. Now we are able to use our new method with no other code changes being required!

Blog at WordPress.com.