My Adventures in Coding

January 13, 2011

PyMongo-Frisk – Now supports Replica Sets

Filed under: MongoDB,PyMongo,Python — Brian @ 12:35 am
Tags: , , , , ,

Yep, we have updated the open source project PyMongo-Frisk (0.0.4) to now support MongoDB Replica Sets! Our library extends the PyMongo Connection class and adds an additional method called “check_health()”. This method when called returns who is the master, a list of all slave nodes, verifies read and write connectivity with the master, and finally verifies it can read from ALL slaves in the Replica Set. The intended usage for this connection class extension would be an internal monitoring page that is pinged regularly by an automated monitor and sets off an alarm if anything fails.

We created this project because we wanted to ensure that our app server had the ability to reach all failover slave nodes. We did not want to wait until a failure of the master occurred to find out the app server could not reach one of our slaves!

To use our extension of the connection class:

from pymongo_frisk import FriskConnection as Connection
connection = Connection(['host1','host2','host3'])
results = connection.check_health()

When called the method returns a dictionary containing info about the health of ALL nodes in a Replica Set:
{'db_master_host': 'host1:27017',
'db_slave_hosts': ['host2:27018','host3:27019'],
'db_master_can_write': True,
'db_master_can_read': True,
'db_slaves_can_read': [('host2',True),('host3',True)] }

Note, the library still supports Replica Pairs using authentication. We have moved on to using version 1.6+ of MongoDB, so we decided to make the jump recently from Replica Pairs to Replica Sets. Replica Pairs will be removed in MongoDB version 1.8.

November 2, 2010

PyMongo – Checking the health of Replica Pair connections

Filed under: MongoDB,PyMongo,Python — Brian @ 9:13 pm

We are currently using MongoDB setup with Replica Pairs (We are waiting until version 1.8.0 to switch to Replica Sets). Our application that uses MongoDB is a Python REST API using PyMongo. Once we started development on this project the first thing we wanted to do was sort out how we were going to monitor the application. We ended up deciding to add a monitoring page to our REST API and used it to display information showing the health of the application. We setup monitors to verify each instance of MongoDB was up and running, monitors to verify that both instances of MongoDB were pairing correctly, and we setup monitors to call our new monitoring page to verify our application had read and write connectivity to the Master in our Replica Pair. However, there was a very important piece not being verified by our monitors:

In the event of a failure of the Master in our Replica Pair, would our application be able to successfully connect to the Slave instance when it took over as the Master?

The issue is that when PyMongo connects to a Replica Pair, if it is able to connect to the first MongoDB instance in the pair, it just connects and never verifies that it can also connect to the Slave instance. So for example if there were some firewall rule that would prevent the application from connecting to the second instance of MongoDB in the pair, we would prefer to find this out BEFORE a failure of the Master occurs!

So our solution was to extend the PyMongo connection class to add an additional method called “check_health()”. This method verifies that it can connect to both MongoDB instances in the Replica Pair, verifies it can write to the Master, read from the Master, and can read from the Slave.

My co-worker James and have created a small open source project called PyMongo-Frisk which you can use to get our additional connection functionality. You use it just like the connection class in PyMongo:

import pymongo_frisk as pymongo
connection = pymongo.connection.Connection.from_uri("mongo://username:password@host1,host2/database")
results = connection.check_health()

The check_health() call returns the results as a dictionary:
{'db_master_url': 'host2', 'db_slave_url': 'host1', 'db_master_can_write': True, 'db_slave_can_read': True, 'db_master_can_read': True}

You can get a copy of the PyMongo-Frisk code from any of the following sources:

Softpedia
pypi
Github

Blog at WordPress.com.