A Positive Root Cause Analysis

At the end of every development iteration we always do a retrospective and a root cause analysis meeting. In our retrospective we cover what went well, what did not go so well, and come up with action items for things to try in our next sprint. In our root cause analysis meetings we usually pick some problem that occurred, try to figure out why it happened, then understand how we could prevent this problem from happening again in the future. Our root cause analysis meetings are usually always on a negative topic. However, during our recent project we decided to change things up and do a root cause analysis of the following question:

Why did our last project go so well?

This may seem like a strange question to ask, but often in root cause analysis meetings we tend to focus only on the mistakes made during a project. When problems occur you need to identify the root causes of those problems to prevent them from happening again. While in a project that goes well it is just as important to recognize what we did right this time to ensure we understand “why”. What did we do different this time that worked? What were the crucial decisions made that kept the project on track? In our root cause analysis we found some key decisions, that in retrospect were important but at the time did not seem critical.

Failing fast when a decision starts to become problematic
For our project we thought we had a clear, straight forward design to work with from the beginning. However, after spending even just a day spiking some ideas our design immediately started to show cracks. Our design that on the surface looked simple, turned out to be far more complicated to implement than we had imagined. A large part of the reason for this problem was that our project was to make changes to an existing application about which no one on our project had any previous knowledge. We immediately had a team huddle, called a “Just In Time” design meeting and corrected our course. As a result, we lost a day, instead of a week or a month going down the wrong path.

Consulting experts early
When we started our project we knew of a few ways to accomplish our task, we had received suggestions that sounded fine, but we really were not sure if our approach was the best solution available. Fortunately we have some very experienced people in our company that have spent many years contracting and as a result have an incredible variety of experiences from which to draw upon. So we called a quick design meeting with one of these experts, showed them what we were thinking of doing and just picked their brain for ideas. It turned out our expert was able to come up with an approach to our problem that not only would allow us to complete the task within the time-line given to us by our business team, but at the same time would allow us to implement a cleaner solution.

Keeping code ownership high
We had no one person on the team that if they were sick for a day it would prevent a task from being completed. During our project we made sure every line of code was written with a pair (We always try to pair program every line of code) and switch pairs regularly. Because of this knowledge sharing we did not have any “Experts” on any one area of the application. We always had at least 2-3 members of the team who were knowledgeable enough on any given area of the application be able to bring another developer up to speed.

Break all stories into small tasks with a clear definition of “Done”
The stories we work on during a sprint always show the business value we are adding, but from a developers perspective there are usually multiple tasks required to complete each story. At the start of each sprint we held a task breakdown meeting for breaking each story down into a set of small tasks. Our team found that having a set of clearly defined tasks for each story was very important to keeping the project on track. With any story we receive from the business team their will be questions and as a result we found that doing this task breakdown meeting helped flush out many of those questions at the start of the sprint, as opposed to after development had already began, which in the past was usually what happened. It made it clear to our team lead and business analyst exactly what work was being done, who was doing it, what tasks had been completed, and what tasks had not yet been started. Also this gave our business analyst and team lead a better idea of when to expect demos since they could see how many tasks were remaining before a story would be completed.

Demo to business team often
We started our project doing a fairly poor job of demoing but this was corrected after one of our sprint retrospectives. Business analysts need to see the work being done and often will think of something that was missed, or see something that perhaps spawns another story. One of the easiest ways to ensure that what is being developed matches what the business team wants is to keep them in the loop and an excellent way to do that is through frequent demos.

Quick feedback from business team
During development there are always going to be cases in business logic spotted by developers that were missed during the initial planning phase. When a developer spots a missed case and brings it up to the business team, quick feedback from the business team can play a major role in keeping the project on schedule. In our last project this turn around time was often hours, if not shorter in most cases.

Keep the systems team involved from the start
The people who will be deploying the application and hosting it should be involved right from the start of the project. Your systems team has experienced many deployments and also know the pain of hosting a problematic application. Allowing the systems team, who will be responsible for the application after development has been completed, to be involved in key decisions can greatly improve the chances of a successful deployment and potentially reduce the cost of hosting the application.

Conclusion
Having your team do a positive root cause analysis can be very useful. It sometimes seems like after a problem in one sprint, we focus in the next sprint so much on improving in that one area that we sometimes slip in areas where we were previously doing well (for our team it was demos). On previous projects I have worked on, we definitely tried to follow each of these best practices outlined in our positive root cause analysis. However, since we moved to Agile almost two years ago, this was the first project where everything just “clicked”.

Django HTTP PUT with XML payload in a test

I am new to Django. I have only worked on two small projects in Django so far, but I am definitely enjoying the experience!

The problem I had recently was that I wanted to create a test for a new HTTP PUT call I was about to write. When creating a PUT call in Django using JSON, writing a test and using the call is very easy. However, this time I was working on a service that allows for a GET and a PUT call with data in an XML format. I had some trouble getting the payload set up correctly, so I thought I would share the solution that worked for me:

   data = get_testdata_from_file(filename=XML_TEST_FILE)
   payload = {'wsgi.input':FakePayload(data), 'CONTENT_LENGTH':len(data)}
   self.client.put(TEST_REST_URL, **payload)

I am sure this is trivial for you experienced Django developers, but if you are a rookie I hope this saves you some time!

Migrating Data to JSON In SQL Server

Recently I was working on a task that required migrating some existing data (Phone Numbers) from being stored as separate columns in a separate table (e.g., dbo.FriendPhoneNumbers), to being stored as a single JSON document in one column of the table containing all related data (e.g., dbo.Friend).

My task was only to complete a one-time migration, so I did not want to spend a lot of time writing a custom application to complete this work. Just for fun I decided to see if I could first accomplish this migration in SQL. I was also curious if such a SQL statement would be efficient enough to be run in a live production environment.

Using the SQL UPDATE statement below I was able to create 100,000+ JSON documents in just a few seconds.

UPDATE f
SET PhoneNumber = phoneJSON.JSON
from dbo.Friend f
join
(
	select FriendID, REPLACE(
		'{'+
		CASE WHEN Home IS NOT NULL THEN '"Home":"'+Home+'",' ELSE '' END +
		CASE WHEN Work IS NOT NULL THEN '"Work":"'+Work+'",' ELSE '' END +
		CASE WHEN Fax IS NOT NULL THEN '"Fax":"'+Fax+'",' ELSE '' END +
		CASE WHEN Cell IS NOT NULL THEN '"Cell":"'+Cell+'",' ELSE '' END
		+'}', ',}', '}') as 'JSON'
	from dbo.FriendPhoneNumbers
) as phoneJSON ON phoneJSON.FriendID = f.FriendID
GO

While writing the script I ran into two issues:

1. Nullable Columns: Wrapped each column in a SQL CASE statement so when a phone number was null, nothing was added to the JSON
2. Trailing Comma: Added a SQL REPLACE to replace “,}” with “}” to handle the case where the last phone number in the list was null

That was all! It took only a few minutes to write this script.

Second Level Caching for Hibernate with Terracotta

Traffic on our system has been increasing, putting more pressure on our database causing the response time of our application to degrade. Since we are currently using Hibernate in our project for a Java app that surfaces a REST API, we decided to look into caching strategies available. After looking around and trying a few technologies we decided to tested out the open source version of Terracotta.

To start let’s just explain the difference between First Level Caching and Second Level Caching in Hibernate.

First-level Cache

First level caching is associated with the Session object and is caching on a per transaction basis. An example would be an object where you make several updates to it as part of a single transaction. Hibernate, rather than submitting several update statements to the database, will pool the group of updates to the object into a single update statement.

Second-level Cache

Second level caching is associated with the SessionFactory object and maintains a cache of previously loaded objects accross transactional boundaries. These cached objects are available to any query for the same object, not just a single transaction. So in this case the entire application has access to retrieve any cached object. For example, if my app performs a lookup for a Product object describing a product, if the application later makes a request for the same product, this object can be retrieved from the Second-level Cache rather than making an additional trip to the database.

Why Terracotta?

Handles asynchronus queueing of writes. This feature offers some failover capabilities for when our database is not available. Originally we attacked the problem separately figuring we needed a caching strategy for reads and a fault tollerant queuing strategy for writes to deal with our current single point of failure (our database). Fortunately Terracotta has functionality that allows us to cover both of these cases.

The Developer Console. This tool allows you to monitor the traffic between your app, the Terracotta server, and your database. With the Developer Console we were able to test out different caching strategies to see which ones gave us the highest percentage of cache hits. For us developers, this was a major plus, having such a useful tool to aid us in tuning our cache settings. It made Terracotta feel much less like a black box.

Simple to install and configure. Since Terracotta just plugs in to Hibernate, there is no real development work required to get it up and running. Setting up a Terracotta server is easy, adding it to the application just meant minor additions to our pom.xml file and our hibernate settings file, and tuning our cache settings was relatively simple using the provided Developer Console.

Install and Start the Terracotta Server

• Download Terracotta
• Install by running the command: java -jar terracotta-3.1.0-installer.jar
• Start the Terracotta server: ~/terracotta/terracotta-3.1.0/bin/start-tc-server.sh
• Start the developer console: ~/terracotta/terracotta-3.1.0/bin/dev-console.sh

Add Terracotta support to your Maven/Java/Hibernate/Spring Application

• Add the Terracotta dependencies to your Maven pom.xml file

<dependency>
 <artifactId>terracotta-hibernate-agent</artifactId>
 <version>1.0.0</version>
 <groupId>org.terracotta.hibernate</groupId>
 <scope>provided</scope>
</dependency>
<dependency>
 <artifactId>terracotta-hibernate-cache</artifactId>
 <version>1.0.0</version>
 <groupId>org.terracotta.hibernate</groupId>
</dependency>

• Add the following properties to the hibernate properties section of the sessionFactory bean in your Spring database settings file (e.g., app/src/main/resouces/sping/database/database.xml)

 <bean id="sessionFactory">
<property name="dataSource" ref="dataSource"/>
 <property name="hibernateProperties">
 <props>
 ...
 <prop key="hibernate.cache.use_second_level_cache">true</prop>
 <prop key="hibernate.cache.provider_class">org.terracotta.hibernate.TerracottaHibernateCacheProvider</prop>
 ...
 </props>
 </property>
...
</bean>

• Add a cache annotation to any Hibernate Entity in your application that you wish to have cached: @Cache(usage=CacheConcurrencyStrategy.READ_WRITE)
• Create a Terracotta cache settings file. This is important because Terracotta by default NEVER evicts an object from cache. Once the memory allocated to Terracotta has been filled, Terracotta will start to write to disk unless eviction settings have been put in place. In the “Resources” folder of your application, create a file called “tc-hibernate-cache.xml”. To create the content for this file you can either look at the Terracotta Hibernate Configuration Reference Guide or use the Developer Console to generate a cache settings file using the GUI tool.

Running your project with Jetty

In our local development environments we use Jetty as our web server (Tomcat on production). To run your app from Maven using the Jetty plugin make sure you first do the following:

• Set the MAVEN_OPTS environment variable with the “javaagent” parameter

export MAVEN_OPTS="-javaagent:/path/to/terracotta-hibernate-agent-1.0.0.jar -cp /path/to/terracotta-hibernate-cache-1.0.0.jar -Xmx512m"

• NOTE: We appended to the MAVEN_OPTS environment variable the value “-Xmx512m” to tell Jetty to use 512mbs (the default is 128). We have found that with an application using Hibernate, 128 makes performance of the application sluggish.
• Now start the app with the command: mvn jetty:run

Conclusion

Overall we have found Terracotta to be a useful tool. It requires very little effort to update an existing project using Spring/Hibernate to use it. Terracotta offers more than just Second Level Caching, but also handles queuing of writes and ensuring data is written to the SOR (System or Record) in the event the database is not available for a brief period. Also, stale cache entries are not an issue as well since Terracotta updates the cache immediately on any write made to the SOR. As a final note, the Developer Console is a really nice addition!

I hope this post helps!

University of Saskatchewan Programming Contest

Marcos and I attended the University of Saskatchewan Programming Contest where local students were competing for prizes and the chance to attend the Regional Programming Contest at the U of S on Oct 30, 2009.

The contest had eight coding problems to solve. To win you just needed to correctly solve the most questions. However, in the event of a tie the team who solved the problems in the shortest amount of time wins. The questions ranged from very simple to extremmely difficult. If you are interested in seeing some sample questions (Or would like to test yourself!) you can check the U of S Programming Contest sample page.

Students competing in the contest ranged from novice to advanced, however there is also an opportunity for anyone who wants to, to form a team and compete for fun! (count us in for next year). The contest is not just about writing code, but more about focusing on problem solving where you prove your solution with a small program. Of course the competitors who are able to solve each problem with small simple applications definitely have an advantage.

While at the programming contest we had the chance to meet the co-ordinator of the programming contest, professor Christopher Dutchyn. He gave us an overview of the contest, how scoring was handled, and walked us through some of the problems teams were attempting to solve. During our discussion, Marcos chanllenged the winners of the U of S contest to a programming contest with Point2 employees.

During our weekly professional development time we plan to put together a team to practice similar problems to the ones solved in the programming contest. We hope to prepare ourselves for a fun challenge. We know the competition will be tough, but we hope to give students a fun practice session before they go on to another competition. We hope to set something up some time over the next few months.

At the end of the contest Point2 presented some prizes to winners and also we told the students a little bit about Point2. We hope to see many of those students applying for positions offered through the U of S Internship Program for next spring.

After having a chance to spend a day on campus it sure makes me miss my time their as a student. The faculty and staff were great hosts.

Validating Xml with a Schema in an NUnit Test (.NET 3.5/C#)

Recently we decided to make some updates to our web app. The front end website is in ASP.NET/C#, while the back end is all in Java and using REST APIs as the interface. One pair of programmers was assigned the task of updating the web app and the other pair was assigned the task of writing the new REST service used by the web app. To ensure that both sides would be able to talk to each other when completed, we wrote an XML Schema to act as our contract and placed it in an SVN external available to both projects.

In our functional test for the REST client side code (.NET/C#) we used a stub from which we could grab the XML being sent. To test that this XML conformed to the schema (contract) we agreed to, we created the following helper class:

Xml Validator Helper Class

using System;
using System.Xml;
using System.Xml.Schema;
using System.IO;

namespace Test.Helper
{
	public class XmlValidatorTestHelper
	{
		private bool isValidXml = true;
		private string validationError = "";

		public String ValidationError
		{
			get
			{
				return "Validation Error: " + validationError;
			}
			set
			{
				validationError = value;
			}
		}

		public bool IsValidXml
		{
			get
			{
				return isValidXml;
			}
		}

		///
		/// Validate an xml document against an xml schema.
		///
		public void ValidXmlDoc(XmlDocument xmlDocument, XmlSchema xmlSchema)
		{
			validateParameters(xmlDocument, xmlSchema);

			XmlReader xmlReader = createXmlReader(xmlDocument, xmlSchema);

			try
			{
				// validate
				using (xmlReader)
				{
					while (xmlReader.Read())
					{}
				}

				isValidXml = true;
			}
			catch (Exception ex)
			{
				ValidationError = ex.Message;
				isValidXml = false;
			}
		}

		private static void validateParameters(XmlDocument xmlDocument, XmlSchema xmlSchema)
		{
			if (xmlDocument == null)
			{
				new ArgumentNullException("ValidXmlDoc() - Argument NULL: XmlDocument");
			}
			if (xmlSchema == null)
			{
				new ArgumentNullException("ValidXmlDoc() - Argument NULL: XmlSchema");
			}
		}

		private static XmlReader createXmlReader(XmlDocument xmlDocument, XmlSchema xmlSchema)
		{
			StringReader xmlStringReader = convertXmlDocumentToStringReader(xmlDocument);
			XmlReaderSettings xmlReaderSettings = new XmlReaderSettings { ValidationType = ValidationType.Schema };
			xmlReaderSettings.Schemas.Add(xmlSchema);
			return XmlReader.Create(xmlStringReader, xmlReaderSettings);
		}

		private static StringReader convertXmlDocumentToStringReader(XmlDocument xmlDocument)
		{
			StringWriter sw = new StringWriter();
			xmlDocument.WriteTo(new XmlTextWriter(sw));
			return new StringReader(sw.ToString());
		}
	}
}

Surfacing Validation Errors in your Test

Finding out that the XML our REST client produces matches the contract is great, however when it fails it would save a lot of time if the test just showed WHY validation failed. No problem, in our validation helper class we added a property to allow us to retrieve the validation error message in the event the test failed and output that as the failure message for the test:

XmlValidatorTestHelper xmlSchemaValidator = new XmlValidatorTestHelper();
XmlSchema myXmlSchema = XmlSchema.Read(...);
XmlDocument myXmlDocument = new XmlDocument();
myXmlDocument .Load(...);
xmlSchemaValidator.ValidXmlDoc(myXmlDocument, myXmlSchema);

Assert.IsTrue(xmlSchemaValidator.IsValidXml, "XML does not match Schema: " + xmlSchemaValidator.ValidationError);

The Pitfalls of Local Optimization

Recently I was on a project where our business team came to us and asked us to build a new service (Let’s call it feature X). Our development team immediately got together and started working out the details: what database to use, programming language, service type (e.g., REST), etc. However, during these first few days of discussion there was one question that was bugging all of us developers:

“Where is the business value in building feature X”

Since we were all curious and wanted to have an answer to this question, we finally grabbed some members of our business and marketing teams and posed this exact question. Our business team responded by outlining three key points where feature X would excite our customers and bring great revenue opportunities for our company. The first two points were big wins in terms of adoption of our product in the short term and not having them now was costing us big time. The third point was something that was important to our business strategy in the long term.

The interesting thing was that looking at points one and two the development team immediately saw a very cheap way to offer that functionality within one week. Not a hack, but rather a change to an area of the application where the design was very flexible. Something our business team would not know. This obviously peaked the interests of our business and marketing team (Our stakeholders). So rather than waiting possibly six months for a project that would complete all of points one, two, and three, we were able to deliver one and two now and point three would only be delayed one week over the course of six months.

Simple Estimation: Is it cheap, or is it expensive

Sometimes business decisions get made without the people making the decisions knowing the cost of each option. Grabbing a developer and taking them to a business team brain storming session for the sole purpose of giving “Is it cheap, or is it expensive” feedback can be very useful in prioritizing new work. As a developer you have a responsiblity to provide this feedback.

Don’t be affraid to ask “Why”

As a developer when you are asked to build feature X, rather than just jumping into development of the feature. Put some thought into what it is that business is really looking to get out of the project. Make sure that what has been asked for is actually the best way to deliver the business value your stakeholders want.

This lead some of my co-workers to refer to a problem in software development known as Local Optimization.

TeamCity – Using Personal Builds to help improve cycle time

So you are working on a large project, many developers committing regularly to the same code base, you run all the unit tests, commit and you break the build. Now everyone on your team is angry (As they should be!). How could this have happened when you ran all of the tests? The reason is that you did not update before running tests locally and committing. The problem is that constantly updating and building can waste a lot of time. Even if you do update, run all tests, then commit, there is still a chance another developer made a commit during that cycle. The solution: Team City Personal Builds.

Personal Builds

The great people at JetBrains have a feature in TeamCity that allows you to use your build server build agents for “Personal Builds”. More great news is that TeamCity plugins that make use of this interesting TeamCity feature are available for IDEs such as IntelliJ and Visual Studio.

So now when you use your IDE tool with the TeamCity plugin you would use the “Remote Run” option to make your commit. TeamCity will then spin up a personal build, apply your code changes to a fresh checkout of the repository, build, and run all unit tests. If your “Personal Build” passes, your code changes will be committed. Now, here is the great part, if it fails, your code changes will NOT be committed, and you will be notified of the failure.

This feature of TeamCity allows developers to get out of the update, run all tests, and commit cycle and offload some of that work to their TeamCity server.

IntelliJ Plugin Setup

• File->Settings->Plugins
• Click on the “Available” tab
• In the search box enter “TeamCity” (The option JetBrians TeamCity Plugin should show up in the list)
• Right click on the plugin and select “Download and install”
• You should see a TeamCity option on your IntelliJ toolbar
• Enter your TeamCity credentials, setup your notifications

So now, when you commit use TeamCity – > Remote Run rather than using the Version Control menu. You will be notified if your commit is accepted or rejected.

Visual Studio Plugin Setup

To download the Plugin, login to your TeamCity server, go to the “My Settings & Tools” page. You will find a download link for the plugin in the “TeamCity Tools” panel on the righthand side of the page.

Hibernate – Cascade delete does not delete ALL related rows

Recently I had a problem where a cascade delete was not deleting all related rows. I am fairly new to HIbernate, so I thought I would post my solution in case anyone else runs into this problem.

To explain the problem, lets say I have a hibernate entity called “Customer” that has a one to many mapping to another hibernate entity called “Subscriptions”. In my test I saved a Customer with several Subscriptions then deleted the Customer and asserted that all related entries in the Subscriptions table were removed. The problem was that each time only ONE entry was removed, but never ALL related entries in the Subscription table.

The “Customer” entity was set up with a relationship to “Subscriptions” as follows:

@OneToMany(cascade = CascadeType.ALL, fetch = FetchType.EAGER)
@Cascade({org.hibernate.annotations.CascadeType.DELETE_ORPHAN})
@JoinColumn(name = "customerID", referencedColumnName = "id", nullable = false)
private List subscriptions = new ArrayList();

The problem turned out to be that I needed to add a “Fetch” annotation:

@OneToMany(cascade = CascadeType.ALL, fetch = FetchType.EAGER)
@Cascade({org.hibernate.annotations.CascadeType.DELETE_ORPHAN})
@JoinColumn(name = "customerID", referencedColumnName = "id", nullable = false)
@Fetch(value = FetchMode.SUBSELECT)
private List subscriptions = new ArrayList();

This solved the problem. Now ALL related rows in the Subscriptions table sub select are removed on cascade delete of a Customer.

Java – REST APIs and Thread Safety

In my current job, a few months ago we ran into some strange issues on production. Customers were complaining that sometimes they would receive incorrect data when accessing our REST APIs. We were never able to reproduce the issue and the strange results customers were receiving seemed entirely random. Until one day while a member of our team was accessing a REST API at the same time as a customer. As a result the customer received the employee’s information! After this discovery we found that when accessing a REST call in our system concurrently with different parameters we were consistently able to reproduce the problem the customer was complaining about. Yep, we had classes that were not thread safe and needed to be fixed!

The Bug

In our case the thread safety issue was related to the use of local variables inside a class declared as a singleton in our Spring configuration. The issue was a race condition between when the lookup parameter (e.g., customerId) was set and when it was used to lookup a requested value (e.g., email). Take a look at this example class:

public class CustomerInfo {
    int customerId;

    public void setCustomerId(int customerId) {
        this.customerId = customerId;
    }

    public String getEmail() {
        return customerRepository.getEmail(customerId);
    }
}

Let’s say we have two customers placing an order:
Customer 1: has an email address of jsmith@nowhere.com
Customer 2: has an email address of jdoe@nowhere.com

Now we have two callers accessing our REST API at the same time so two threads are using our CustomerInfo class:

• Thread A: setCustomerId(1) – customerId=1
• Thread B: setCustomerId(2) – customerId=2
• Thread A: getEmail() – customerId=2 so email jdoe@nowhere.com is returned
• Thread B: getEmail() – customerId=2 so email jdoe@nowhere.com is returned

The Fix

So even though the code calling the getEmail() method was calling it immediately after setting the customerId with the setCustomerId() method, if two threads were running at the same time the chances of the customerId being updated by one thread and affecting the results of another were very likely. There was no reason why we needed to set a customerId value in the class, this was just extra code that was not needed. So removing unnecessary code and fixing our thread safety issue ended up being connected:

public class CustomerInfo{
    public String getEmail(int customerId) {
        return customerRepository.getEmail(customerId);
    }
}

The way we were able to find and track down this concurrency bug was by writing a concurrency test first.

A Simple Concurrency Test

A concurrency test for a REST call does not have to be anything complicated. Just take a REST call you want to test and write a test following this basic outline:

• One REST call to test
• A list of valid parameters to use with that REST call and an expected value for each parameter.
• This expected value is used to check that the request was successful (e.g., parameter CustomerID=2938 returns Email=johnsmith@gmail.com).
• Pool of threads, each calling the REST call for each of the parameters in the list.
• Verify every REST call in each thread returns the expected value.

Now run the test and if any of the REST calls return data that does not match the expected value, you know you have some code that is not thread safe. Now that you have this failing test in place you can proceed with finding and fixing your thread safety issues!