My Adventures in Coding

December 27, 2009

A Positive Root Cause Analysis

Filed under: Agile — Brian @ 11:35 pm
Tags: , , ,

At the end of every development iteration we always do a retrospective and a root cause analysis meeting. In our retrospective we cover what went well, what did not go so well, and come up with action items for things to try in our next sprint. In our root cause analysis meetings we usually pick some problem that occurred, try to figure out why it happened, then understand how we could prevent this problem from happening again in the future. Our root cause analysis meetings are usually always on a negative topic. However, during our recent project we decided to change things up and do a root cause analysis of the following question:

Why did our last project go so well?

This may seem like a strange question to ask, but often in root cause analysis meetings we tend to focus only on the mistakes made during a project. When problems occur you need to identify the root causes of those problems to prevent them from happening again. While in a project that goes well it is just as important to recognize what we did right this time to ensure we understand “why”. What did we do different this time that worked? What were the crucial decisions made that kept the project on track? In our root cause analysis we found some key decisions, that in retrospect were important but at the time did not seem critical.

Failing fast when a decision starts to become problematic
For our project we thought we had a clear, straight forward design to work with from the beginning. However, after spending even just a day spiking some ideas our design immediately started to show cracks. Our design that on the surface looked simple, turned out to be far more complicated to implement than we had imagined. A large part of the reason for this problem was that our project was to make changes to an existing application about which no one on our project had any previous knowledge. We immediately had a team huddle, called a “Just In Time” design meeting and corrected our course. As a result, we lost a day, instead of a week or a month going down the wrong path.

Consulting experts early
When we started our project we knew of a few ways to accomplish our task, we had received suggestions that sounded fine, but we really were not sure if our approach was the best solution available. Fortunately we have some very experienced people in our company that have spent many years contracting and as a result have an incredible variety of experiences from which to draw upon. So we called a quick design meeting with one of these experts, showed them what we were thinking of doing and just picked their brain for ideas. It turned out our expert was able to come up with an approach to our problem that not only would allow us to complete the task within the time-line given to us by our business team, but at the same time would allow us to implement a cleaner solution.

Keeping code ownership high
We had no one person on the team that if they were sick for a day it would prevent a task from being completed. During our project we made sure every line of code was written with a pair (We always try to pair program every line of code) and switch pairs regularly. Because of this knowledge sharing we did not have any “Experts” on any one area of the application. We always had at least 2-3 members of the team who were knowledgeable enough on any given area of the application be able to bring another developer up to speed.

Break all stories into small tasks with a clear definition of “Done”
The stories we work on during a sprint always show the business value we are adding, but from a developers perspective there are usually multiple tasks required to complete each story. At the start of each sprint we held a task breakdown meeting for breaking each story down into a set of small tasks. Our team found that having a set of clearly defined tasks for each story was very important to keeping the project on track. With any story we receive from the business team their will be questions and as a result we found that doing this task breakdown meeting helped flush out many of those questions at the start of the sprint, as opposed to after development had already began, which in the past was usually what happened. It made it clear to our team lead and business analyst exactly what work was being done, who was doing it, what tasks had been completed, and what tasks had not yet been started. Also this gave our business analyst and team lead a better idea of when to expect demos since they could see how many tasks were remaining before a story would be completed.

Demo to business team often
We started our project doing a fairly poor job of demoing but this was corrected after one of our sprint retrospectives. Business analysts need to see the work being done and often will think of something that was missed, or see something that perhaps spawns another story. One of the easiest ways to ensure that what is being developed matches what the business team wants is to keep them in the loop and an excellent way to do that is through frequent demos.

Quick feedback from business team
During development there are always going to be cases in business logic spotted by developers that were missed during the initial planning phase. When a developer spots a missed case and brings it up to the business team, quick feedback from the business team can play a major role in keeping the project on schedule. In our last project this turn around time was often hours, if not shorter in most cases.

Keep the systems team involved from the start
The people who will be deploying the application and hosting it should be involved right from the start of the project. Your systems team has experienced many deployments and also know the pain of hosting a problematic application. Allowing the systems team, who will be responsible for the application after development has been completed, to be involved in key decisions can greatly improve the chances of a successful deployment and potentially reduce the cost of hosting the application.

Having your team do a positive root cause analysis can be very useful. It sometimes seems like after a problem in one sprint, we focus in the next sprint so much on improving in that one area that we sometimes slip in areas where we were previously doing well (for our team it was demos). On previous projects I have worked on, we definitely tried to follow each of these best practices outlined in our positive root cause analysis. However, since we moved to Agile almost two years ago, this was the first project where everything just “clicked”.


September 18, 2009

The Pitfalls of Local Optimization

Filed under: Agile — Brian @ 3:59 pm

Recently I was on a project where our business team came to us and asked us to build a new service (Let’s call it feature X). Our development team immediately got together and started working out the details: what database to use, programming language, service type (e.g., REST), etc. However, during these first few days of discussion there was one question that was bugging all of us developers:

“Where is the business value in building feature X”

Since we were all curious and wanted to have an answer to this question, we finally grabbed some members of our business and marketing teams and posed this exact question. Our business team responded by outlining three key points where feature X would excite our customers and bring great revenue opportunities for our company. The first two points were big wins in terms of adoption of our product in the short term and not having them now was costing us big time. The third point was something that was important to our business strategy in the long term.

The interesting thing was that looking at points one and two the development team immediately saw a very cheap way to offer that functionality within one week. Not a hack, but rather a change to an area of the application where the design was very flexible. Something our business team would not know. This obviously peaked the interests of our business and marketing team (Our stakeholders). So rather than waiting possibly six months for a project that would complete all of points one, two, and three, we were able to deliver one and two now and point three would only be delayed one week over the course of six months.

Simple Estimation: Is it cheap, or is it expensive

Sometimes business decisions get made without the people making the decisions knowing the cost of each option. Grabbing a developer and taking them to a business team brain storming session for the sole purpose of giving “Is it cheap, or is it expensive” feedback can be very useful in prioritizing new work. As a developer you have a responsiblity to provide this feedback.

Don’t be affraid to ask “Why”

As a developer when you are asked to build feature X, rather than just jumping into development of the feature. Put some thought into what it is that business is really looking to get out of the project. Make sure that what has been asked for is actually the best way to deliver the business value your stakeholders want.

This lead some of my co-workers to refer to a problem in software development known as Local Optimization.

March 7, 2008

Agile and Object Oriented Principles

Filed under: Agile,SD West 2008 — Brian @ 8:48 pm

I had the opportunity to attend a talk on Agile by Robert C. Martin while I was at SD West. The following is a quick summary of the main points stressed in his talk.

In order for a team to really adopt Agile, the project must be structured in a way to allow for small, thin vertical strips of the application to be released independently of any other feature in the application. If we have a story for feature A, then we should be able to implement the story for feature A, test feature A, and release feature A without affecting feature B. If this is not possible in your current project, then you must work to moving in this direction. This division is extremely important at working towards an Agile system that has few bugs and is quick to QA.

Example Scenario

I make a change to feature A, well it is not just feature A that needs to have QA look at it, it is now also feature B that needs a complete regression test. But wait, what if QA is unaware that feature B is dependent on code that was changed in feature A, then that testing may get missed. Now, also because we can’t release feature A independently of feature B we now have to release both together. Now, we have created production bugs that are costly to fix in feature B and now we have maintenance tasks that will now take up time in the next sprint. This is time that could have instead been spent on other new development work to add more business value to the product. Also those bugs in feature B will now require testing time in the next sprint and possibly regression testing of feature A or possible breaking feature C, etc.

So, as was stressed in the talk, you should always be working towards small applications that can be independently released, if not, you my be stuck in this loop.

March 6, 2008

How to be Agile with Fragile Legacy Code

Filed under: Agile,SD West 2008 — Brian @ 8:30 pm
Tags: ,

I attended a talk by Vandana Shah while at SD West 2008 that had a different perspective on transitioning to Agile. Most discussions on Agile focus on teams in the initial stages of a project, not after several years of code has been written. In the short term the costs of adding legacy code appear cheap, but in the long term become increasingly expensive in terms of maintenance and QA costs. The basic approaches to reducing this cost later on, by investing in your code now is to reduce complexity, increase test coverage, introduce best practices (coding standards for both code and unit tests), and as a result reduce the number of issues that make it to the QA cycle and production.

Unit Testing (Adding tests to legacy code)

So where do you begin? Start by adding tests to cover existing functionality in your product. The product is stable and in production, so start covering it with tests. There are two types of tests

  1. High Level – these are the tests that ensure the system does what it is supposed to, protects the business logic. These tests allow us to confirm that the functionality the user expects from the system has not been broken by any additional changes. These tests cover the bugs that will be the most likely to occur
  2. Low Level – these are the tests that cover individual pieces of logic, the small units that make up the high level tests

You should start with high level tests, these are the tests that offer the most business value. It is important to capture the overall functionality of the system with tests first when dealing with legacy code, since these are the tests that will provide the least hindrances to future refactoring.

Teams need training on Unit Testing and often they demand it (especially when it comes to refactoring existing legacy code to get it into a testable state). Document it on the wiki and update it regularly!!!!! Providing this training early on prevents a mess that needs to be cleaned up later.

Switching to Agile

Switching an existing team with a legacy application over to Agile requires a different approach then starting Agile with a new team working on a new application. It is a mistake to take the “All or nothing” approach to SCRUM, the “We must do it all now”. Agile in this type of environment should be introduced in a slow incremental approach where each new process is brought in one at a time, refined in the retrospective, then as that process stabilizes another is introduced. So the cycle is “Start small, test, refine, add more”. This approach may feel like a slower change, but often can make the transition significantly easier. Switching a teams entire process all at once, can lead to frustration amongst the team and loosely followed processes.

Problems with Large Projects (Code size)

Often developers will change their development process just based on the size of the existing application. For example, you have a very large project, it is very slow in your IDE tool, you want to use a refactoring tool, but are unable to because of the size of your project and how slow the tool runs on the project. So what happens? The team does not use a refactoring tool. This is a mistake that misses focusing on the core issue: The project is too big, it needs to be broken up. It is also a hindrance to TDD. Quick feedback is extremely important for TDD, long build times and slow running unit tests end up hurting this process. Developers do not want to wait for the tests to finish, so will write more and test later.

Regular Software Cleanup

Make time in each sprint to not just write unit tests, but to also remove dead code and duplicate code as part of the step of adding those unit tests.

Testing and QA

Allow automation to happen over time, DO NOT force it. Establish the right processes first, everytime those processes get ironed out, QA will see common areas that can be easily automated, then you can move in that direction.

Piggy Backing

One approach discussed to introducing Agile methodologies into an existing team was called “Piggy Backing”. Do not introduce a new process all at once, just introduce new items to the existing process slowly overtime, piggy backing on what the team is already doing. The team will evolve to where you want them to be and often get their quicker, with far less pain and less resistance from the team

Principles and Practices of SCRUM

Filed under: Agile,SD West 2008 — Brian @ 8:26 pm

Mark and I attended a talk called “Principles and Practices of SCRUM” this afternoon, presented by Rob Myers from NetObjectives.

When a team commits to completing a set of stories for a sprint, it is up to the Product Owners to constantly be viewing the Burn Down or Burn Up charts very early on in the sprint and check the velocity. If the velocity shows that the team will not be on track to finish all the stories that have been put in the sprint, it is extremely important to pull stories early in the sprint and not late. The reason is because when a team feels they have more work then they can handle they will rush to try and finish it all for the end of the sprint. This is bad because this is where many bugs get created. Pulling stories early on is far better for the team. It is better to under shoot and add extra items at the end of the sprint, then to over shoot and not complete items or have to pull items at the end of the sprint.

I thought this discussion was interesting and made a lot of sense.

March 5, 2008

Better Software No Matter What

Filed under: Agile,Design Principles,Refactoring,SD West 2008 — Brian @ 7:46 pm

The talk from Scott Meyers was excellent. He is a great speaker and really understands the root issues that lead to many quality issues in software. Mark has already posted some comments, so I will just touch on a few common issues Scott kept stressing time and time again.

One of the basic principles of developing quality software is consistency, from the UI to the code and even the unit tests.

Without consistency you will never be moving toward quality software. One of the ways to go about doing this is to provide developers with a set of guidelines and best practices. If developers are not given this guidance they will each develop code and unit tests as they feel the tests should be done, which is fine, but will never result in consistency. To be able to move towards consistency provide the development team with guidelines for: code styles, naming conventions, unit test layout, refactoring standards, etc.

The point he stressed was the most important of all, above everything else is “Interfaces must be easy to use correctly, but difficult to use incorrectly”. He was talking about not just user interfaces, but all the way down through the classes, the methods, the APIs, etc. If the interface is not clear then you have failed to fix the number one issue. If the user interface is confusing, users will leave, if the code is hard to understand, developers will be less inclined to refactor and instead will make the smallest change possible every time which will lead the project further and further from being quality software.

The afternoon talk from Scott Meyers was just as good as the morning talk. I have to say that his talk is one of the most useful presentations I have ever attended, including conferences I have been to in the past. Scott Meyers really knows a lot about software quality and explains it well in his talk. He stresses in the talk about how ALL bugs are an embarrassment, not just big issues but even minor issues that are considered “cosmetic” and often ignored. Minor issues are often the ones that do not prevent a user from performing the task they wish to do, but are enough to cause annoyance and frustration which does not create a good impression of the product.

Blog at