Jag Venugopal's Blog

June 22, 2009

Agile Software Development: Avoiding Sucker’s Choices

Filed under: Information Technology,Project Management — Jag @ 7:41 pm

“Agile” seems to be one of the more misused terms in software engineering these days. The Manifesto for Agile Software Development defines agile as any process that emphasizes:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

 These points are like mom and apple pie. There can be no arguing the ideals that they represent. No one in their right mind would consider processes and procedures more important than individuals on a team. Neither would any sane person pat themselves on the back for well-written documentation if the software did not work. Only the absolutely lunatic (or staff at software monopolies) would consider customer collaboration irrelevant. Similarly, any project manager worth his salt ought to be able to respond to change in a manner that meets business needs.

 However, in many situations, the meaning and intent of “agile” has been perverted to an ad-hoc, laissez faire approach to project implementation that pays little heed to either managing to a plan (schedule, cost) or any predictable software development process. In my personal experience with various allegedly agile methodologies, and their equally dubious practitioners, the ideal is great – it sells a lot of books and conference seats, but falls short in providing working systems. What follows is my take on how the agile manifesto is morphed into software engineering absurdity, and what to do about it.

Properly Documented Requirements

 I know of one “agile” project manager that believes business analysis is a profession doomed to extinction. His preferred method of requirements capture is an object model (in Visio), supplemented by meeting minutes. The jury’s still out on his project, but all of my experience points in a different direction — well-written and to-the-point requirements.

 Sure, writing requirements takes time. But more often than not, it not the writing that takes time (team members with good skills can whip out large documents in half a week); what does take time is the process that happens before the writing. Such things as user workshops, focus groups, interviews, issue follow-ons, hunting for redundant or contradictory requirements, etc.

 A second pet peeve of mine, which is more magnified in an agile environment is requirements being slapped together to meet arbitrary deadlines, without much thought to how well-written and internally consistent they are. A project that I was peripherally involved in produced such voluminous requirements (mostly because they didn’t bother to write it well) that it was all discarded as being un-reviewable, un-maintainable, and impossible to code from.

 When someone insists that they prefer working code to elaborately-written documentation, they’re giving you a sucker’s choice. No one insists on thick binders of documents; instead, what I am talking about is requirements documented by competent business analysts, using the fewest words possible, and yet conveying the intent. This document can and must be reviewed and signed off by the customer. It educates the rest of the team, including QA specialists, technical writers, and project managers.

 To WBS Or Not To WBS

Any team, whether agile or not, needs to be able to provide a clear work breakdown structure — a high-level WBS for the entire project, and a detailed WBS for the current iteration. The WBS must contain a hierarchical structure of tasks, with proper dependency relationships between them.

 The presence of a properly thought-out WBS provides a level of assurance that the project team knows what it is doing; what tasks are needed to achieve a goal, and what the ordering of goals is. A hierarchical structuring to the WBS allows for tracking the completion of intermediate items during the iteration. Being able to demonstrate these intermediate items allows the team to prove that it is on track. Well-thought out predecessor and successor relationships ensure that the team is working on those tasks which can be driven to completion (ones for which all predecessor tasks have completed). This helps prevent the 90% completion syndrome where each task quickly reaches a reported 90% level of completion and stays there for the entire iteration (and often beyond), because upstream tasks are still incomplete.

 Arguments are made by agile proponents that such WBS creation and dependency identification is a waste of time. My response is that a well-prepared team ought to know what it needs to do, or the preparation is insufficient for a short-duration iteration. If the team is not able to provide a well thought-out WBS with clear explanations of each work item, then it would make sense to spend an initial 5-10% of the iteration duration in figuring out such a WBS.

 Estimates for Individual Tasks

Once a WBS has been created for an iteration, the team ought to be able to estimate each individual task. Estimates are never perfect, but an imperfect estimate, made in good faith, is much preferable to none at all. Estimates allow for the project team to confirm that their WBS is achievable within the desired completion dates of the iteration. When resources are named to each task, the estimates also provide a good idea of resource loading. This helps prevent situations where resources are overbooked on multiple simultaneous tasks, or are underutilized. As estimates and actuals are collected, they can be reviewed by the team to identify over- or under-estimation, and to appropriately recalibrate.

 Digression: Three Point Estimates Aren’t

There is literature suggesting that a three-point estimate can be better than a single-point estimate. In a three-point estimate, a developer provides an optimistic, most-likely and pessimistic estimate for each task. These are averaged out using a formula with some weightings to come up with a single point estimate. The argument goes that this approach forces a developer to think through all possibilities rather than the most optimistic scenario alone.

 The theory sounds good, and the aims are noble. However I have not seen this being effective, in practice. When a developer has fixated on a number as an estimate, I have found that they generate the other two points using math, rather than thinking through all scenarios. For example, a developer may decide that coding a module takes 4 days, and wants the average to show up as such. Rather than give much thought to the optimistic and pessimistic scenarios, the developer merely finds two numbers equidistant from their estimate (say, 2 days and 6 days), so that the average turns out to be their preferred number.

 The project manager is sometimes faced with an argument from “agilists” that the iteration deadline is known, and the WBS is known, so could the PM please back off and let the team do its work. There is a grain of truth to their complaining — estimating takes thinking and work. However, it is an investment worth making, if only to confirm that the iteration can be successful. Absent proper estimates, the project manager is flying blind, relying only on assurances that the iteration goals can be achieved in the promised timeframes. This may work if the iterations are very small (say <= 2 weeks). In such small iterations, it immediately becomes clear if the team is having trouble meeting its goals in a timely manner, thus allowing for remedial action to be taken. However, the longer the iteration period, the greater the risk that the team may experience an unpleasant surprise. An iteration of one month’s duration that takes a month and a half has just lost 15 days on the overall project. With proper estimation, this might have been avoidable.

 Project Tracking

Tracking the status of tasks on a weekly basis can be a lightweight effort for individual developers, yet can provide assurance that the project is on track. In my projects, I ask developers for just two key pieces of information for any task that they have worked on during the week:

  • Number of hours worked on the task
  • Number of hours remaining on the task

 For a well-prepared developer who knows what they are doing, both numbers should be easily trackable in under 15 minutes per week. This allows the project manager to determine actual task completion percentages, and roll it all the way up the WBS. This method of calculating task percentage complete is far better than asking the developer directly for a task’s completion percentage, the numbers are concrete and meaningful, rather than having to guess a completion percentage. If, week after week, the remaining hours on the task are steady, then a conclusion can be drawn that the estimates were poor, or that the developer is having trouble completing the task. This allows for remedial action to be initiated.

 In case of outsourced resources, rigorous project managers correlate hours worked reported each week to contracting invoices. If a developer bills for 40 hours, then it is easy to determine if they have demonstrated 40 hours of progress on the project, using this tracking method. This allows for better cost and vendor management, than just completion percentages. It may seem excessive to apply the same tracking to one’s own employees’ hours, but I’ve seen it done in consultancies that live and die by the billable hour.

 An easier approach to project tracking is to break up each task into no more than 20 hours’ effort. When a task is started, the project manager marks it as 50% complete. When it is completed, then and then only, is it marked 100% complete. There are no other levels of completion. Using this approach, each task can be completed in one, or at most, two reporting periods.

 Project tracking using one of the above two approaches will allow the project manager, whether the project is traditional or agile, to accurately track and report on project progress. Of greater importance, it surfaces problems early and provides an opportunity to address the underlying issues. Such tracking is initially met with resistance in settings where the team is not accustomed to it. This resistance is usually dissolved by promising the team that the tracked hours will only be used to update the project plan, and will not affect such areas as pay and promotion (obviously, the PM will need to abide by their promise). Another approach to dissolving resistance is to seek the buy-in and advocacy of the line managers, to whom the project resources report.

 Responding To Change Over Following A Plan

Inherent in the title is yet another sucker’s choice. It is indeed possible, even imperative that a competent project manager follow a plan, yet be flexible enough to respond to change, when the circumstances demand it. To make this an either/or choice is to trap the unsuspecting project manager into having no scope control. The right word to have used instead of “over” in the section title is the word “and”.

 Responding in a deliberate way to change is easier when there is a change management plan (with due apologies to the agilists). Any change management plan should address:

  • Detection of the change — how do we know that a change has occurred? How can we prove that it is a change?
  • Analyzing the business impact of the change
  • Analyzing the overall project impact of the change
  • Getting the change approved/rejected
  • Planning for the change
  • Implementing the change

 I’ll talk about a change management process in another post — the only point I want to make here is that responding to change does not need to be difficult. When requirements are written well, and they have been signed off, it is relatively simple to detect and discuss changes — in a planned way.


The agile manifesto, and indeed the proponents of various agile and extreme methodologies present many sucker’s choices. In this post, I’ve tried to debunk some of them. The responsibility of the project manager when faced with a sucker’s choice is to replace the word “or” in the question with “and”, and then go about finding a solution. While elements of the agile movement have their merit, one must acknowledge that other practices and beliefs run contrary to common sense. It is up to the project manager to pick the good, discard the bad, and move on.


Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: