...And How Are You Going to USE That Information?

I cited in my last post a case where Counting Wasted Time. I ended by giving my favorite response when someone is asking to collect data about the project:

...And how will you use that information?

Don't get me wrong. I really want a great answer to that question. I seek metrics that help understand the project. To paraphrase the NRA:

Metrics don't kill projects; people do.

But I digress. Some additional examples of useless information gathering will help you see the importance of the question, "And how will you use that information?"

Example 1: Revised Estimates

In a recent fixed price project, we dutifully re-estimated stories at the start of an iteration. We had release level estimates with which we did our original planning. The iteration estimates were supposed to set our "commitment" level for the iteration; they were supposed to set client expectations.

We over-delivered by 100%. We repeated the exercise, and over-delivered again. So on the third cycle, we asked the question, "How did we use the estimates we made?" The team, including the project manager, acknowledged that we hadn't looked at them since we collected the numbers. So we dropped them.

(BTW as I cited in yesterday's post about Counting, I have rarely found use in resizing just in case things have changed.)

Example 2: Defect Tracking Database

We haven't tracked bugs in the last three development projects I have participated in. How can this be?

One of the key Lean practices is feedback; feedback is most effective when rapidly given. This leads to a positive metric of story cycle time, a measure of the time from the start of analysis until story acceptance by the customer. So if a bug blocks a story, then the clock is running on that story until it is fixed and moves forward. Developers on a Lean-oriented team will attack bugs rather then work on development of new stories. The net results is few bugs live for more than a day. In tracking a recent project, over half the days were finished with zero open bugs.

Instead, we post the bug's existence on a red task card on our standard card wall. When it is fixed and verified, we take the card down. I would like to say that we throw the card away, but we still like to put them into a stack and admire them (for some reason). I do have to admit to a certain curiosity myself. But in the last couple of projects, I have yet to see anyone looking back at those cards!

There are great bug tracking tools out in the market. Many IT shops have standardized on one, and have process guidelines requiring their use. An Agile/Lean project shouldn't need any of them.

[In a bit of cruel fate for a vendor, I was recently asked to give a statement on a recently announced bug tracking tool that purports to solving all your tracking needs; indeed, it is the next generation of tracking tools, rendering all other obsolete. Needless to say, I commented that it was a solution in search of a problem as far as I was concerned. If you need that level of sophistication, your project is going to fail anyway.]

Bad Reasons to Collect Information

There are two reasons for collecting metrics that are, in fact, good reasons not to collect metrics:

Because I might need them: I consider this a euphemism for "I don't know." I lump into this same category, "We always do it this way" and "We have to". In the case of "I don't know", we drop the counting for an iteration or so, and see if we miss it. It is always hard to argue against a short-term experiment. In the case of "we always do it this way", start the discussion to drill down and figure out why we did it that way originally, and see if the reason still exists (especially if you have introduced Agile and/or Lean practices). This is just good development practice, but in particular is a good Lean practice. In the case of "we have to", it is time to engage the process architects and gauge if this is a battle worth fighting. I tend to relish that idea having done the process role in IBM many, many years ago, but we won't discuss my personality defects here. Regardless, the tactic is the same: Find out what it was useful for and see if the need still exists.

I need to understand how well everyone is doing. Translation: "I need to pin the blame on someone when things go wrong." Now you have hit a hot-button of mine. Blame is not a way to run a team. Blame creates fear, and fear kills motivation and productivity. And kills fun! Agile teams, especially collocated Agile teams, have many positive ways to address inhibitors to productivity. This includes many ways of addressing skill deficiencies which will always exist in software projects because of the wicked pace of change. So if someone attempts to collect information for the primary purpose of assigning blame, well, let's just say they have to get past me first.

There Are Good Metrics

Less I have led you astray (and you will use these posts to justify collecting no information), there are many good uses of collected information. I will cite my favorite Agile/Lean project metrics in a future post.

But for now, do you have examples of gathering useless information?

Counting Wastes Time

It's there. You see it. You have this overwhelming urge to count it. It's a moral obligation, a call-to-arms; your personal mission.

I am referring to counting artifacts and measuring artifacts. I am talking about project managers and team leads, and sometimes programmers, testers, and analysts.

And if you stop to think about it, it is a big time waster. A better core philosophy:

"Never count your money while you're sitting at the table. There'll be time enough for counting when the dealings done." -- Kenny Rogers from The Gambler

This thinking is based in Lean, a process borrowed from manufacturing. One of the practices Lean espouses is the identification and elimination of waste. Waste is broadly defined as anything that does not add value to the delivered product.

Agile software development produces loads of artifacts, like stories, tasks, test cases, and code. All of these artifacts have different useful lives. Code endures. Tasks are transient artifacts of development. There is a time and place for counting each. But if you count outside of those boundaries: waste!

Here is a real life example from an Agile project. It occurred between the client project administrator (sort of a cross between a administrator and a project manager) and myself in my role as tech lead and process coach. The discussion was around task cards. A task is a small unit of development work identified by the programmer for a story, which itself has an average of 10 or so tasks, with a huge variance.

• PA: "I need to track task cards."
• Me: "There are going to be a lot of them."
• PA: "I still want to track them."
• Me: "Could you just track stories? Development only takes a couple of days."
• PA: "I still want to track them."
• Me (seeing a pattern here): "Well, if you really want to..."
• PA: "Great. We need to number them."
• Me (I am beginning to catch on): "So that you can track them?"
• PA: "Yes."
• Me: "You are free to put any number on them you want. They're on the wall." (Note the technique of delegating to the person who seems to have all the extra time.)
• PA: "And I need the initials of the pair that worked on them."
• Me (Can't see a quick way around that.) "Okay. We mark the finished as we go. We'll put our initials on it then."
• PA: "I need an estimate for each task."
• Me: "Two-to-four hours."
• PA: "No, I need an estimate for each task."
• Me: "Two-to-four hours. We break the story down into 2-4 hour tasks. That is the estimate."
• [Repeat several more times.]
  
• PA: "Okay, I will mark a range of 2-4 hours. Then I will need the actuals."
• Me: "No." I really try not to use this word with clients, but sometimes there isn't a better word.
  
• PA: "But how can I track how long it actually took?"
• Me (trying to use powerful Vulcan logic): "If we gave you a number, it is only an estimate of the actuals. We have to estimate to accommodate meetings, distractions, bathroom breaks, and the like. Even we don't know how long it actually took."
• PA (after long processing pause): "But I need actuals."
• Me (feeling a little guilty suggesting it): "You can track how long it takes to do the story, and just divide it."
• PA: "Okay, I guess that will work."

I was being a bit unfair to our project administrator. The whole conversation would be rendered moot as soon as she saw the number of tasks that were created, started, and completed each day (about 20-25 on an average day). The counting only lasted a week.

Most consultants would (and do) succumb to the pressure to count. Some may even advocate it. Frankly, I exploited my gray hair and used my authority voice to carry the day in this case.

Imagine the overhead if this process had been established? Even a few minutes per task add up when there are 3000 tasks on a project. And the killer question for this:

...And how will you use that information?

How I Discovered Lean

Before I delve more into Lean, the best management practice to support Agile, I would like to step back and talk about how I got to Lean. The path I took influences my thinking about the Agile/Lean integration. If you want to skip this post, that is fine. It adds little to the overall discussion.

I have been writing code a long time. I was around when waterfall was introduced as a management process. Two things happened to me in the late 80's:

• I had a chance to study Just-In-Time (JIT) when getting my Masters degree, taught by a leading authority, and
• A colleague from IBM Research gave me a copy of Smalltalk.

I found both topics fascinating, but did not connect them at the time.

The Lean (JIT) Path

Lean is the new name for JIT, a more appropriate name as JIT moved from manufacturing into other sectors. In my graduate studies, I actually wrote a paper for applying JIT to software testing. The thesis was a focus on waste.

When doing testing under waterfall at the time (20 years ago), specialized testers take the code base and run their predefined manual scripts. As a bug was found, it was documented extensively, then the tests were continued. Often, another bug was found quickly. It was documented as well, and tests were again continued. And so on.

As a recipient of these bugs (ie, the programmer that injected the bug in the first place), the first bug was usually real. The second bug, however, was often the same bug just showing up again. As I analyzed the bugs, it would get more and more difficult to discern whether a bug was really a new bug or not. I just wanted to throw those later reports away. But the process dictated that I close the bug as "unable to reproduce" or "duplicate" to complete the bug cycle.

Armed with JIT and its idea of waste, I saw the situation in a different light. The tester was creating work for me by testing after finding the first couple of bugs. It was unlikely these bugs were new. Regardless, I would probably ask the tester to reconfirm the bug after fixes to the earlier bugs. This was all waste!

Rather, the tester should stop testing after finding a couple of bugs. (At the time, I suggested 3 might be the magic number. Now I lean more toward 1 is enough.) My professor loved the idea; the first time I tried it with real testers, they were horrified! The testers were measured on completely running the tests (whether there were bugs or not). They get credit for finding bugs. Did I want them to just sit around and do nothing? Surely, it was more productive for them to at least try the tests.

But it wasn't more productive for the team for them to try more tests. They were better off just sitting around (or better yet, test something else). After a month or so of agony, they began to understand the message, dumped their original metrics, and starting becoming more effective testers.

The OO Path

Although I had early exposure to Smalltalk, my current job didn't let me do anything except play around with it. I do remember loving the introductory books and the cool tools. But when it came time to actually solve a problem with it, I was absolutely clueless.

When I went out on my own as an independent, I finally began to understand the paradigm. For me, the trick was coding GUI interfaces. In most UI frameworks, the system calls the code when something needs to be done. The application is not in control -- the user is! Once I got this reversal, the message-passing metaphor of Smalltalk began to click. I eventually wrote commercial Smalltalk, found a mentor who was part of Ward Cunningham's original team, and began to really learn Object-Oriented Programming. This was still, however, a long 5-6 year process.

Then Came Agile...

I was just starting an OO consulting gig when I attended OOPSLA '98. At a birds-of-a-feather session, Kent Beck, Ron Jeffries, Ward Cunningham, Martin Fowler and others presented eXtreme Programming (XP). There was no book, just a one page handout of the 9 practices and a diagram of how they related to each other.

Having been raised in the Ward Cunningham School of Objects, it instantly clicked. My client agreed to these ideas, and we embarked on an XP project.

I realized quickly that we were in a production process. And Lean was a perfect fit for code production under Agile. We started focusing on waste and cycle times. We immediately started running cycles faster than the stock XP suggested, simply because we could.

I have exclusively done Agile projects since then (1998). I continue to apply more and more Lean management principles as I go.

A Different Path to Agile-Lean

Mary and Tom Poppendieck lecture and write about Lean being applied to Software Development. They describe it very well in their two books. Mary got to the Agile-Lean by a completely different route.

What is marvelous is that we (and many others) have gotten to the same point. Lean is the appropriate management model for Agile.

Lean - An Agile Management Process

In Three Critical Enablers for Agile, the second critical enabler is use of a management process consistent with the Agile engineering practices that continually produce software. For this, I use Lean, a manufacturing process borne in Toyota and responsible (IMO) for the current dual dominance of Japanese companies in both automotive production and quality.

Think Production, Not Building

I cannot over-emphasize how radical project management is when shifting to Lean. Of all the Agile roles, the role of project manager changes most in Agile. It dwarfs the impact Agile has on the development, testing, and analysis roles. But to gain the benefits Agile promises, changing the thinking around project management is critical.

Agile focuses on continually producing working software. Traditional phased software development (aka waterfall) focuses on moving the project steadily forward one phase at a time, doing its best to ensure everything is right before building upon that phase in the next.

Traditional project management is based on traditional engineering; civil engineering is my favorite analogy. Software is like building a bridge. You need to know what is needed (a bridge). You need architecture (suspension bridge). You need design (number and spacing of struts). You need to build it in a specific order (base before pillars; pillars before road surface). You need to test it (weight bearing tests, plus inspections). Waterfall software processes follow this model. The prevalent tool for traditional project manager is the Gantt chart, a schedule of activities and dependencies that can track progress.

Before disparaging waterfall too much, it is worth saying waterfall works. In my years at IBM, I participated in many waterfall projects that delivered quality software on time and on budget. Indeed, I have been a part of a thousand programmer project spanning three years and delivering. So don't count me among those who say waterfall is bad. I remember the true chaos before waterfall.

But as I believe Agile is a better process than waterfall (and I have tons of experience with each), I believe that Lean is a better management process for Agile than foisting traditional project management onto an Agile project.

Lean management practices focus on production; this factor alone makes it a good fit for the constant production of software characteristic of Agile. But Lean management differs significantly:

• No useful information can be captured in Gantt charts
• It is not important what someone will be working on next week
• Individual assignments are subsumed by team efforts

Basically for the traditionally trained project manager, it is bizarro-land. Round is square; down is up; when means why; and who means when.

But all is not lost. Lean brings powerful new tools to the project manager, tools that give insight into the project success throughout the project. At this point, I feel completely naked on a project without these tools. They are my new security blanket. These Lean processes and tools:

• Track production, giving you new insight into the real progress
• Show skill imbalances with clear direction on actions
• Remove much guesswork out of the question, "Are we going to ship on time?"
• Provide superior techniques for productive feedback to individuals
• Support dynamic adjustments to schedule, content, and resource

Sound too good to be true? Consider that we will be stealing most of these techniques from manufacturing management practices that are at least as old as the engineering practices we use today. We are not inventing new things here, but rather borrowing from a different source, manufacturing instead of civil engineering.

No Management Alternative

I have seen many (and heard of many more) Agile projects that recognize the uselessness of most of the traditional project management practices. There solution: abandon the practices altogether, but substitute nothing else. Everyone is a professional, after all, and they will be doing their best.

For a short time, this seems to work well. But for a longer project, someone will start to count the cost at some point, and ask the question about when it will be done, how much more will it cost, and do you really need all that staff that is working on it now.

All these are valid questions. A matching management process should be able to answer these questions without over-burdening the team. I believe the Lean management processes do just that.

Key Manufacturing Practices That Are Borrowed

I focus on three key things to borrow from Lean to aid in managing Agile software development:

• Feedback is the source of quality,
• Identification and elimination of waste, and
• Reduction in the number of specialists.

Each of these will be the subject of future posts.

Are you using non-traditional management practices in your Agile team?

Some Agile/Lean References

This is just a quick pointer to a couple of Web items.

• Applying Agile to Ruby is a presentation I did last year at JAOO in Denmark. This Webcast has been nicely edited by the InfoQ guys for viewing. It is not just about Ruby, but about real Agile development practices using complementary Lean principles.
• Boost productivity with lean software development is a survey article on the use of Lean in the UK written by freelance technical writer Cath Jennings. Not only are some of my comments there, but also comments from David Norton of Gartner, a gentleman who really "gets" Agile and Lean.

Keystrokes are NOT Gold

Programmers tend to treat their keystrokes as gold. Perhaps they are driven to counting by project managers. Perhaps their typing skills are atrocious. Regardless, there is a great reluctance to discard code that is wrong. It is wrong because it doesn't work.

A programmer hates being wrong; at least in that respect, a programmer is like every other human being. But in the case of code that doesn't work, there is an almost irrational drive to make it right by adding more code, even if:

• It means abandoning the design,
• Coding practices must be abandoned,
• Complementary changes must be injected into completely irrational areas of system, or
• Changes to key interfaces must be renegotiated.

Traditional phased process (aka waterfall) tacitly accept and encourage these practices. Testing is done by someone else, usually at some point in the future; this lulls the programmer into thinking the code probably works, and it is someone else's job to validate that assumption. Bugs (the real name of code that doesn't work) maybe fixed by someone else. The chances that I, as the original programmer, might have to make it work are reduced.

But then comes Agile, upsetting this comfortable, lazy acceptance of bad code. Two practices in particular inhibit letting the bad code slide: Test-first development and pair programming.

Test-first has the programmer write the test first, then enough code to make sure it works. Then write another test. The effect of this is that code that doesn't work is immediately recognized. Since all the tests are run before check in (or at least run by automatic continuous integration tools), code that injects a bug somewhere else is detected. But what if the programmer skimps on the tests? What if the boundary conditions are ignored so that personal/team/manager schedule expectations are met?

The second practice, pair programming, makes that more difficult. Now two programmers have to conspire to ignore poor test coverage. But that focuses too much on the negative. Rather with a partner sharing the problem, the second pair of eyes helps identify the marginal cases, and helps find alternative solutions. It becomes almost a mutual quest to solve.

The New Problem of Keystrokes

Agile creates an environment where bugs are rapidly detected. So programmers can be lulled into taking bad code and continuing to change it in the hope that it will (magically) start working. In the ThoughtWorks programmer training course, one objective is to get programmers to overcome their reluctance to throw code away and start over.

The first trick to teach programmers to work in small cycles. In the class, we give a problem to the students (always working in pairs). The students are to write one or more tests and get the tests working. And they have only 15 minutes! Invariably, students start with a test that will prove everything works. When the timer goes off at 15 minutes, they are actually stunned that all the time has passed. They are nowhere near getting it to work. They ask for another 15 minutes, certain that they will finish in that time. Invariably, almost all fail again. Sometimes I give them more time, but it is only to get them to realize that their quest is fruitless. Then you finally have pity on them and show them the secret: write smaller tests. Get something working in the 15 minutes. That "resets" the 15 minute clock.

By the second week of the course, pairs are competing on how many cycles they got done in 15 minutes. They discover that in these tiny steps, they are marching to completing the work with absolute confidence that the solution works when they get there. Indeed, they are stunned when they write that ultimate test of the overall behavior, and it takes just a few lines of code to get it to work. A stark contrast to the frustration they felt when biting off too much.

Now the second trick kicks in. If you can't get the code working quickly, throw it away and start over. In these small cycles, they haven't written that much. The old code worked; the clock is ticking; rather than trying to get the bad code to work, they back up and start over. Code repositories are a huge tool in this; modern tools allow you to revert all the code in a couple of mouse clicks. Now the programmers don't treat that bad code as gold.

The top reasons to discard the code are:

• It doesn't work. So why are you spending more time creating a bug?
• If the reason seems to be a muddled design, throw the code away and re-think the design. Kent Beck said, "Code is where design meets the harsh reality of dawn." If the design is bad, the code will scream it out. Listen to the code. So why are you investing more time implementing a bad design?
• The programmers don't understand the problem yet. Getting stuck can be a sign of a lack of understanding. Step back and discuss the problem some more. So why do you think you will understand the problem better by continuing to write code?

See the trend? Continuing to write code is a waste of time. Discard and start over. It is not gold, but rather .

If you are a programmer, try this out: If the code seems to be going nowhere, throw it out and start again. Let me know how it goes...

Three Critical Enablers for Agile

I have been more successful with Agile processes than any alternative I have tried in my 40 years of programming. My early work predates even phased processes like waterfall. Roy Singham, our founder, often teases me by claiming I invented waterfall. Not true; however, I was one of the programmers IBM experimented on with waterfall in the early 70's. I even did a stint in an IBM programming practices group. So when I say have been more successful with Agile, I have tried quite a few other ways.

I have found that there are three critical enablers for successful Agile projects:

• Writing code in a style that let's me change it easily
• Using a compatible management practice (not Gantt charts)
• Throwing heaps of processing power at it

Writing Code That Can Change

I have previously discussed this in The Secret Assumption of Agile. If you can ignore future requirements while working on the requirement for today, you gain enormous productivity. But you can only ignore the future if your code can tolerate the constant changes demanded by that ignorance of the future.

The premise of Agile is that you absorb requirements as they come in, and you accommodate the change the new requirements demand. Does that mean the code you write today may get thrown away? Possibly. Perhaps, even likely. I estimate that for every 100 lines of code I write, I probably wrote and discarded 200 other lines of code. Wouldn't it be faster if I wrote it right the first time? Sure, but not if I have to think about it. What if in order to get it right, I would have to think about it (design, model, review) for the time it would take me to write it 10 times in Agile? That was my experience in waterfall. We often measured how much of the development dollar was actually spent coding (versus reviewing, being in meeting, creating designs, etc.). The answer was ten cents (10%). If I invented the best programming IDE in the world and could instantly write code in it, I would only save 10% of the programmer cost in a waterfall process. So don't spend too much time worrying about how much code I threw away. Rather, measure how quickly the development was done.

For me to spend most of my time coding, I need to code in a style that will not slow me down when I make changes in the future.

Agile-Compatible Management Process

The second critical enabler for Agile is to realize that common project management techniques are not compatible to Agile. These management practices assume that

Software is like building a bridge.

In other words, it is a step-wise process requiring that each step be completed before moving to the next (or almost completed). Hence I need complete requirements before architecting; complete architecture and design before starting construction; and construction must be done in a precise order (physics requires that the pillars be laid and cured before building a road bed on top of them).

But this doesn't sound like Agile. We are overlapping the phases almost completely. And software is not bound to the physics of civil engineering. It is called soft- for a reason.

Rather, Agile is much more like a production process. Indeed with Agile, I think

Software is liking making cars.

What is the best process for making cars? The Toyota Manufacturing Process, renamed Just-in-Time (JIT) when moving outside Toyota, and subsequently renamed Lean when moving outside of manufacturing.

Look for many more posts on the use of a Lean management process for managing Agile.

For now, consider the use of a Lean management process a critical enable for Agile success.

Processing Power

Agile thrives on processing power. I like using the term horsepower. It directly effects the productivity. Consider the typical use of processing power in Agile:

• Using OO languages with interpreters and virtual machines
• Running complete suite of unit tests before checking in
• Use of powerful refactoring IDEs
• Continuous builds being run with suites of tests
• Key code metrics being collected constantly (test coverage, duplicate code checking, etc.)

In an Agile project, you wouldn't choose to not do any of these things. ("Let's use Notepad. It is runs faster than eclipse." "We could use C instead of Java. Java runs too slow on my machine.") When some aspect of the development process is taking too long, the first choice should be to throw more horsepower at it. It is much cheaper than people's time.

Agile in the 70's?

If Agile is so much better, why did we adopt waterfall processes in the 70's? Why didn't Agile emerge earlier? Perhaps we were stupid. I would rather think that we didn't have the enablers:

1. We didn't have the languages (Java, C#, Ruby) and techniques (OO, rules engines) to write code that could be changed. Our paradigms were actually the opposite: Design your data structures, and the code writes itself.
2. JIT (Lean) hadn't made it out of the automotive manufacturing sector.
3. We didn't have the horsepower. I have more memory in my watch than I had in my first computer. And don't think of comparing an entry iPod to the first PC.

So basically, we couldn't do Agile in the 70's. But we can now!

Is your organization still stuck in the 70's?

Ideal Developer Ratios for Pods

In the last post, I postulated that an ideal team structure exists for Agile, a pod. The pod is a collocated, self-contained team able to execute the project. Let's drill down into the structure of this "ideal" Agile team a bit more, focusing on the programming skills the pod should have. These observations specifically apply to a pod using OO programming techniques (see the secret assumption of Agile) to write code that is easily modified. It is a language independent assumption, whether using Java, C++, C#, Ruby, or any of their brethren.

I will describe the team composition using the skills taxonomy I introduced earlier, the Master-Journeyman-Apprentice model. Briefly, an apprentice is the typically trained programmer; the journeyman has acquired the vocabulary of objects, including principles and design patterns; the master readily crafts object solutions to business problems.

The first rule of thumb for the team is that apprentices when paired together are not productive. Indeed, such a pair can be negatively productive, developing code that must be re-opened and reworked before delivery. (Of course, you do pair within the pod; a solo apprentice is truly a destructive force in the code base.)

So the guideline against apprentices pairing together leads to the first rule of an agile team:

Masters + Journeymen >= Apprentices

Simply put, the number of Masters and Journeymen needs to outnumber the Apprentices so that the Apprentices have a more experienced person with which to pair. That keeps everyone on the team productive. It also provides the mentoring to move the Apprentice to a Journeyman (see Building a Better Programmer). With care, the Apprentices gradually become Journeymen, providing more pairing options or the ability to absorb additional Apprentices.Master programmers provide continuous architectural and design direction to a project. They recognize the need for new components to support emerging stories, and pioneer these new components themselves to ensure a sound base exists for the rest of the team. How many Master programmers do you need? Fortunately, not that many:

Masters >= 10% (Masters + Journeymen + Apprentices)

In my experience, you can get away with as few as 10% of your developers as Master programmers; it is safer if you can work that percentage up to 20%, particularly in the early project stages. That may be the same number of Master programmers, but just keeping the team smaller (with fewer Apprentices early as recommended above.)

Pitfalls of a Team Out-of-Balance

The most common destructive mistake for an Agile team is to have too many Apprentices. Not only do they not produce in pairs, the best you can hope for is that they didn't damage the code base today. As individuals, it is an almost certainty that damage has been done. But the damage goes further than the code base; an Apprentice who hasn't positively contributed to the project is going home not feeling good about themselves. That creates an unnecessary morale problem for the individual and the team.. I don't know of any programmer who goes into work saying, "Today I am going to create some bugs." That this happens is not a failure of the programmer, but in the team creating the situation. So bring on apprentices only at the rate they can be productive to the team; otherwise, be courageous and defer the staffing, ignoring what your spreadsheets are telling you.

Even if you can tolerate 50% Apprentices in theory, you should not start a project with that mix. The early few weeks are the time in which the initial application architecture and design become visible. Apprentices can rarely contribute to these efforts, and can feel useless. There is one exception to this guideline: An Apprentice who has domain knowledge is very valuable to this initial team; providing key insights in what will need to be done in the long run. So start your projects with few, if any, Apprentices, and stage them into the project as the rate you can productively absorb them.

If you are attempting a reasonable size project without Master programmers, you are running a great risk of failure. Master programmers are the key to building the application architecture and design incrementally. Agile works by having such talent embedded in the team. Without it, significant up-front architecture and design is required to provide the road map for the Journeymen.

Many Agile projects seem to start fine, then seem to hit a wall in a couple of months. It is almost like Agile is not working. Refactoring is common, and is being cited more and more often in stand-up meetings. This is actually a sign that the design is weak, a reflection of the lack of continuous guidance a Master programmer provides. So ensure you have sufficient Master programmers on the project, particularly in the formative stages.

The flip side of the Master programmer requirement is that they may not be required in small or enhancement projects. A small project for certain short-term benefits may not reach a crisis point before it is complete. Similarly, if enhancements are being made to an existing, healthy code base, Journeymen are fully capable to following the tracks laid down by previous Master programmers. So Master programmers are not required for small enhancement projects.

Ideal Skill Ratios in Pods

So are, we have talked about pods, ideal team Agile team structures, and the proper mix of developer skills in the pod. In this post, I would like to describe the needs for other skills in the pod.

An ideal pod would contain:

• 4-6 pairs of developers (8-12 total)
• 2-3 business analysts
• 2-3 testers
• 1 ambassador (aka project manager or iteration manager)
• 1 customer or customer representative (rotating on need)

So adding to our pod formulas:

Analysts + Testers + Managers + Customers = 80% (Developers)

So if I calculate how many programmers a project should need, I add another 80% to cover the other roles. So between this ratio and the productivity uplift I give to the programmer skills (5:2:1 for Masters, Journeymen, and Apprentices), I have the tools for doing my Agile project plans for a pod. Of course, this is just the starting point for the plans. I adjust skills constantly throughout the project.

For example in the first release of the project I am currently working on, we ran it successfully with 3 pairs of developers, two analysts, a single tester, and a single project manager. The next release had the same 3 pairs of developers, but only a single analyst and a single tester; metrics showed we should have had a second tester, however. Overall, however, the project is successfully running light on non-developer resources.

On the other hand, an early project on which I worked at ThoughtWorks ran its last release with one project manager, one combined customer/business analyst, 3 pairs of developers, and over a dozen testers! The customer was writing the stories herself, perfectly able to keep work flowing to the developers. The developers were a Master-heavy mix, and produced accordingly. This high productivity drove the need for up to 17 testers at one point; no investment in automated acceptance tests had been made.

Despite these examples to the contrary, I like the above ratios for planning. It is my starting point, and I adjust accordingly.

Pods -- Ideal Agile Team Structures

I described in a previous post that you probably have the wrong team working on your Agile project. It is wrong because you can't find the ideal team members initially, it is wrong because the needs of the project change as the project progresses, and it is wrong because the members of the team themselves will grow and change. The core recommendation from that post was to constantly tweak your team.

Typical, productive Agile teams seem to conform to a certain shape. This observation is borne from my experience in almost a decade of Agile projects, and is reinforced by colleagues at ThoughtWorks from their experiences. I am calling such ideal Agile team structures pods. I offer yet-another-term for several reasons:

• The term team doesn't convey enough,
• I haven't read of an alternative term,
• Several other good words are already in use,
• I like the collocation connotation of pod,
• I like the self-contained connotation of pod (i.e., all the needed skills are collocated)
• It provides an organization unit for broader people-oriented discussions
• There is a quirky sci-fi angle I can't resist

I will be exploring many aspects of pods in a series of posts. Let me quickly summarize where this is going:

• There is a particular mix of developer skills in a pod (a mix of Masters, Journeymen, and Apprentices;
• There is a particular mix of team skills (analysts, developers, testers, management);
• The team is self contained and collocated;
• The team is empowered, making decisions in its areas of competence;
• The team is self-managing (including self-apprasing);
• Team metrics provide insight into the effectiveness and composition of the team; and
• The team is semi-permanent, spanning project life cycles.

Much of this is common sense. Much of this is obvious to practioners of Agile. But much like the practices of XP, the total is greater than the sum of the parts. I still strive to create the perfect pod for projects on which I work, but I haven't yet achieved it.

Do you have specific characteristics for your ideal Agile team?

You Have the Wrong Team

As a project manager, you have just staffed your project. Sorry to be the bearer of bad tidings, but you have the wrong team. If you an experienced project manager, you already know this. But you did the best you could from the resources you had available. It is the wrong team because:

• You are missing the experience you need in some key areas, and
• The team members don't necessarily work well together.

Agile practices help here, particularly the practices of collocation (and you do sit with your team, don't you?) and pair programming. If you are missing experience, collocation and pair programming lets you grow that experience by judicious pairing. Knowledge transfer is quite rapid and efficient in this environment. Collocation also dampens down the "finger pointing" that often plagues teams. ("He doesn't understand!", "She is so stubborn!", and the like.) It is much more difficult to talk behind someone's back when she is sitting in the room with you.

If team members still have difficulty working together, your being in the room provides insights to the solution. Is it a matter of tone (challenging, dismissive, personal rather than business)? Then coach on interaction style. Is it disagreement on technical matters? Ensure swift resolution before it becomes personal.In this area, Agile practices can be a disadvantage:collocation and pair programming will exacerbate negative relationships. Whereas you might be able to ignore or defer action when each team member sits in their own cubes, such deferrals are quite destructive in an Agile team with its close proximity.

Net-Negative Producing Programmers (NNPP)

Agile luminaries have coined the term NNPP to identify team members whose overall impact to the project is negative. That is, the work they do is less than the harm they do to their colleagues' productivity. An NNPP may be productive outside of an Agile environment. In a cube, their griping has no listeners. There challenging, terse tone goes unheard. But given their visibility in a collocated environment...

I recently saw on the Web a report purporting to show the impact of one bad apple on the rest of the team. A single individual typically can negatively impact other team members numbering in double digits! So indeed, one bad apple can spoil the whole bunch.

There are several ways to identify such individuals:

• Observe. Being in the room, you can see the behavior. Even more important, you can see the impact on overall team. The situation can be corrected by coaching, or ultimately, removal.
• Ask the team. The team knows who is acting incorrectly. Borrow a lesson from Survivor, and let the team vote someone off the island.
• Who gets picked last? If pairing is practiced, who is the person no one wants to pair with?

If you fail as a project manager to act quickly on identified NNPP, morale issues will quickly ensue. Your inaction, in particular, will be glaringly obvious to the team. Your leadership, dedication, and even competence will come under question.

Once more for emphasis: Failure to address an NNPP problem will rapidly destroy team productivity.

The Continuing Issue

The real fatal flaw in staffing projects, and Agile projects in particular, is not realizing that the project staffing needs change throughout the project. Projects that acquire their staff, and then stick with that staff throughout, will incur excessive costs if not fail outright. Staffing must be adjusted throughout the project.

A few years back on an eight-month Agile project, I made 10 staff changes in the first 6 weeks. The core development team had 16 different participants, but only 4 were on the project for the entire time. Rather than being a disaster for such team instability, the project set high water marks for productivity and quality that have yet to be touched.

Why does staffing need to be adjusted throughout a project? It is for several reasons beyond staff quitting and needing to be replaced:

• The needs of the project change throughout the project. DB tuning will be important at some stage, but not others. Knowledge of systems with which the project much integrate will be needed at some time. The general load of analysis, development, and testing will drift during the project.
• Team members will grow during the project, particularly on projects lasting more than six months. Exploiting this growth is part of good team management. Again from the above example, I reduced the number of developers to six toward the end of the project. They produced so much code, however, it took 17 testers to keep up. The developers skills had increased that much.
• An inbred team will cease innovating. Studies of innovation reveal a significant drop in innovation in a team that has been together for as little as six months. New blood is needed, even if only to challenge the patterns of current thinking.

As a project manager, you must constantly be alert to tweaking the team. You may raise eyebrows by constantly shifting your team composition; ignore them. When the need for an expert has passed, roll her off the project. Bring in a specialist when the need arises; exploit collocation and pair programming to harvest the knowledge and release the specialist. Perhaps that resource you really needed was not available at the start of the project. Bring them on board when they are available.

Successful team tuning requires building a strong relationship with two groups. First, be willing to share your experts with your colleagues. Hopefully, they will reward your good will with the loan of a needed expert of their own. Don't hoard resources when the need has passed; rather, release them to reduce your costs and maintain the morale of the resources. Second, build great relationships with your staffing or resource management team. Send them flowers or buy them a case of beer (or both); they will start to call you when a resource becomes available that you might use.

Application development is chaos; react or face corresponding chaos in your team.

Building a Better Programmer, Part 2

To increase the productivity gains promised by the Master-Journeyman-Apprentice (MJA) model, programmers must progress up in the model. In Part 1 of Building a Better Programmer, we detailed the techniques by which programmers can gain the skills resulting in improved productivity. In this post, I will discuss paths to programmers enrichment, likely timeframes for the process, and failure rates. I will again draw from the MJA model for learning OO skills, a key enabler for Agile application developement.

Apprentice to Journeyman

If you can move your Apprentices to Journeymen, you can effectively double your productivity for those complex, business critical systems you maintain and enhance. There are certain enablers that are required for this transition:

• Willing Apprentices (supported in their efforts by management)
• Journeymen or Masters from which to learn

I have used a couple of paths to assist the transition of programmers from Apprentices to Journeymen, both in ThoughtWorks and in the industry with good results in both cases. For OO programming, the goal is the same in both cases: Teach the Apprentice to think like an object programmer. That means teaching the Apprentices to break problems down in a different way, to recognize when programmers are structured poorly per OO principles, and build up a repertoire of techniques by which to do this.

Path 1 involves a combination of training and mentoring. Training assists in quickly providing the new vocabulary for the Apprentice, and mentoring provides the continued support for applying the new techniques to the daily problems the programmer addresses. Without mentoring, the value of the training is significantly diminished. Without mentoring, the pressures of delivery will drive the Apprentice to apply the old, familiar techniques to solve the problem. Old habits are reinforced; new techniques are forgotten. And worse, the new techniques become labeled "theoretical" or "academic" or "irrelevant to real problems." Net result: education wasted.

Path 2 forgoes the training, and simply provides the mentoring. The training is gained by working closely with Journeymen and Masters. Pair programming is important in this environment; collocation is absolutely essential. Path 2 takes longer; the Apprentice must pick up the vocabulary in the course of the work. Path 2 is probably more expensive; the productivity gains come later, and the time of Journeymen and Masters is taxed in doing bits of training here and there as the need arises.

There is a Path 3: Pick it up from reading the books. Self-taught Journeymen are rare, but I have met them. They also took a frighteningly long time to make the transition, with a lot of discarded bad code in the wake. I cannot recommend this approach for even the brightest of programmers.

If you can adopt Path 1 (training and mentoring), you should expect most of your programmers (60% or more) to make the transition from Apprentice to Journeyman in about six months. Some will be there quicker; some will take a bit more time. Expect failures; in my experience, about 20% cannot make the transition at all. The individuals who will not make the transition can be identified pretty quickly (a month or less); they can be placed back in the roles they had before and continue to be productive for the organization. The usual cause of the failure to transition is that they lack the conceptual skills to be effective OO programmers. I have seen a few other cases of failures in senior people. In these cases, the senior person realizes that the junior people understand this better than they do, and choose to disengage rather than reveal the lack of a gap. I take these failures personally; I feel I didn't create an environment where they could learn safely.

I developed specific training for this transition over a decade ago. The first class affectionally called it OO Bootcamp. I brought the course with me to ThoughtWorks several years ago, and it still carries that moniker today. I will post more on OO Bootcamp in the near future.

Journeyman to Master

The transition from Journeyman to Master is much more difficult. You should not expect more than 20% of your staff to ever make the transition. Most of your good candidates have deviated from strict application development, and are probably serving as your system and application architects, effectively ceasing their learning as programmers.

But with the whopping productivity boost and the ability to leverage Masters to make more Masters, it is well worth investing in their growth.

Masters must be excellent programmers. Their vocabulary is deep and wide. By and large, they have extensive experience applying their knowledge to real world problems. Similarly, their conceptual skills are excellent. This allows them to manipulate objects in their heads, forming solutions to difficult problems. Finally, domain knowledge is helpful, but not quite essential.

If you describe a business problem to a Master programmer, she will see objects dancing in her head, forming patterns that solve the problem. This comes from developing an inherent sense of object relationships, a suite of problem-to-object-solution maps, and confidence born of solving similar problems before.

Hence, the transition from Journeyman to Master can take a while. In general, expect a couple of years. I have seen a few instances of faster transitions (only six months or so); in all cases, the individuals were extremely bright, conceptual thinkers, and had many years of programming experience under their belts. (They were also colleagues from ThoughtWorks India!)

Managing Programmer Improvement

In the prior post, I listed the techniques to enable programmer skill growth. In order of importance, they are pair programming, collocation, communities, and training. Project managers and senior managers should facilitate these practices as much of this as possible.

There is one other charter I would give a project manager when a focus on skills is important: Bring back better programmers. In other words, measure the project manager on both delivery and skills growth. If the project manager delivers the project but doesn't grow the people, then, well, at the least a permanent productivity boost to the organization has been lost; at worse, gross negligence has occurred, and I don't want to use that project manager again.

Building a Better Programmer, Part 1

Programmer productivity varies much more widely than project management spreadsheets would suggest. Differences within a moderately sized team can be 5:1, 10:1, or higher, discounting any incompetent programmers. In a previous post, I discussed how to assess the skills of the individuals. In yet another post, I extolled the more expensive talent was actually a huge bargain with their productivity boost.

But knowing the skills, are you stuck with this team? Is your only option as a project manager to swap team members out to improve your mix of skills? The answer is a resounding no. You can improve your mix of skills.

One of the attractions of the Master-Journeyman-Apprentice (MPA) model for me is that it clearly indicates the path to improving the skill mix:

• Apprentices need constant supervision as they begin to acquire skills and apply them for the first time. Gross, project-impacting mistakes await unsupervised Apprentices trying to apply their new knowledge for the first time. (Would you let a first year med student perform open heart surgery on you?) The natural supervisors? Journeymen (and not project managers!)
• Journeymen should choose the Masters from whom they will learn. It was the case with craftsmen of the Middle Ages, and is just as true today. They need exposure to such people to improve their skills to the next level. Self-taught Masters are quite rare; more often, Master programmers I know learned at the hands of another Master programmer.

Another alternative model for Master/Journeyman relationship come from academia: The relationship between PhD's and their doctorial candidates at universities. The doctorial candidate selects a professor under which to study. In exchange for guiding the doctorial candidate, the professor gains additional research capacity in his field of study. When the professor feels the student is ready, he presents him to a committee of other PhD's for examination. This committee decides whether the candidate is qualified to join their ranks. So a self-controlled Academy of Master Programmers is another interesting model.

It behooves the project manager to create an environment to supporting this learning. The rewards for supporting this learning are large productivity boosts without a corresponding growth in costs (yes, you should pay your Journeymen and Masters more, but they will be happy with less than 100% or 500% raises).

Enabling elements (in order of importance):

• Pair programming. Not only is pair programming as productive as two individuals working alone, it creates an amazing learning opportunity by pairing Apprentices with Journeymen, and Journeymen with Masters.
• Collocation. Conversations cannot occur unless people are speaking distance from each other. Throw in a convenient whiteboard, and knowledge transfer will take place.
• Project teams. Project teams that are deliberately formed to expose key individuals (Journeymen) to a Master programmer will create communication between them; knowledge will grow.
• Communities. Whether formal or informal, encourage and support communities around the skills (OOP, database, dynamic languages, .NET, etc.). Communities are quite a bit less effective than the above alternatives, primarily because they involve participants choosing to participate on their own time. This caters to a subset of candidates only.
• Training. Training is largely ineffective for all but Apprentices. But for Apprentices, it inbues them with new terminology and techniques. Hands-on training is much preferred to lectures. All training is ineffective if the techniques learned cannot be immediately applied to current project work. There is near zero retention for material that is first used a year after it is taught. Consider this when staff can sign up for education, but will have little opportunity to use the material in their normal job.

A subsequent post will talk about specific paths of transition from Apprentice to Journeyman, and from Journeyman to Master.

Masters, Journeymen, and Apprentices: Part 3

In the previous blogs on this topic, I claimed that there is a huge productivity difference among Masters, Journeymen, and Apprentices when working on significant, business critical systems:

Role	Productivity
Apprentice	1x
Journeyman	2x
Master	5-10x

This is observed productivity from many Agile projects.

So, what do you do with this information?

1. Assess the skills of your developers
2. Adjust project staffing, absorbing all the Masters and Journeymen you can find (the rate difference does not equate to the productivity difference)
3. Work to boost the skills of your developers

Let's start with the first point: Assessing the skills of your developers. For this example, these guidelines here apply to OO skills in particular, but you can easily morph these into other skills.

• Apprentices are those programmers that are not yet Journeymen. You do expect Apprentices to be competent in a programming language and associated tools (IDE, unit test tools, debuggers). They can write code without fumbling and resorting to syntax guides. They can use the right tool at the right time. They may be learning what bad OO code looks like, but lack the tools to figure out how to get rid of it.
• Journeymen have developed a broad OO vocabulary. They understand OO principles, and apply these principles in their code. They have a rich set of design patterns they exploit, and a broader set they understand. They can quote solutions to code smells drawing from their refactoring vocabulary. They are clever users of their tools, exploiting the nuances missed by Apprentices. Journeymen stuggle, however, when trying to design an object solution to a business problem. They are not yet comfortable with exploiting their OO design skills to address these problems.
• Masters are Journeymen with the extra ability to decompose a business problem into its parts. They also discern refactoring paths for ugly code, particularly where a series of patterns must be applied. The source of their productivity lies in their ability to find a great solution (simple and elegant) quickly, whereas Journeymen may have a few false starts.

So how do you assess where each of your developers fit? The easiest way is to have a Master programmer work with each programmer for a while (as little as 30 minutes). That will be enough for a Master to discern the current level of a programmer. You might also get a feel for the potential of a programmer in that short time, but that is a bit more dangerous.

You don't have a spare Master programmer around to do the assessment? For Apprentices and Journeymen, there is a good alternative. Since the difference is largely knowledge based, testing is a valid approach. Test their vocabulary; show them bad code, and ask how they would fix it. Personally, if a programmer can describe the Composite design pattern to me with three or four places where it would be appropriate, I am pretty sure he is at least a Journeyman.

How about Masters themselves? That is a bit tougher, but I would propose the following: Master programmers know Master programmers when they meet them. They talk a different language; they discuss problems in higher, abstract terms. They can relate experiences using almost any design or refactoring pattern. Verify that the legendary programmer in your organization is a Master programmer by getting references from luminaries in the field. Once you have found one, let her figure out who the others are. Chances are she knows a couple, and those couple know the rest.

Warning: Don't be surprised if you have no Master programmers. Don't be surprised if some of your architects think they are, but are really not.

More in the next post about growing Journeymen and Masters.

Masters, Journeymen, and Apprentices: Part 2

I previously suggested that classifying programmers as Masters, Journeymen, and Apprentices provided a better way to understand and manage the productivity differences among programmers. The analogy to craftsmen of the Middle Ages is deliberate:

• It reinforces the image that programming is a learned craft, not some special science. I am definitely in the camp of thinking with The Pragmatic Programmers.
• Skills like OOP proficiency seem to be learned best from others; books faire poorly in teaching these skills. This aligns with the craft model well.
• It defines some tiers of productivity, but not too many to make planning difficult.
• The level of independence (or rather, the need for supervision) is modelled well. Apprentices need constant watching; journeymen do not.
• Journeymen in the Middle Ages would find masters under which to study. There was an implicit agreement to learn and teach that was largely voluntary on both parties. I like the self-selected learning concept.
• It is a skill classification, not a job title. This aligns better with Agile and Agile teams.

My colleagues have suggested a few nice refinements or alternative systems.

One idea I like is breaking the Apprentices into two classifications: Those who are learning the skill, and those who are not (because of ignorance or just not choosing to invest in that particular skill). This segregation aligns with the different treatment of the two groups. Learners are encouraged, nurtured, and provided with opportunities to learn. The non-participants are educated in the advantages of the skill to solicit their participation.

Another colleague, Dan North of ThoughtWorks, uses a three-tier system based on his school of Japanese martial arts: Shu, ha, re, with re corresponding to apprentices. In his school, the lowest tier, re, focus on the mechanics of the moves: blocks, counterpunches, avoidance. Once the body learns the moves, the mind moves toward applying them in situations. This is the domain of the ha level of competence. Practioners achieving the shu level are innovating; in an unexpected situation, these practioners may conceive of new techniques on the fly because of their fundamental understanding of the principles.

The appealing aspect of Dan's model is that ha are expected to teach re, and shu are expected to teach ha. Indeed, you cannot achieve the shu level unless you show you are also competent to convey the knowledge to others. Dan's model reflects my belief that some programming skills (like OOP proficiency) are most easily conveyed in person, not by books.

Masters, Journeymen, and Apprentices: Part 1

I have proposed the heresy that all programmers are not created equal. I further dissed most typical correllations to programmer productivity (title, salary, etc.) may not be useful in predicting differences in productivity. Thus I have turned to a different system.

I classify developers as Masters, Journeymen, or Apprentices. This is a taxonomy I picked up from Ken Auer of RoleModel Software; I think he got if from a former company he worked at, KSC. But I have seen other references in the literature to it as well. The model refers to craftsmen of the Middle Ages, but applies to many disciplines today that involve learning a skill.

Briefly, an apprentice is a developer who is still learning the requisite skills. Constant supervision is necessary because an apprentice may not know when she/he is doing something wrong. Techniques are being learned, but the why of the technique is still a bit vague. Stringing together techniques is a bit beyond the apprentice.

A journeyman has mastered the techniques of the craft. Supervision is rarely required. Given a task, journeymen can complete it in reasonable time with quality results. But they are not yet masters.

A master has all the journeyman skills, but also is able to innovate within the craft. Given a new problem, the master can conceive of a solution. Then the journeymen can be turned loose to help him implement it.

In an Agile team, the skill I am looking for is object-oriented programming (OOP) because Agile hinges upon the ability to write code that can change. Most programmers don't know how to do that, particularly with object-oriented programming. Don't confuse object-oriented programming with object-oriented languages like Java, C#, C++, or Ruby. Using one of those languages (sadly) does not ensure an understanding of object-oriented programming.

An apprentice in OOP is a effective programmer and can construct loops, methods, and classes, but is still learning how to separate the business problem into objects. Encapsulation is being learned (which means no getters or setters for the classes). Code is being placed into the appropriate classes rather in giant god class. if statements litter the code as the apprentice is still learning design patterns to eliminate them. Class sizes and method sizes tend to be larger than smaller. Code requires comments to aid in understanding, and blank lines are common in the middle of methods to group code together. An apprentice is beginning to understand the patterns of bad code (the imfamous code smells), but lacks the tools to understand how to eliminate them.

The journeyman is adept in OOP techniques. The classes exhibit crisp conceptualizations distinguishing them from each other. Boundaries are clear; methods are small. Design patterns are common, and exist in inverse proportion to the number of if statements. else statements are quite infrequent, and case statements have vanished. (I write about one case statement every 18 months or so.) Given a problem and a broad outline of a design, journeymen will implement it with little need for supervision or support.

A master has all the above skills, plus a near magical ability to hear a problem and see objects for a solution. They never seem stuck, and always seem to have a solution to the code smell that is stumping the journeyman. A quick conversation with a master can remove a barrier slowing down a journeyman, or reset the direction being taken to save significant effort.

So why bother with this classification? The corresponding productivity model, particularly on significant, business critical applications. Let's say a regular programmer produces at 1x. An apprentice is essentially a 1x programmer. Then a journeyman is a 2x programmer. There is that much difference in learning those OO techniques. But a master... Well a master is a 5x-10x programmer! They can literally run circles around an apprentice. Even more important, when an apprentice can't get anything done because of the complexity of the problem, the master cuts through it like a hot knife through butter.

A true example from a former client project: In the third release of a project, we reduced the team size to six programmers. These six programmers were keeping 17 acceptance testers hopping just to keep up! The client was running around the halls trying to find additional testers. The six programmers: four masters and two journeymen, a virtual team of around 40 in productivity. No wonder it took so many testers to keep up.

So in my resource planning spreadsheets, I literally identify apprentices, journeymen, and masters as separate resources, and use 1x, 2x, and 5x multipliers to get a virtual body count. I use the virtual body count to compare to non-OO waterfall estimates of the same work. And I never count a swap of a master for a journeyman.

More musings on the M-J-A model in later posts...

The Secret Assumption of Agile

I assert that there is a secret assumption behind successful Agile. "Secret" may be a bit too strong, but it is certainly under-articulated if not completely unarticulated.

Agile succeeds when you write code that is easy to change.

I grew up with phased (waterfall) processes. I was taught that a needed design change discovered during the coding phase would cost you ten times the cost of knowing about the need during design. A design problem found in test would cost 100 times the cost of avoiding the problem during the design phase. Based on this cost model (which seemed correct in my experience following it for 20 years), code reviews, design reviews, detailed diagrams, rigorous architecture specifications were all cost saving techniques no sane project would ever do without.

Agile touts that it can easily accommodate change at a reasonable cost. That is, if you knew about the requirement in some detail before the project started, the cost to implement the requirement would be 1x. If you find out about this requirement in the middle of an Agile project, the cost to implement will be about 1x as well (maybe 1.1x, for example) and not the 10x or 100x a waterfall process would predict for a change that would affect the design and adjacent code.

What is going on here? Are all our experiences in waterfall development simply calculation mistakes? Were we using the wrong languages? Were we simply stupid?

"No!" to all of this (although I have had my moments of brilliant stupidity). What has happened is that we have discovered ways to write programs that minimizes the excessive cost of change. I count several technologies that facilitate change, including:

• Normalized relational databases
• Rules engines
• Object-oriented programming
• SOA (decomposition of an application into loosely coupled entities)

All of these techniques share two aspects in common:

1. They simplify the problem to allow other sound techniques to be applied, and
2. They eliminate duplication so that when change is needed, it can be applied to one and only one place with confidence.

My favorite among these techniques is object-oriented programming. By this, I do not mean using an OO language, although that is very helpful. I speak about a style of programming that exploits the fundamental OO concepts.

An example may help here. At a client a few years ago, we were asked to rewrite an existing system. The existing system was a Web-based application, Java platform, with Oracle database backend. The new system was a Web-based application, Java platform, with Oracle database backend. No difference. The reason for the rewrite: The system could no longer be changed in a timely fashion, and business was being lost. The rewritten system has exceeded expectations, met all business needs, and is easy to maintain and enhance (which has been done.) The original system had 79 Java classes. When we finished with the new system, it had 1400 Java classes. We found 1400 different, unique things to model. When a new requirement was being considered, it may impact 20 classes. But the remaining 1380 classes were untouched and ran their unit tests perfectly. Writing code in this particular OO style is different.

Are the Agile pundits (and I am one of them) deliberately hiding this "secret" assumption of Agile? I don't think so. But all the luminaries in Agile that I know have been programming in this style for many years (and most for decades). It is inate to their thinking. When I was exposed to this style of programming 15 years ago, it took a while for me to understand. But after that point, even my Excel macros in VBA are structured very differently.

There will be more discussion of how to learn this style in future posts.

Collocate to Communicate: Part 1

Collocation presents challenges. Kent Beck advocates starting an Agile project by bringing in a wrench (spanner for my UK colleagues) and screwdriver on the first day and fix the furniture. In my experience, fixing the furniture is important, but the furniture police are lurking, and in a large company can outnumber the Agile team. Nevertheless, I have nearly always gotten the team collocated on an Agile project.

The furniture police (I actually have never met one) are not the only inhibitors. Most of the challenge is from recalcitrant team members not wanting to leave the comfort of their little spot in the office. "Oh, where am I going to put my pictures?" "What if the school calls?" (I'll address the abject terror of pair programming later.) Tom Allen of MIT's Sloan School cites a key success factor for getting the team to work in the room in his book Managing the Flow of Technology: Get the manager to move in with the team! So I tend to focus on the manager, and the rest of the team becomes a bit less reluctant.

So why bother? Is it really worth all the effort? Yes!

I recently saw a statistic that indicated that a collocated team gets a 15% productivity boost by doing nothing other than sitting together. That certainly echoes my experiences, and is not surprising. Consider what Tom Allen says: There are three reasons people talk to each other.

1. They have common interests,
2. They work for the same manager, and
3. The chance that you and I talk varies inversely by the square of distances between our chairs. (Tom Allen got tenure for this one; he was formerly an aerospace engineer for Boeing.)

In layman's terms, the last point means that if I double the distance between our chairs, there communication will be reduced by a factor of four. If I double the distance again, the chance of communicating drops by a factor of sixteen. Tom also measured the "intellectual distance" created by a staircase or elevator: 100 yards. Now square that. If the team you work with is on the next floor with a staircase or elevator separating you, you might as well put the team on the other side of the globe. The distance impact is huge.

So taking Tom's rules in mind, how do you manage this situation? The first factor, common interests, you have no control over as a manager no matter how many lunches or recognition events you schedule. Mark that one off.

With the other two, you can play around. A bad solution: Sitting a department that works for the same manager together. Yes, the department will communicate, but they will also point fingers at all the other "idiots" in groups they need to work with, whether it be customers, sponsors, or colleagues that carry a different job title. Yet overwhelmingly, that is what I see.

With Agile teams, I want to rip the team members from their department caves and place them together. They will still talk to their colleagues back in the cave -- that's Tom's second rule. But now they are sitting beside those previously labeled "idiots", and are discovering that they are not so stupid after all.

There are even more benefits I will get into later. But for now:

Collocation is key to successful communication, and communication is critical to Agile success. So I encourage you to persuade, plea, or even bully your management into creating collocation space.

All Programmers are NOT Created Equal

All programmers are not created equal. There. I have said it. It is our equivalent to saying, "The emperor has no clothes."

There is much evidence that, indeed, all programmers are created equal. My exhibit for this: hundreds of thousands of spreadsheets built by project managers. Okay, maybe I will count that as hundreds of thousands of exhibits. I have even contributed at least several hundred of them myself. It doesn't make it true, no matter much it makes the spreadsheet easier to use.

Okay, so it may not be true that all programmers are equal. But it is it approximately right? In my experience (many, many years) on real projects with real delivery deadlines, there is an order of magnitude difference between programmers (that is, 10x difference). Even after throwing out the incompetent programmers (who produce zero), the really good programmers are ten times better than the really mediocre programmers. So even in approximate terms, all programmers are not equal.

Ask a programmer you know about this. (You may be a programmer yourself.) Do they know a really good programmer? How good is she/he? You will likely hear of legendary feats, impossible tasks done in a weekend, feats of heroism worthy of superhero status.

So as a project manager, you should seek these resources out. The "one" on your spreadsheet can become a virtual "ten". That is contingency, risk management, and a cheap insurance policy, all rolled up into "one".

How do you find these programmers for you project? Sorry, more bad news here for the project manager: These resources don't automatically correlate to most of your company metrics.

• Some correlation with experience: Look for years programming; discount years of being an architect or any other non-programming role that doesn't involve personally writing code.
• Negative correlation with years since directly writing code for the job.
• Slight correlation with title/level: Does your organization have tiers of programmers (Dev 1, Dev 2, ...)? Higher levels are more likely to be stronger programmers. If your company has a true meritocracy (versus pure seniority system), a top-tier programmer must have been good at some point to be promoted. So there is a bit better chance that it is still true. Caveat: Negative correlation with years since directly writing code for the job.
• Negative correlation with non-programmer titles: Speciality titles are signs that (1) the skill is now gone, or (2) "Those that can't program, [fill in your favorite job title]". Be wary of architects, particularly if they come with adjectives (web-services architect). Designer titles are a little less risky, but have the same flaws. Ask the key question, "Do you write code?" Run away from any answer like, "I am past that" or "I leverage others".
• Some correlation with salary: The better programmers are probably paid more. Someone has recognized that they are worth holding onto. Most organizations are not as dumb as we think about matters of survival.

So, if all these indicators are not so good, how do I as a project manager know I am getting at least my fair share of these super-programmers? Easy. Ask the programmers. You may not recognize their legendary feats, but they know about them. You may need to prod them a bit; you may need to tolerate some geek-speak. Nod knowingly, and listen for the names. Another way to prod: Ask them, "If you were marooned on a island with the natives that are cannibals, and you had to program your way off the island, who whould you want to be there to help you?"

How many of these super programmers do you need? As many as you can get. What happens if you don't have them? Well, that is a topic for another day.

[This is part of the People thread of this blog. See The 3 P's of Agile: Process, People, and Pods for the overview.]

The 3 P's of Agile: Process, People, and Pods

This is the first post on the topic of Process, People, and Pods. This series of blogs will explore Agile programming beyond the traditional boundaries of "an alternative to waterfall".

So where does this name come from?

Process refers to the Agile alternative to a phased process (aka waterfall). But the process discussion will go beyond that. Agile has its roots in the Just-In-Time (JIT) manufacturing process; Lean is the more popular name for these techniques as this thinking explodes past the manufacturing sector. So Process will also address management techniques that promote Agile versus traditional engineering techniques like Gantt charts that cripple Agile productivity.

People addresses the different way to think about the most important component of software development: The people that conceive it, create it, and make it available for use. Let me summarize how radical the Agile thinking should be here. If you don't believe that waterfall is the right methodology, what do you think about people that are titled after phases of waterfall? In People discussions, we will expose the flaw in thinking of about people as architects, analysts, designers, developers, and testers. Not only is Agile productivity rooted in breaking this thinking, there are positive moral benefits to be gained.

Pods takes the people point to the next level. If the job titles are wrong, what about how we organize departments around these titles? Pod refers to an ideally structured Agile team. If the name invokes B-grade sci-fi flicks, you have caught a bit of the roots of the name. It also comes from thinking of the synergy of a bean pod full of seeds. In Pod discussions, we will discuss ideal Agile organizations. What about the career tracks we define? I would suggest that Agile teams are organized differently. Agile teams should act differently, be trained differently, and be appraised differently.

I look forward to feedback on all of this!