Process, People, and Pods


Showing posts with label OO. Show all posts
Showing posts with label OO. Show all posts

Sunday, 19 August 2007

Keystrokes are NOT Gold

Programmers tend to treat their keystrokes as gold. Perhaps they are driven to counting by project managers. Perhaps their typing skills are atrocious. Regardless, there is a great reluctance to discard code that is wrong. It is wrong because it doesn't work.

A programmer hates being wrong; at least in that respect, a programmer is like every other human being. But in the case of code that doesn't work, there is an almost irrational drive to make it right by adding more code, even if:

  • It means abandoning the design,
  • Coding practices must be abandoned,
  • Complementary changes must be injected into completely irrational areas of system, or
  • Changes to key interfaces must be renegotiated.

Traditional phased process (aka waterfall) tacitly accept and encourage these practices. Testing is done by someone else, usually at some point in the future; this lulls the programmer into thinking the code probably works, and it is someone else's job to validate that assumption. Bugs (the real name of code that doesn't work) maybe fixed by someone else. The chances that I, as the original programmer, might have to make it work are reduced.

But then comes Agile, upsetting this comfortable, lazy acceptance of bad code. Two practices in particular inhibit letting the bad code slide: Test-first development and pair programming.

Test-first has the programmer write the test first, then enough code to make sure it works. Then write another test. The effect of this is that code that doesn't work is immediately recognized. Since all the tests are run before check in (or at least run by automatic continuous integration tools), code that injects a bug somewhere else is detected. But what if the programmer skimps on the tests? What if the boundary conditions are ignored so that personal/team/manager schedule expectations are met?

The second practice, pair programming, makes that more difficult. Now two programmers have to conspire to ignore poor test coverage. But that focuses too much on the negative. Rather with a partner sharing the problem, the second pair of eyes helps identify the marginal cases, and helps find alternative solutions. It becomes almost a mutual quest to solve.

The New Problem of Keystrokes

Agile creates an environment where bugs are rapidly detected. So programmers can be lulled into taking bad code and continuing to change it in the hope that it will (magically) start working. In the ThoughtWorks programmer training course, one objective is to get programmers to overcome their reluctance to throw code away and start over.

The first trick to teach programmers to work in small cycles. In the class, we give a problem to the students (always working in pairs). The students are to write one or more tests and get the tests working. And they have only 15 minutes! Invariably, students start with a test that will prove everything works. When the timer goes off at 15 minutes, they are actually stunned that all the time has passed. They are nowhere near getting it to work. They ask for another 15 minutes, certain that they will finish in that time. Invariably, almost all fail again. Sometimes I give them more time, but it is only to get them to realize that their quest is fruitless. Then you finally have pity on them and show them the secret: write smaller tests. Get something working in the 15 minutes. That "resets" the 15 minute clock.

By the second week of the course, pairs are competing on how many cycles they got done in 15 minutes. They discover that in these tiny steps, they are marching to completing the work with absolute confidence that the solution works when they get there. Indeed, they are stunned when they write that ultimate test of the overall behavior, and it takes just a few lines of code to get it to work. A stark contrast to the frustration they felt when biting off too much.

Now the second trick kicks in. If you can't get the code working quickly, throw it away and start over. In these small cycles, they haven't written that much. The old code worked; the clock is ticking; rather than trying to get the bad code to work, they back up and start over. Code repositories are a huge tool in this; modern tools allow you to revert all the code in a couple of mouse clicks. Now the programmers don't treat that bad code as gold.

The top reasons to discard the code are:

  • It doesn't work. So why are you spending more time creating a bug?
  • If the reason seems to be a muddled design, throw the code away and re-think the design. Kent Beck said, "Code is where design meets the harsh reality of dawn." If the design is bad, the code will scream it out. Listen to the code. So why are you investing more time implementing a bad design?
  • The programmers don't understand the problem yet. Getting stuck can be a sign of a lack of understanding. Step back and discuss the problem some more. So why do you think you will understand the problem better by continuing to write code?

See the trend? Continuing to write code is a waste of time. Discard and start over. It is not gold, but rather .

If you are a programmer, try this out: If the code seems to be going nowhere, throw it out and start again. Let me know how it goes...

Three Critical Enablers for Agile

I have been more successful with Agile processes than any alternative I have tried in my 40 years of programming. My early work predates even phased processes like waterfall. Roy Singham, our founder, often teases me by claiming I invented waterfall. Not true; however, I was one of the programmers IBM experimented on with waterfall in the early 70's. I even did a stint in an IBM programming practices group. So when I say have been more successful with Agile, I have tried quite a few other ways.

I have found that there are three critical enablers for successful Agile projects:

  • Writing code in a style that let's me change it easily
  • Using a compatible management practice (not Gantt charts)
  • Throwing heaps of processing power at it

Writing Code That Can Change

I have previously discussed this in The Secret Assumption of Agile. If you can ignore future requirements while working on the requirement for today, you gain enormous productivity. But you can only ignore the future if your code can tolerate the constant changes demanded by that ignorance of the future.

The premise of Agile is that you absorb requirements as they come in, and you accommodate the change the new requirements demand. Does that mean the code you write today may get thrown away? Possibly. Perhaps, even likely. I estimate that for every 100 lines of code I write, I probably wrote and discarded 200 other lines of code. Wouldn't it be faster if I wrote it right the first time? Sure, but not if I have to think about it. What if in order to get it right, I would have to think about it (design, model, review) for the time it would take me to write it 10 times in Agile? That was my experience in waterfall. We often measured how much of the development dollar was actually spent coding (versus reviewing, being in meeting, creating designs, etc.). The answer was ten cents (10%). If I invented the best programming IDE in the world and could instantly write code in it, I would only save 10% of the programmer cost in a waterfall process. So don't spend too much time worrying about how much code I threw away. Rather, measure how quickly the development was done.

For me to spend most of my time coding, I need to code in a style that will not slow me down when I make changes in the future.

Agile-Compatible Management Process

The second critical enabler for Agile is to realize that common project management techniques are not compatible to Agile. These management practices assume that

Software is like building a bridge.

In other words, it is a step-wise process requiring that each step be completed before moving to the next (or almost completed). Hence I need complete requirements before architecting; complete architecture and design before starting construction; and construction must be done in a precise order (physics requires that the pillars be laid and cured before building a road bed on top of them).

But this doesn't sound like Agile. We are overlapping the phases almost completely. And software is not bound to the physics of civil engineering. It is called soft- for a reason.

Rather, Agile is much more like a production process. Indeed with Agile, I think

Software is liking making cars.

What is the best process for making cars? The Toyota Manufacturing Process, renamed Just-in-Time (JIT) when moving outside Toyota, and subsequently renamed Lean when moving outside of manufacturing.

Look for many more posts on the use of a Lean management process for managing Agile.

For now, consider the use of a Lean management process a critical enable for Agile success.

Processing Power

Agile thrives on processing power. I like using the term horsepower. It directly effects the productivity. Consider the typical use of processing power in Agile:

  • Using OO languages with interpreters and virtual machines
  • Running complete suite of unit tests before checking in
  • Use of powerful refactoring IDEs
  • Continuous builds being run with suites of tests
  • Key code metrics being collected constantly (test coverage, duplicate code checking, etc.)

In an Agile project, you wouldn't choose to not do any of these things. ("Let's use Notepad. It is runs faster than eclipse." "We could use C instead of Java. Java runs too slow on my machine.") When some aspect of the development process is taking too long, the first choice should be to throw more horsepower at it. It is much cheaper than people's time.

Agile in the 70's?

If Agile is so much better, why did we adopt waterfall processes in the 70's? Why didn't Agile emerge earlier? Perhaps we were stupid. I would rather think that we didn't have the enablers:

  1. We didn't have the languages (Java, C#, Ruby) and techniques (OO, rules engines) to write code that could be changed. Our paradigms were actually the opposite: Design your data structures, and the code writes itself.
  2. JIT (Lean) hadn't made it out of the automotive manufacturing sector.
  3. We didn't have the horsepower. I have more memory in my watch than I had in my first computer. And don't think of comparing an entry iPod to the first PC.

So basically, we couldn't do Agile in the 70's. But we can now!

Is your organization still stuck in the 70's?

Thursday, 9 August 2007

Ideal Developer Ratios for Pods

In the last post, I postulated that an ideal team structure exists for Agile, a pod. The pod is a collocated, self-contained team able to execute the project. Let's drill down into the structure of this "ideal" Agile team a bit more, focusing on the programming skills the pod should have. These observations specifically apply to a pod using OO programming techniques (see the secret assumption of Agile) to write code that is easily modified. It is a language independent assumption, whether using Java, C++, C#, Ruby, or any of their brethren.

I will describe the team composition using the skills taxonomy I introduced earlier, the Master-Journeyman-Apprentice model. Briefly, an apprentice is the typically trained programmer; the journeyman has acquired the vocabulary of objects, including principles and design patterns; the master readily crafts object solutions to business problems.

The first rule of thumb for the team is that apprentices when paired together are not productive. Indeed, such a pair can be negatively productive, developing code that must be re-opened and reworked before delivery. (Of course, you do pair within the pod; a solo apprentice is truly a destructive force in the code base.)

So the guideline against apprentices pairing together leads to the first rule of an agile team:

Masters + Journeymen >= Apprentices

Simply put, the number of Masters and Journeymen needs to outnumber the Apprentices so that the Apprentices have a more experienced person with which to pair. That keeps everyone on the team productive. It also provides the mentoring to move the Apprentice to a Journeyman (see Building a Better Programmer). With care, the Apprentices gradually become Journeymen, providing more pairing options or the ability to absorb additional Apprentices.Master programmers provide continuous architectural and design direction to a project. They recognize the need for new components to support emerging stories, and pioneer these new components themselves to ensure a sound base exists for the rest of the team. How many Master programmers do you need? Fortunately, not that many:

Masters >= 10% (Masters + Journeymen + Apprentices)

In my experience, you can get away with as few as 10% of your developers as Master programmers; it is safer if you can work that percentage up to 20%, particularly in the early project stages. That may be the same number of Master programmers, but just keeping the team smaller (with fewer Apprentices early as recommended above.)

Pitfalls of a Team Out-of-Balance

The most common destructive mistake for an Agile team is to have too many Apprentices. Not only do they not produce in pairs, the best you can hope for is that they didn't damage the code base today. As individuals, it is an almost certainty that damage has been done. But the damage goes further than the code base; an Apprentice who hasn't positively contributed to the project is going home not feeling good about themselves. That creates an unnecessary morale problem for the individual and the team.. I don't know of any programmer who goes into work saying, "Today I am going to create some bugs." That this happens is not a failure of the programmer, but in the team creating the situation. So bring on apprentices only at the rate they can be productive to the team; otherwise, be courageous and defer the staffing, ignoring what your spreadsheets are telling you.


Even if you can tolerate 50% Apprentices in theory, you should not start a project with that mix. The early few weeks are the time in which the initial application architecture and design become visible. Apprentices can rarely contribute to these efforts, and can feel useless. There is one exception to this guideline: An Apprentice who has domain knowledge is very valuable to this initial team; providing key insights in what will need to be done in the long run. So start your projects with few, if any, Apprentices, and stage them into the project as the rate you can productively absorb them.

If you are attempting a reasonable size project without Master programmers, you are running a great risk of failure. Master programmers are the key to building the application architecture and design incrementally. Agile works by having such talent embedded in the team. Without it, significant up-front architecture and design is required to provide the road map for the Journeymen.

Many Agile projects seem to start fine, then seem to hit a wall in a couple of months. It is almost like Agile is not working. Refactoring is common, and is being cited more and more often in stand-up meetings. This is actually a sign that the design is weak, a reflection of the lack of continuous guidance a Master programmer provides. So ensure you have sufficient Master programmers on the project, particularly in the formative stages.

The flip side of the Master programmer requirement is that they may not be required in small or enhancement projects. A small project for certain short-term benefits may not reach a crisis point before it is complete. Similarly, if enhancements are being made to an existing, healthy code base, Journeymen are fully capable to following the tracks laid down by previous Master programmers. So Master programmers are not required for small enhancement projects.

Tuesday, 17 July 2007

The Secret Assumption of Agile

I assert that there is a secret assumption behind successful Agile. "Secret" may be a bit too strong, but it is certainly under-articulated if not completely unarticulated.

Agile succeeds when you write code that is easy to change.

I grew up with phased (waterfall) processes. I was taught that a needed design change discovered during the coding phase would cost you ten times the cost of knowing about the need during design. A design problem found in test would cost 100 times the cost of avoiding the problem during the design phase. Based on this cost model (which seemed correct in my experience following it for 20 years), code reviews, design reviews, detailed diagrams, rigorous architecture specifications were all cost saving techniques no sane project would ever do without.

Agile touts that it can easily accommodate change at a reasonable cost. That is, if you knew about the requirement in some detail before the project started, the cost to implement the requirement would be 1x. If you find out about this requirement in the middle of an Agile project, the cost to implement will be about 1x as well (maybe 1.1x, for example) and not the 10x or 100x a waterfall process would predict for a change that would affect the design and adjacent code.

What is going on here? Are all our experiences in waterfall development simply calculation mistakes? Were we using the wrong languages? Were we simply stupid?

"No!" to all of this (although I have had my moments of brilliant stupidity). What has happened is that we have discovered ways to write programs that minimizes the excessive cost of change. I count several technologies that facilitate change, including:

  • Normalized relational databases
  • Rules engines
  • Object-oriented programming
  • SOA (decomposition of an application into loosely coupled entities)

All of these techniques share two aspects in common:

  1. They simplify the problem to allow other sound techniques to be applied, and
  2. They eliminate duplication so that when change is needed, it can be applied to one and only one place with confidence.

My favorite among these techniques is object-oriented programming. By this, I do not mean using an OO language, although that is very helpful. I speak about a style of programming that exploits the fundamental OO concepts.

An example may help here. At a client a few years ago, we were asked to rewrite an existing system. The existing system was a Web-based application, Java platform, with Oracle database backend. The new system was a Web-based application, Java platform, with Oracle database backend. No difference. The reason for the rewrite: The system could no longer be changed in a timely fashion, and business was being lost. The rewritten system has exceeded expectations, met all business needs, and is easy to maintain and enhance (which has been done.) The original system had 79 Java classes. When we finished with the new system, it had 1400 Java classes. We found 1400 different, unique things to model. When a new requirement was being considered, it may impact 20 classes. But the remaining 1380 classes were untouched and ran their unit tests perfectly. Writing code in this particular OO style is different.

Are the Agile pundits (and I am one of them) deliberately hiding this "secret" assumption of Agile? I don't think so. But all the luminaries in Agile that I know have been programming in this style for many years (and most for decades). It is inate to their thinking. When I was exposed to this style of programming 15 years ago, it took a while for me to understand. But after that point, even my Excel macros in VBA are structured very differently.

There will be more discussion of how to learn this style in future posts.

Monday, 16 July 2007

Collocate to Communicate: Part 1

Collocation presents challenges. Kent Beck advocates starting an Agile project by bringing in a wrench (spanner for my UK colleagues) and screwdriver on the first day and fix the furniture. In my experience, fixing the furniture is important, but the furniture police are lurking, and in a large company can outnumber the Agile team. Nevertheless, I have nearly always gotten the team collocated on an Agile project.

The furniture police (I actually have never met one) are not the only inhibitors. Most of the challenge is from recalcitrant team members not wanting to leave the comfort of their little spot in the office. "Oh, where am I going to put my pictures?" "What if the school calls?" (I'll address the abject terror of pair programming later.) Tom Allen of MIT's Sloan School cites a key success factor for getting the team to work in the room in his book Managing the Flow of Technology: Get the manager to move in with the team! So I tend to focus on the manager, and the rest of the team becomes a bit less reluctant.

So why bother? Is it really worth all the effort? Yes!

I recently saw a statistic that indicated that a collocated team gets a 15% productivity boost by doing nothing other than sitting together. That certainly echoes my experiences, and is not surprising. Consider what Tom Allen says: There are three reasons people talk to each other.
  1. They have common interests,
  2. They work for the same manager, and
  3. The chance that you and I talk varies inversely by the square of distances between our chairs. (Tom Allen got tenure for this one; he was formerly an aerospace engineer for Boeing.)

In layman's terms, the last point means that if I double the distance between our chairs, there communication will be reduced by a factor of four. If I double the distance again, the chance of communicating drops by a factor of sixteen. Tom also measured the "intellectual distance" created by a staircase or elevator: 100 yards. Now square that. If the team you work with is on the next floor with a staircase or elevator separating you, you might as well put the team on the other side of the globe. The distance impact is huge.

So taking Tom's rules in mind, how do you manage this situation? The first factor, common interests, you have no control over as a manager no matter how many lunches or recognition events you schedule. Mark that one off.

With the other two, you can play around. A bad solution: Sitting a department that works for the same manager together. Yes, the department will communicate, but they will also point fingers at all the other "idiots" in groups they need to work with, whether it be customers, sponsors, or colleagues that carry a different job title. Yet overwhelmingly, that is what I see.

With Agile teams, I want to rip the team members from their department caves and place them together. They will still talk to their colleagues back in the cave -- that's Tom's second rule. But now they are sitting beside those previously labeled "idiots", and are discovering that they are not so stupid after all.

There are even more benefits I will get into later. But for now:

Collocation is key to successful communication, and communication is critical to Agile success. So I encourage you to persuade, plea, or even bully your management into creating collocation space.