What is a differentiator then? A differentiator is a set of features that gives a product a unique capability. Take Google Docs as an example. In terms of feature count it cannot compete with Microsoft office. The key differentiator that incites people to use Docs is collaboration. In Google Docs two people can work from different locations on the same document at the same time without loosing edits. Without this differentiator Docs would never have taken off the way that it did. Microsoft is fighting back with Office2010 which promises to include similar capabilities, but it has taken years to rework the Office products in this direction.

If you want to compete with an incumbent it is much smarter to find and exploit a differentiator than it is to compete on feature parity. It’s kind of like the tortoise and the hare, you can get to the finish line before your competition realizes that the race has started.

One reason great ideas take so long to spread is that they are often misunderstood and miscommunicated as more and more people try to adopt them. John Seddon has some damning criticism of the way that Deming’s ideas have been peverted by a number of Lean consultants. By copying lean manufacturing practices instead of lean thinking, they have managed to make lean synonymous with mindless standardization in some quarters. Seddon shows that you need a very different set of tools when applying Lean ideas to the service sector as compared with manufacturing.

Great ideas are also susceptible to the tendency to create a religion of sorts around them (and the people behind them). Great ideas are meant to be the building blocks of even greater ideas, not an endpoint. People often quote Deming as if his statements are facts simply because he uttered them. That is a disservice to both Deming and the reader. It also ignores the fact that the world that Deming was part of has changed.

The 94% quote

A prime example of the deification of Deming is what can be called the “94% is the system” quote. When people discuss how best to change an organization, Lean proponents will invariably cite Deming and argue that since he has shown that 94% of the potential for improvement is in the system there is little point in working with organizational culture. This is disingenuous for two reasons. First of all, Deming was not being precise when he said this. The actual quote is:

“I should estimate that in my experience most troubles and most possibilities for improvement add up to proportions something like this: 94% belong to the system (responsibility of management), 6% special” (Out of the Crisis, p. 315)

The first thing that is apparent here is that Deming’s thinking is a lot better than his english… More to the point is that he is obviously not saying that 94% is an accurate measurement. If you think about it, there is no way Deming could have measured the relative importance of these things with this kind of precision.

The second (and most important) reason you should treat this quote with caution is that Deming was speaking of one particular context. Most of Deming’s work is in manufacturing. Even if 94% was the correct number in manufacturing, it is very unlikely that all other types of business follow the same rule. The relative importance of “the system” must vary depending on what your context is.

If lean is ever going to become more mainstream people must start treating it less as a religion with its own gods and more as a collection of insights that have to be carefully tailored to the context you are working in.

I would not be surprised if words such as “chaos” and “naive” cross your mind. After all, requirements are an integral part of the way people usually define a project. Even if you have adopted agile development you will probably think that this is going too far. Agile is (among other things) based around the notion of change and incremental corrections of strategy based of feedback. Agile practitioners will often advocate writing less detailed requirements and accepting that these will change as the project progresses. Less detailed requirements is one thing, removing them altogether is something completely different.

The what and how

Consider the statement “The customer describes what is needed and the supplier describes how to solve this”. This is intuitively correct for most people. Intuition often allows us to make correct decisions much faster than if we have to use reasoning. The problem with intuition is that it is not always right. Something that is intuitively right can still be very wrong. At the core of the statement above is the idea that what you want can be completely separated from how this will be satisfied. The customer uses information about the business to arrive at what is wanted and the supplier translates this into a working solution. The problem with this setup is that “what” can never be separated from “how”? Let me show you why…

Time travel

Lets suppose that you are in a meeting room. You are part of a team that is formulating requirements for a system that is to be used by criminal courts to handle trials. It is vital that this system is able to store information about each trial for future reference. But what about video? Should the system store a video recording of each trial? My guess is that after some deliberation you and the team will agree that this is either a “must have” or a “should have”. There are obvious benefits and the costs should not be that large.

Now here is the twist. When you entered the meeting room you were transported back in time forty years. You think it is 2010 but the other people in the room know it is 1970. After you get over the strange hair styles and clothing you focus on the requirements. You suggest to the others that video recordings of each trial should be included as a requirement. The mood changes. The rest of the group looks at you as if you are completely insane. There is no way they are going to agree to your idea.

This twist can of course be applied the other way. When you entered the room you were transported forty years into the future. Now, when you suggest that video could be a good idea you get equally incredulous looks, but for the opposite reason. In an age where the cost of video capture and storage is zero there is no point in discussing whether to have it.

The implications

Maybe you feel that this video example is unfair. After all, there are other requirements that are viable today and forty years either way. Maybe so, but would these requirements be valid fifty years back? Requirements are often “obvious” when we take the technology behind them for granted. We all “know” that storage of snippets of text is incredibly cheap. This has been true for a long time. But what about the things that are just now becoming easy to do? As technology progresses, customers risk missing opportunities by not including requirements that they think are too expensive. Other requirements may be outdated. How many systems being build today have the probably needless requirement of supporting fax?

Alternatives to requirements

What is the alternative? I suggest using an unlikely source for inspiration, the advertising business. In advertising, a customer typically does not write a contract for a single project. The contract is instead the foundation of an extended relation. The advertiser works with the customer, looking at what the customer does and what the customer wants to achieve in the future.  The advertiser then formulates ideas that are elaborated on together with the customer. Some of these ideas result in “projects”, others are more open ended. Over time the customer uses a number of (imperfect) measures to assess how well the advertiser is helping it achieve its goals. Most of the goals a customer cares about are not related to a particular project, they are formulated in terms of the business itself.

I wonder what the software business would look like if it worked in this way. Customers would not write requirements. They would describe what they were doing today and what their goals were for the future. If a customer was contracting an external supplier, the selection criteria would primarily be based on the track record of that supplier. Suppliers would actively work with customers in order to figure out what to do. This kind of operation would demand a different kind of supplier. One that is not focused on “the project”. One that knew that IT systems can cause problems as well as solve them. And maybe most importantly, one that quickly identified developments in IT that the customer should take advantage of.

Update: Martin Fowler has written a post that ties in nicely with this one.

Eric is half right

Many replacement projects are started for the wrong reasons. People seem to make up these reasons more as rationalizations than as real arguments. They will also base their decisions on impossibly optimistic estimates of the costs involved. These projects should be converted into enhancement projects that build on the legacy system in an intelligent way.

Eric is half wrong

But there are also many replacement projects that are necessary. The reasons vary but one stands out in my experience: a core architectural design parameter has been invalidated since the system was built. Two common examples of obsolete design parameters are:

• The system was built around the notion of not overloading the computing power of the server. Data is collected during the day but most of the processing is done at night in batch operations. With modern servers this design has become superfluous. People have come to expect that systems process information immediately.
• The database is designed with the primary goal of using as little space as possible. The design tends to be inflexible and cryptic. Fields are often used for multiple purposes. With the costs of storage plummeting there is no reason to conserve disk space in this way.

In these kinds of situations it can be exceedingly difficult to align the system with the way one wants to do business. The system becomes a constraint that prevents the business from exploiting new possibilities. Customers start to notice that you are operating in an outdated manner: “You will have to wait until tomorrow before your changes will be visible”, “Describe your problem in 200 characters”.

An alternative approach

The interesting question to ask is “How do you plan a replacement project so that it stands a chance of succeeding?”. The answer is in the strategy you use. The core mistake most replacement projects make is that they try to replace the legacy system in a “big bang” style. This leads to a “at least as good as the old system” mentality. Scope is contingent on an unknown quantity, the full feature set of the legacy system. In addition, the project has no way of getting real feedback since nothing is released before “everything” is completed.

There are two general approaches (and an infinite number of combinations of these) that are far superior to the big bang approach:

• Treat the replacement project as an enhancement project for as long as possible. Build new components that cooperate with the legacy system. Gradually replace enough of the legacy system until the business can bear the pain of retiring it.
• Create a completely separate system and use it for a small subset of the business. A common way to do this is to route new business of one type to the new system. Grow the system so that it gradually can handle more and more types of business.

There are many possible ways to pursue these approaches. Each legacy system provides a unique context that heavily influences how you should tackle the problem. The Agile Release Strategy wiki describes a large number of possible approaches together with their inherent strengths and weaknesses.

The hard truth

There are some core truths that you must face if you want to succeed with a replacement project:

• Pain is a necessary and useful part of a replacement project. Make sure that the business understands and accepts this from the outset.
• A certain amount of investment in the legacy system will be absolutely necessary. Do not see investments in the legacy system as waste, see them as insurance.

Replacement projects are hard, but with the right strategy you can lower the risk of failure to an acceptable level.

The second talk is a 10 minute lightning talk called: Feil utrullingsstrategi dreper prosjekter. (Poor release strategy kills projects). This is based on my work with the Agile release strategy wiki.

• The product owner does not have enough time and is often not available
• The product owner is not able to make decisions or decisions made are quickly overruled

So why do so many product owners struggle to fill their role? The answer lies in how organizations select the product owner. Most organizations choose a product owner based on domain knowledge. The person that is most knowledgeable about the parts of the organisation that will be affected by the new system gets the job. In practice this often leads to a manager at the appropriate level becoming product owner. Unfortunately, a strategy that intuitively feels correct can sometimes be completely wrong. A contributing factor to the problems many organizations have in finding a suitable product owner is a lack of experience in the workings of agile development. After many years as a product owner for hire I have concluded that the following criteria are essential in order to master the product owner role (in addition to having domain knowledge):

• Time
  The role is time consuming. It is difficult to combine with other responsibilities in other than the smallest of projects.
• Understanding of agile development
  A skilled nurse knows which tool the surgeon will require next. A product owner must anticipate what the developers will be asking for and prepare answers in time. The product owner role in an agile project requires a very different approach compared with being the customer’s voice in a traditional project.
• Analytic skills
  The product owner must be able to quickly understand how processes work and how they can be improved. It is of little use to be an expert in a process if you don’t have the ability to analyze why it is designed the way it is and how it ought to be changed.
• Technical understanding
  In order to be able to balance the developers the product owner must understand the consequences of strategic technical decisions. It is furthermore an illusion to believe that requirements can be defined independently of technology. A third point is that the product owner is responsible for the development of a release strategy. This requires a combination of technical and business understanding.
• Project management experience
  Making sure that the product that is developed is actually put into use requires project management skills.
• Unbiased
  A product owner must be able to take a lot of heat and argue for sometimes painful choices without being perceived as biased.

If these are the factors that decide whether a product owner will be able to do a good job the question of where to find one needs to be rethought. It is easier to understand why so many organizations have such difficulties finding a candidate that is up to the job. Fortunately, there are alternatives to just taking the best available candidate and hoping that things will work out:

• Mentor
  A skilled mentor can aid an inexperienced product owner in the journey to becoming proficient in all aspects of the role.
• Assistant
  More hands on than a mentor, an assistant works together with the product owner in planning work and executing tasks. The assistant should also act as a mentor so that the product owner becomes more self-sufficient over time.
• Hired product owner
  Many people balk at this alternative. They feel that if they hire the product owner they will lose control of the project. This is based on the common misunderstanding that the product owner owns the product alone. A good product owner does not take decisions in isolation. A good product owner works with the organization to figure out the best course of action. The trick is to be one step ahead of the developers. One thing to note (if you are contracting out the project) is that you should never hire a product owner from the same company that supplies the rest of the team. A vital part of the role is in balancing the power that the team has in the day to day workings of the project. This will not work as well if the product owner comes from the same external company as the team.

The product owner role is critical to the success of a project. Too many projects suffer from having the wrong person on the job.

Addenum: This post has some interesting thoughts on the product owner role.

Instead of inspiring me, Kai´s posts annoyed me. At first, it was the superficial things that stood out. The self-heroics: “We were 20 years ahead of our time (Tom even more)”. The misrepresentation of agile: “It is all centered around what suits the developer.”. Having met both Tom and Kai on a number of occasions I was well aware of their tendency to sound like preachers speaking to the unenlightened masses. These superficial things irritate but they are not really interesting to debate. Some people get on your nerves and that is your problem as much as it is theirs.

But then I started to see something a lot more interesting. The core message of these posts is that you must focus on stakeholder value. This is very true and something I have focused on for the past 20 years (maybe I was 20 years ahead of my time as well…). It is also true that a lot of projects loose sight of what the primary goals of the project are. What is not true is the claim that stakeholder value is the only thing that is important. The core thing that the Gilbs fail to appreciate is that agile methods balance two things. One is the maximizing of value creation. The other thing is the maximizing of the chances of actually delivering something. These two goals are sometimes in conflict! Projects that dogmatically focus on stakeholder value are working on the right things but still risk failing completely. The simple reason agile focuses on “working software” is that this is one of the primary ways of insuring that the system being worked on will actually work.

The pragmatic approach to agile development emphasizes stakeholder value but does this in a context of many other forces. Developers that are learning new tools, languages and paradigms. A constantly changing environment for both the business and the developers. Partial and inconsistent knowledge. And much much more…

The chaos surrounding the adoption of object-orientation gradually subsided. Objects became boring. That said, there are still far too many developers that write hopelessly poor code. In the coming decade, agile will probable fare the same fate but that will depend on us not loosing focus on what is important.

Further reading: William Caputo has also written about the Gilb posts.

Agile and lean thinking in software are entering the early majority phase of adoption. This is as always a confusing period since the beginners greatly outnumber the more experienced practitioners. Misunderstandings are rife and a lot of people are struggling to transition to a new way of thinking about software development. It is against this background that I get very frustrated when someone tries to sell the core ideas in agile and lean thinking as being “intuitive”. Selling agile and lean as intuitive is wrong and dangerous on two levels. I’ll explore each of these levels separately.

Intuition is influenced by experience

The first and reasonably obvious (some would say intuitive) reason that one should not sell agile and lean as intuitive is that you will simply confuse your audience. The first time most people come in contact with lean and agile ideas their intuition will tell them that these ideas must be wrong. Agile and lean are counterintuitive for a beginner. Just think about how you reacted the first time someone explained the concept of lean production (“you mean I should not maximize use of the most expensive resource by having buffers?”), refactoring code (“if it works, don´t fix it”) or agile scope/cost/time management (“if you do it that way you will lose control”). Most peoples minds reel when they start grappling with these ideas. By telling them that all this is intuitive you are just insulting them and making them feel stupid. Much better then to acknowledge that these ideas are counterintuitive for the beginner.

The intrinsically counterintuitive

Now for the really interesting part. Many people will argue that the counterintuitive becomes intuitive once you are an experienced practitioner. You just have to unlearn a lot of old stuff first. This argument is dangerous precisely because it is almost correct! The crucial question is whether all ideas become intuitive once you internalize them. I firmly believe that there are ideas that are intrinsically counterintuitive. Two obvious examples of counterintuitive things are relativity theory and quantum mechanics. Some of the predictions that are derived from these theories are just too strange to be understood at an intuitive level.

The idea that our brain is a “tabula rasa” has been proven false by many cognitive studies. Our brains are hardwired in some respects and our perception of reality is anything but completely objective. Accepting that our brains are inherently fallible is for me one of the core breakthroughs in the modern understanding of humans. All good systems have to be built around the notion of humans being error prone. On a personal level this means that we should always we wary of our reasoning. Even if you are an expert in agile development you should never forget that your mind sometimes will fool you into “believing” in the wrong thing. Things that are intuitively “right” will sometimes prove to be wrong. I think that part of the beauty of being an intelligent being is the ability not to follow your intuition.

Just as I was finishing of this piece I stumbled across new research that in some ways takes my argument even further:

Clever fools: Why a high IQ doesn’t mean you’re smart

Here is a great talk by John Seddon about systems thinking. A lot of counterintuitive insights:

Cultural change is free

/Niklas