Software Process: Here is why I got interested

You know the feeling: you are assigned to a new project and asked to give an “estimate” of the length of time the new project will take. You think about it for a moment—or for a day—and then give the date. But you have no confidence in the date, so before you give it you double it, or triple it, hoping that the new date will give you just enough time to take care of most of the unexpected things that might come up.

Both you and I know that this is not adequate, since this almost never yields the correct date. I never trusted the date since I never understood why the fudge factors were needed. But the biggest problem I had with estimation was that I didn’t know how to get better at it, since I didn’t understand how it worked.

It was in early 1992 that I first paid attention to something called the software process. I read Watts Humphrey’s Managing the Software Process. Humphrey points out some of the common process issues that every software organization has to face and how to get a handle on them. I started applying his ideas in a small group, not quite what he designed them for. Shortly after Humphrey finished writing his book he realized that the Capability Maturity Model that he described must be applicable for a person as well. He proceeded to write a book for the individual engineer: A Discipline for Software Engineering—The Personal Software Process. That book put the role of the individual engineer in a new perspective. Later, I had a chance to attend Humphrey’s seminars for engineers, managers and executives and I started teaching it, and mentoring people on its use as well.

Today I know that you get a better overall estimate if you estimate the size of the software first. Then you apply to the size estimate your productivity rate to get the project effort estimate (in hours). If you don’t know what your productivity is, then you will have to estimate it, and reestimate the project effort as soon as you have real data. If you work with short iterations then in a few weeks you should have real data, so your effort estimate will become more accurate.

Once you had figured out how many hours will the project take you can move on to figure out the date when it will be done. First, you consult your calendar and see how many hours you will have available for project work. Second, map the project hours onto the available calendar hours. This will give you a date that is pretty realistic. You might not like it, but this is a realistic date.

A major benefit to coming up with the date like this is that it gives you three levers that you can use to control the the end date:

1. You can reduce the scope of the project,
2. Be more productive (likely you can change this lever only in the long term), or
3. Work more hours (only a short term solution, because if you sprint when you are running a marathon, sooner or later you will run out of steam and you will never finish).

Since these three are independent variables, they require different actions. In the short term, you can reduce the scope of the project if you can’t add more hours either by working more or adding other people to the project. As we know from Fred Brooks, adding more people to an already late project will only make it later.

In 1999 I read The Fifth Discipline, The Art and Science of The Learning Organization by Peter Senge. This work, together with two follow-up works, The Fifth Discipline Fieldbook and The Dance of Change, pointed out to me the importance of looking at systems, not just software engineers, development teams, customers, companies, and suppliers. Senge’s work on systems seems comparable to Peter Deming’s work on quality. Senge’s work is not yet appreciated by most organizations.

To close the improvement circle I came back to the individual engineer. It is the individual engineer’s work that determines how good the product will be. The PSP (Personal Software Process) augmented with the TSP (Team Software Process)—both by Watts Humphrey—promise that by helping the engineering team get better—one engineer at a time—the end product will get better as well. Humphrey and his fellows from the SEI has thought the PSP class to thousands of engineers and gathered enough data to back up his claims. The doubters are welcome to try it for themselves.

There are many new challenges in the field of software engineering and I don’t anticipate running out of things to learn and practice. The main focus for the near future is:

• Integrating size measurement into object-oriented analysis and design;
• Studying and practicing Six Sigma and figuring out ways to apply DMAIC (Define, Measure, Analyze, Improve, and Control) in the context of the Software Life Cycle;
• Studying and practicing Systems Thinking and the other four management disciplines: Personal Mastery, Mental Models, Shared Vision, and Team Learning to create an organization that lives up to its full potential.