Tag Archives: Software Process

Software Engineering Essentials for the Working Software Developer

The previous article in the series mentioned topics that are rarely if ever discussed in a Computer Science curriculum. The topics in this article are also often skipped. Unfortunately, the knowledge of these software engineering topics is quite important for a working software developer who has software to build and deadlines to meet.

This the fourth article in the series on Starting in a Software Development Career.

Structure: Components, Modules, and Subsystems

The previous part of this series, on Computer Science Essentials, started out with the importance of assigning responsibilities to parts of the software system. But what are those parts? Here is a quick rundown of some terminology to think and reason about the structure of a software system.

This terminology was born out of the need to enable a team to build a reasonable size system, between 100,000 to 500,000 lines of code. Anything beyond that size should be considered “a system of systems.”

In most programming languages the simplest structural software element is a statement. However, program language statements don’t stand alone, they are grouped into a function. Functions are grouped into a class, struct, or namespace depending on the language that you are using. Classes are grouped into a component; components form a module; modules form a subsystem, and—at the highest level—subsystems form a system.

The structural elements above are only a recommendation, a rule of thumb. Once adopted, they imply a certain size and level of abstraction for the team that is using them. You may have to customize this for your situation. Start by establishing a hierarchy of parts and target sizes for each structural element. Here is a recommendation on sizes for you to start with:

  • Function: 10 lines of code (LOC); range: 2 to 20 LOC
  • Class, Struct, or Namespace: 200 LOC; range: 100 LOC to 1 KLOC
  • Component: 2,000 LOC (2 KLOC); range 1 to 4 KLOC
  • Module: 20 KLOC; range 10 to 40 KLOC
  • Subsystem: 50 KLOC; range 20 to 100 KLOC
  • System: 250 KLOC; range 100 to 500 KLOC

The objective of establishing the nomenclature for the levels of abstraction and sizes is to speed up the conversation about the system. This provides the vocabulary for a team to talk about the system.

Why should you care about structure? What is the meaning of these structural elements? The function and the class are basic abstractions, and they extend the capabilities of the language and its fundamental libraries. The class is an abstract data type that extends the language. Components, modules, and subsystems group related functionality in the problem domain. These elements enable the team to quickly talk about these without getting into the details of their implementation as they are composing parts and building new functionality.

Revision Control

When building software some of the paths that you and your team go down on will be in error. You will have to backtrack and restart work from a known good state. This is where a revision control system comes to your rescue. In a nutshell: the revision control system allows you—the individual developer—to carry out experiments, recover from mistakes, and—for your team—to share the work with your collaborators: allows you to co-create.

In order to benefit from revision control, you have to make your thinking explicit, and use the revision control system to record your decisions. For example: you just received a defect report that states that the shipping charges are not computed correctly. Next you create a branch, figure out what the problem is, implement the fix, verify that it all works, then merge your branch into the main code line. If your fix fails, then you can restart your work from the point where you started from.

Similarly, create a branch for any new work you are doing on the system. Working in the branch creates a moment of calm and stability while you implement the new feature. The time you work in the branch should be short (a few hours to a couple days at most). Any longer than that and the moment becomes a frozen epoch, and you face a mountain of changes as you try to bring your new feature into a system that has evolved while you were not looking.

Issue Management

Issue management is one of the dark knights of software development. It is powerful, but often avoided by developers and their teams. Sometimes, it can feel like a lot of work. Issue management describes a group of related activities.

Define the work. Issue management is an expression that refers to the management of defects, tasks, features, or any other items of work that are collectively known as issues or work items. Think of an issue as a unit of work. Start by writing down what the work is; this becomes an “issue.” The issue needs to have at least two parts: a name, and a description. Find a good name that helps you quickly recall what this work is when you see the issue name in the list. Once you named the issue, next describe it in sufficient detail that you can recall what that work is even a week or two later. Now that you defined the issue, you can start managing it.

Order and schedule the work. You must decide in what order and when to do the work. The simplest method is sequencing issues one after another in the order in which they can be completed based on the dependencies between them. You could also schedule the issues on the calendar. Or you can use a more complicated scheduling algorithm that distributes the work across many people and based on constraints and rules. When you start out and your team is relatively small, just stick to the simple stuff.

Perform the work. Do them in the order that you arranged them and how you scheduled them. This sounds simple. What complicates it is that you need to be mindful that if you see that the defined work no longer makes sense, then you need to change it, and quite possibly you need to define new work. You may need to change the ordering and scheduling, too.

Capture the progress and the completion of the work. While you work, remember to record your progress. If something takes longer than an hour, set the status of the item to “In Progress,” especially if you are working on a team. After all, you don’t want two people to work on the same stuff. When you are done, mark it complete. It feels good to finish stuff!

Revision control and issue management should work together. Revision control systems provide a capability to associate the source code you write with an issue from the issue management system. This way you can tie the definition of the work to the work you did as a result of the issue you defined. This capability comes in most useful when you are trying to figure out how something went wrong.

Design

The hidden secret of design is that everybody says that they design, but you find almost no artifacts of the design activity, excepting a few confusing whiteboard sketches. Without artifacts, there are only thoughts about design.

Why no artifacts? One reason might be that is easier to talk about it, than to write it down. Design is a high-intensity mental activity. You have to envision how the system will look or work, before actually building it. You have to draw pictures and write some prose that talks about both the system’s structure and its behavior.

Here are the essentials: you needed to draw “boxes,” “lines,” and write some “text.” The boxes are software parts. The lines are dependencies between those parts, and the text explains their responsibilities.

The beauty and freedom of design is that starting from the top down you can invent the boxes that you need to implement your system. At the top level you divide up all the responsibilities of the system across the parts (the boxes) that you have identified. Draw the lines between them: this tells the story of how those boxes communicate between them.

Next, focus on one of the new “boxes” that you envisioned and the responsibilities that you assigned to it. How will it perform those? Design what’s inside that box. The method is the same: identify the boxes that are inside it, draw the dependency lines, and assign the responsibilities. No magic. Some part of the system has to do the work, that part must be identified, and the responsibilities assigned.

Whenever you can use “boxes” that already exist, so that you don’t have to write them. Then your job is to fill in the gaps, write those software parts that you don’t yet have.

Check your design by “role-playing” or walking through step-by-step the use case that the system has to support. Do you have a software part identified to perform every bit of the use case? Does the part have sufficient responsibilities to do the work? If yes, move on to implementation.

Architecture

Each of the topics discussed so far has several books written about it—this is true for architecture as well.

“The software architecture of a system is the set of structures needed to reason about the system, which comprise software elements, relations among them, and properties of both.” (#1) The reason you should care about architecture goes back to assigning responsibilities and making fundamental decisions about the system’s structure and desired behavior. These decisions will ultimately place constraints on the system’s implementation.

As you are starting out in your career in software development you’ll encounter architecture mostly in negative statements, like: “Someone needs to make architecture decisions around here.” and similar. Developers usually decry the lack of architecture, but rarely step up to the plate to actually do something about it. Architecture decisions must balance both technology and business strategy constraints. And this seems to be the stumbling block for most developers. A software system’s architecture must support the business strategy the organization decided to follow. However, most developers don’t want to get into business strategy, only technical strategy, therefore not that many people work on real architecture.

The need to focus on and the understanding of business strategy is also what distinguishes architecture from design. A system’s architecture in some sense also represents the totality of technical decisions based on the business strategy the organization adopted. Once the architecture is in place, then the design decisions that follow implement those decisions.

Just like with design, there is always an architecture for any system, even if no deliberate action was taken to devise one. This is fine sometimes, at least until the business makes some strategic changes. Then everybody wonders why things are the way they are with the system. As with some other activities, the value of architecture often becomes clear only when change happens.

Summing up: when you write software through a series of deliberate actions, you may still not get the software that your users want, but at least you will know how you got there. That gives you a chance to make the necessary corrections to get to what your users want.

Enjoy the journey!

  1. Bass, Len, Paul Clements, and Rick Kazman. Software Architecture in Practice. 3rd Ed. Boston: Addison-Wesley, 2013.

Software Development Strategy

strat-e-gy: a plan of action or policy designed to achieve a major or overall aim.

The New Oxford American Dictionary

The dictionary definition points us in the right direction. The software development strategy is your highest level plan for achieving your software project’s objectives. According to Watts Humphrey, the strategy is “the order in which product functions features or requirements are defined, designed, implemented, and tested.”

What does this mean? This means that the plan of action part of your strategy lays out what you build first, what you build next, and so on. It also lays out how are you going to validate the strategy. The strategy also lays out the policies for risk mitigation and decision making. Sections of the strategy may exists both along the developer – customer continuum and also along the system abstraction levels continuum. For details about these continuums see: Multilevel Planning and Component-Based Development.

The System Architect is responsible for the overall strategy. The strategy has to be sound enough that designers and implementors working on parts of the system receive from it the necessary guidance to aid their technical decisions. A well conceived strategy pushes the decision making down to the lowest possible level, closest to where the problems are encountered, while at the same time provides for the oversight necessary to build a coherent system.

Accompanying the development strategy is the testing strategy. The testing strategy focuses on the various testing activities, including the system validation and verification activities. It is constructed similarly and serves the same purpose as the development strategy, only for testing. The owner of the testing strategy may be the System Test Architect, if there is one, otherwise this is also owned by the System Architect.

Why do you need a strategy? Without a strategy it is difficult, if not impossible, to execute an iterative, incremental development process. The strategy ties the iterations into a coherent whole, and allows the team to move confidently toward the goal. Your first attempt at creating a strategy may feel awkward, but with practice you’ll get better at it. Why don’t you get started today?

Effective On-Task Time

Software development is knowledge work. We use knowledge, take input as knowledge, and the output we produce is knowledge. Our output is executable knowledge. According to Peter Drucker, the 21st century’s defining characteristic is knowledge work. We, software engineers, are right in the middle of it. Drucker writes that manual worker productivity during the 20th century increased 50 times. He admonishes us to improve knowledge worker productivity during this century as much as we improved manual worker productivity during the last century. He postulates that the developed nations of the future will be those that will make their knowledge workers productive.

Unfortunately, we seem to have a hard time measuring productivity. Economists have a straightforward measure of productivity: the value of the output divided by the time required to create it, expressed in dollars per hour. We need to figure out how to make this definition operational for software developers. A component of this measure is time.

Let’s start with Effective On-Task Time or EOT. This is the time that you spend creating the deliverable that is the target of your task. You keep this number pure, devoid of anything that didn’t directly contribute to the output. Your objective is to measure the time that the task took. You can find references to the concept of “Task Time” in a number of places. Watts Humphrey writes that “task time is what engineers spend working on scheduled tasks.”

And herein lies the heart of the matter: EOT is about working on something that you planned to work on. It is about your ability to meet a commitment. The plan is your commitment. The time that it took to complete the task is reflective of your ability to understand the work required to meet that commitment.

EOT doesn’t include time for any activities that don’t matter to the work of creating that deliverable. The interruptions from your co-worker, phone calls, and breaks don’t count. Some of the activities may tangentially work toward it, but your objective is not to account for every minute, but rather to count those distinct minutes when you know that you are moving the needle forward.

At the end of the day, EOT matters first and foremost for you. It helps you understand how good you are, how well you understand your work. It builds your confidence. When you repeatedly make aggressive plans and meet them, you know that you are becoming the engineer that you always wanted to be: outstanding.

References:

Build a Rough Plan for Your Work

Planning is a topic that I find myself coming back to time and time again. I’m asked enough questions about it so I decided to write down some of the answers. Folks who have not had to do any planning are stumped when asked to make a plan. I found that planning can be rather intimidating until you start with writing down what are the deliverables and what is the time frame. This will get your thinking started in the right direction. This and upcoming notes are here to help you get started on planning your own work.

Consider this scenario: you get assigned to a project, but instead of being the one you dreaded, this is one that you are excited about! The technical lead held an effective iteration kick-off meeting. You picked up work on a couple of features that you thought were really cool. You are expected to make a commitment to deliver the features at the end of the two-week iteration. Can you do it? How sure are you? How do you begin to build a plan for your work?

The first step is to decide how much time you can afford to spend on each feature.

  1. Kickoff (1 day)
  2. First feature (4 days)
  3. Second feature (4 days)
  4. Wrap up and demo (1 day)

Next, estimate how you intend to spend the 4 days you budgeted for each feature. You must think of your time as money. Spend it well. Let’s suppose that your technical lead had done a good job coming up with granular features, and you can indeed implement each feature in 2-4 days. Here is how you can “spend” your time during the iteration:

  1. Iteration kick-off meeting (1/2 day)
  2. Plan the iteration (1/2 day)
  3. Design the first feature (1 day)
  4. Implement the first feature (1 1/2 days)
  5. Test the first feature (1/2 days)
  6. Prototype the second feature (1 day)
  7. Design the second feature (1/2 day)
  8. Implement the second feature (1 1/2 days)}
  9. Test the second feature (1/2 day)
  10. Fix the defects found in integration & system test (1 day)
  11. Demo the iteration results to the team. (1/2 day)
  12. Buffer (1 day)

That’s it. This is your rough plan for the iteration. If you wish, you can call this a time budget and not a plan. If you are to deliver the features at the end of the two weeks, this is all the time you have. The key point is that this plan, even with its limited details, shows you what time can you afford to spend on each feature. The presence of the buffer in your plan is the clearest acknowledgment that you don’t yet know enough. When new things come about, you won’t have to replan your entire plan. You can just eat into your buffer. However, once your buffer gets exhausted, then you need to rework your plan.

Looking at this plan you might think that you won’t be looking at times at both features. That should not be the case, especially if they are related. It just means that now you have a way to get the work started. As you get into the details of the work, you will refine this plan. For ideas on how to move forward take a look at What do you know? and Guidelines for Detailed Planning. There are many other aspects of planning, e.g. getting some data on how you are doing, and how to improve your planning skills. For now though, just get started. As the old Chinese proverb puts it:

The one thousand mile journey starts with the first step.”

Ready! Set! Plan!

Agile & CMMI & me

Back in the second half of the eighties I have been using some agile practices, but we didn’t call it agile back then. We regularly pair programmed because my friend and I shared one computer; neither of us wanted to just sit and do nothing. Our workflow also included writing test programs (much like unit tests today).

In the early nineties I read “Managing the Software Process” by Watts Humphrey. This was my first introduction to the CMM. I found many of the concepts valuable, even though, I was working alone. The knowledge presented was applicable to my software engineering work, with the necessary adjustments for a group size of one.

Many years later, after the agile movement got traction, I was surprised that so many people had such strong feelings about the CMM given my positive experiences with it. In time I understood that there were many organizations that got their CMM implementations off track. They focused on paperwork, instead of changing the day-to-day work of practicing software engineers.

After over two decades of CMM & CMMI and over a decade of agile, the software engineering community is realizing that a sensible strategy is to integrate the best of both worlds. A recent paper published on the SEI website by Hillel Glazer, Jeff Dalton, David Anderson, Mike Konrad, Sandy Shrum urges us to do so: CMMI or Agile: Why Not Embrace Both! It seems that I am not the only one thinking so. Scott Ambler wrote a review of the paper at DDJ called: Agile CMMI: Complimentary or Oxymoronic? Read the paper, read Scott’s review, and decide for yourself.