Chris Sterling’s Blog

Managing Software Debt
Continued Delivery of High Values as Systems Age

Many software developers have to deal with legacy code at some point during their careers. Seemingly simple changes are turned into frustrating endeavors. Code that is hard to read and unnecessarily complex. Test scripts and requirements are lacking, and at the same time are out of sync with the existing system. The build is cryptic, minimally sufficient, and difficult to successfully configure and execute. It is almost impossible to find the proper place to make a requested change without breaking unexpected portions of the application. The people who originally worked on the application are long gone.

How did the software get like this? It is almost certain the people who developed this application did not intend to create such a mess. The following article will explore the multitude of factors involved in the development of software with debt.

What Contributes to Software Debt?

Software debt accumulates when focus remains on immediate completion while neglecting changeability of the system over time. The accumulation of debt does not impact software delivery immediately, and may even create a sense of increased feature delivery. Business’ responds well to the pace of delivered functionality and the illusion of earlier returns on investment. Team members may complain about the quality of delivered functionality while debt is accumulating, but do not force the issue due to enthusiastic acceptance and false expectations they have set with the business. Debt usually shows itself when the team works on stabilizing the software functionality later in the release cycle. Integration, testing, and bug fixing is unpredictable and does not get resolved adequately before the release.

The following sources constitute what I call software debt:

• Technical Debt[1]: those things that you choose not to do now and will impede future development if left undone
• Quality Debt: diminishing ability to verify functional and technical quality of entire system
• Configuration Management Debt: integration and release management become more risky, complex, and error-prone
• Design Debt: cost of adding average sized features is increasing to more than writing from scratch
• Platform Experience Debt: availability and cost of people to work on system features are becoming limited

Software Debt Creeps In

Figure 1.1: A relatively new system with little debt accrued.

Figure 1.1 displays a system that has minimal amount of software debt accrued. A few low priority defects have been logged against the system and the build process may involve some manual configuration. The debt is not enough to significantly prolong implementation of upcoming features.

Business owners expect a sustained pace of feature development and the team attempts to combine both features and bugs into daily activities, which accelerates the accrual of debt. Software debt is accruing faster than it is being removed. This may become apparent with an increase in the number of issues logged against the system.

Figure 1.2: An aging software system slowly incurs significant debt in multiple functional areas.

As a system ages, small increments of software debt are allowed to stay so the team can sustain their velocity of feature delivery. The team may be complaining about insufficient time to fix defects. Figure 1.2 shows a system that has incurred software debt across all functional areas and components.

At this point, delivery slows down noticeably. The team asks for more resources to maintain their delivery momentum, which will increase the costs of delivery without increasing the value delivered. Return on investment (ROI) is affected negatively, and management attempts to minimize this by not adding as many resources as the team asks for, if any.

Even if business owners covered the costs of extra resources, it would only reduce the rate of debt accrual and not the overall software debt in the system. Feature development by the team produced artifacts, code, and tests that complicate software debt removal. The cost of fixing the software debt increases exponentially as the system ages and the code-base grows.

Figure 1.3: The aging system has accrued significant debt in all functional areas and components.

Software debt in the system continues to accrue over time, as shown in figure 1.3. At this point, new feature implementation is affected significantly. Business owners may start to reduce feature development and put the system into “maintenance” mode. These systems usually stay in use until business users complain that the system no longer meets their needs.

Managing Software Debt

There are no magic potions for managing software debt. Software can accrue debt through unforeseen circumstances and shortsighted planning. There are some basic principles that help minimize software debt over the lifespan of the product:

• Maintain one list of work
• Emphasize quality
• Evolve tools and infrastructure continually
• Always improve system design
• Share knowledge across the organization
• And most importantly, hire the right people to work on your software!

Maintain One List of Work

One certain way to increase software debt is to have multiple lists of work. Clear direction is difficult to maintain with separate lists of defects, desired features, and technical infrastructure enhancements. Which list should a team member choose from? If the bug tracker includes high priority bugs, it seems like an obvious choice. However, influential stakeholders want new features so they can show progress to their management and customers. Also, if organizations don’t enhance their technical infrastructure, future software delivery will be affected.

Deployed software considered valuable to its users is a business asset, and modifications to a business asset should be driven from business needs. Bugs, features, and infrastructure desires for software should be prioritized together on one list. Focus on one prioritized list of work will minimize confusion on direction of product and context-switching of team members.

Emphasize Quality

An emphasis on quality is not only the prevention, detection, and fixing of defects. It also includes the ability of software to incorporate change as it ages at all levels. An example is the ability of a Web application to scale. Added traffic to the web site makes performance sluggish, and becomes a high priority feature request. Failed attempts to scale the application result in a realization that the system’s design is insufficient to meet the new request. Inability of the application to adapt to new needs may hinder future plans.

Evolve Tools and Infrastructure Continually

Ignoring the potential for incremental improvements in existing software assets leads to the assets becoming liabilities. Maintenance efforts in most organizations lack budget and necessary attention. The International Organization for Standardization (ISO) standardizes on four basic categories of software maintenance in ISO/IEC 14764[2]:

• Corrective maintenance - Reactive modification of a software product performed after delivery to correct discovered problems
• Adaptive maintenance - Modification of a software product performed after delivery to keep a software product usable in a changed or changing environment
• Perfective maintenance - Modification of a software product after delivery to improve performance or maintainability
• Preventive maintenance - Modification of a software product after delivery to detect and correct latent faults in the software product before they become effective faults

Most maintenance efforts seek to prolong the life of the system rather than increase its maintainability. Maintenance efforts tend to be reactive to end user requests while business evolves and the technology decays.

To prevent this, attention must be given to all four categories of software maintenance. Budgets for software projects frequently ignore costs for adaptive, perfective, and preventive maintenance. Understanding that corrective maintenance is only part of the full maintenance picture can help an organization manage their software assets over it’s lifespan.

Improve System Design Always

Manage visibility of system design issues with the entire team. Create a common etiquette regarding modification of system design attributes. Support the survival of good system design through supportive mentoring, proactive system evolution thinking, and listening to team member ideas. In the end, a whole team being thoughtful of system design issues throughout development will be more effective than an individual driving it top down.

Share Knowledge Across the Organization

On some software systems there is a single person in the organization who owned development for 5 years or more. Some of these developers may find opportunities to join other companies or are getting close to retirement. The amount of risk these organizations bear due to lack of sharing knowledge on these systems is substantial.

Although that situation may be an extreme case of knowledge silos, a more prevalent occurrence in IT organizations is specialization. Many specialized roles have emerged in the software industry for skills such as usability, data management, and configuration management. The people in these roles are referred to as “shared resources” because they use their specialized skills with multiple teams.

Agile software development teams inherit team members with specialized roles, which initially is a hindrance to the team’s self-organization around the work priorities. Teams who adhere to agile software development values and principles begin to share specialized knowledge across the team, which allows teams to be more flexible in developing software based on priorities set by business. Sharing knowledge also reduces the risk of critical work stoppage from unavailable team members who are temporarily on leave.

Hire the Right People!

It is important to have the team involved in the hiring process for potential team members. Teams will provide the most relevant skills they are looking for, thus, allowing them to review and edit the job description is essential. Traditional interview sessions that smother candidates with difficult questions are insufficient in determining if the candidate will be a great fit. Augmenting the interview questions with a process for working with the candidate during a 1 to 2 hour session involving multiple team members in a real-world situation adds significant value to the interview process. Before hiring a candidate, teams members should be unanimous in the decision. This will increase the rate of success for incorporation of a new team member since the team is accepting of their addition.

Another significant hiring focus for organizations and teams is placing more emphasis on soft skills than technical expertise. I am not advocating ignoring technical experience. However, it is critical in an agile software development organization or team to have people who can collaborate and communicate effectively. Soft skills are more difficult to learn than most technical skills. Look for people who have alignment with the hiring team’s values and culture.

In Summary

As systems age they can become more difficult to work with. Software assets become liabilities when software debt creeps into systems through technical debt, quality debt, configuration management debt, design debt, and platform experience debt.

Applying the six principles in this article will lead to small changes that over time will add up to significant positive change for teams and organizations. The goal of managing software debt is to optimize the value of software assets for our business and increase the satisfaction of our customers in the resulting software they use.

References

1. Ward Cunningham - “Technical Debt” - http://c2.com/cgi/wiki?TechnicalDebt
2. “ISO/IEC 14764: Software Engineering — Software Life Cycle Processes — Maintenance” - International Organization for Standardization (ISO), revised 2006

10 Comments »

In the past I have blogged on software debt in terms of:

• Going beyond code to manage software debt in “Clean Up, Clean Up, Everybody Do Your Share”
• Problem with catching up on software debt in “The IT Manager’s Dilemma with Software Debt”

Although there are reasons defined or implied in those blog entries I think it is important to discuss it more directly. The following image will be the backdrop for the rest of this blog entry.

Software that has been around for some time usually gets difficult to work with. The value of the software depreciates as more debt creeps into the software because the costs have not went down and probably went up to keep up with maintenance. So the costs are rising to maintain the system and the value of the software is depreciating. A business asset that has these kind of characteristics is in jeopardy of getting scrapped. Sometimes the only thing stopping us from scrapping the software is that we have nothing else to replace it and the business processes it currently supports are dependent on it.

Depreciation of software is inevitable but when we manage software debt diligently with a disciplined approach we can significantly prolong the depreciation from going below the minimum acceptable value for the software. In fact, if we manage software debt in the process of adding business functionality we will increase the software’s value and therefore further prolonging significant depreciation.

I will be at Agile Vancouver speaking on “Managing Software Debt” on November 6th. This presentation has been given in multiple variations over the past year and a half. Although I won’t go into detail here, the following 6 principles are what I see as essential to managing software debt effectively:

• Maintain one list of work
• Emphasize quality
• Evolve tools and infrastructure continually
• Improve system design always
• Share knowledge across the organization
• And most importantly, hire the right people to work on your software!

I will post on more details of managing software debt in the near future.

Comments Off on Why Should We Manage Software Debt?

Refactoring is an essential practice for teams developing solid software and continually evolving the design to meet new customer needs. From the home page managed by Martin Fowler, who wrote the original book on refactoring, it says:

“Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior.”

On a project that is well tended in terms of its design and structure the act of refactoring can be elegant and liberating. It will allow teams to continually inspect and adapt their understanding of the code and the customer’s need as they gather feedback on the software.

On a legacy system the act of refactoring can seem overwhelming. Although the refactoring craft revolves around the mindset that continually making small incremental improvements will lead to overall improvement in the design figuring out where to start and stop in a legacy system is unclear. This blog entry is for teams who are unclear on where to begin refactoring and how much refactoring is sufficient day to day.

Where to Start Refactoring?

On all teams that I have worked with either as a team member or coach this question starts with the answer:

When the feature you are working on exposes the need to refactor

The problem is that this does not give a full answer with context and understanding for the particular project we are discussing. I like to answer the following questions before starting to refactor:

• Does this change directly affect the feature I am working on?
• Would the change add clarity for the feature implementation?
• Will the change provide automated tests where there currently are none?
• At first glance does the refactoring look like a large endeavor involving significant portions of the system components?

For the first 3 questions if the answer is “yes” then I am leaning towards starting a refactoring of the code. The only caveat is the answer to the last question. If the answer to this question is “yes” then I will use my experience with the system help me generate a gut feeling about the size of this refactoring compared to our initial cost estimate of the feature implementation. If the size of the refactoring is significantly larger than the cost estimate given to the Product Owner / Customer then I may decide to bring the refactoring up to the team for discussion. As we teach in the Certified ScrumMaster course:

The Product Backlog estimate is a mutually agreed budget. If the team is going to exceed the budget, it needs to escalate the decision. Otherwise, the team introduces scope creep.

When to Stop Refactoring?

Using the word “stop” is misleading since it means we just decide to quit at some point in time without regard for the situation. I would not suggest this at all. Instead the following conditions are what I use to figure out when I have finished a refactoring:

• Am I refactoring code structures that are not directly affected by the current feature?
• Are there any more code structures that are directly affected by the current feature I am working on that have not been refactoring sufficiently?
• If the refactoring is blowing our mutually agreed upon budget for this feature with the Customer then I should bring this up to the team to discuss further progress
• If we decide as a team that the refactoring can be absorbed into the current iteration without affecting delivery on our original iteration commitments then continue to refactor
• If the refactoring cannot be absorbed into the current iteration without affecting delivery on our original iteration commitments then we must bring it up to the Customer to discuss how to proceed

If the situation leads us to bringing up the refactoring to our Customer then the options for are to:

• Simplify the original feature - the team could bring a suggested simplification to the customer
• Remove the feature from the current iteration commitments
• Remove one of the other features from the current iteration commitments

Wrap Up

Starting a refactoring should be identified in the course of implementing a piece of business value prioritized by the Customer. We should gauge the size of the refactoring against the cost estimate given to the Customer. Wrapping up a refactoring should not be done without regard for the quality of our implementation going forward. The result of a refactoring should be improved code structure while preserving existing functional behavior. Use the questions, conditions, and options to manage your refactorings along with delivery of business value to your Customers.

1 Comment »

From time to time I speak with a team member, ScrumMaster, functional manager, or Product Owner that asks how do we handle maintenance tasks. After we get past the obvious question “why do you have a problem with maintenance tasks?” we discuss team configurations to handle their current maintenance pains.

Traditional methods of managing maintenance tasks tell us to separate maintenance tasks from feature delivery because it always slows us down when we do both at the same time. I agree that it slows us down and I would look at this as an opportunity to start fixing the root cause of this issue. Usually this has something to do with our software development processes not providing us the ability to build integrity into our product as we go. Instead we are constantly reacting to what we might have broken hours, days, months, and years ago once it gets found. It costs much more to fix the problem when it has persisted so long in the system with new code surrounding and depending upon it now. Our first objective must be to start the long payback period on the system’s software debt that we have allowed to accrue.

The next step is to figure out how we manage our day to day activities. I would like to discuss the “Team Member in Siberia” anti-pattern before I plot out potential solutions for maintenance of existing systems.

Team Member in Siberia

Context

Teams find maintenance work diverts their focus from new product feature development. In order to combat these diversions the team, functional managers support the team members, and customers wish to seperate the maintenance work from new product feature development.

Problem

Separation of maintenance work that diverts focus of teams from new product feature development means:

• Certain team members will be singled out for maintenance
• If maintenance work is on a specific platform that an individual team member has the most knowledge about they may be singled out to always handle those work items
• The team members who are developing new features for the product may be creating some of the maintenance issues that divert focus but do not have to deal with their ramifications
• Team members who are allocated to working on maintenance work may find that meeting with other team members is a waste of time since nobody else on the team is collaborating with them to handle those work items. (a feeling of micro-management may ensue)
• Maintenance issues usually are not as fun to work on and may be seen as a demotion to some team members

There is not a one-size-fits-all approach to deal with the “Team Member in Siberia” anti-pattern immediately. As a Scrum Coach and Certified Scrum Trainer I suggest that the principal “if it’s not in the Product Backlog it doesn’t exist” is essential. Once we divert work into a separate list it is as if we have a separate Product Backlog. One team should not be pulling items from multiple Product Backlogs. I would also suggest that if you have one product then you have a single prioritized Product Backlog to represent it.

Questions to Ask

Some questions that you can ask when the teams are in the context described above and exhibiting one or more of the problems listed are:

• Are all team members working on the same product?
• Are we diverting a tenured team member onto work they do not enjoy?
• Are the work items that are diverting our focus recurring problems?
• Why do we have enough maintenance work that it causes us to think about allocating team capacity or a separate list of work to it?
• Do we have a risk of knowledge getting tied up in one person’s head and therefore we are not respecting the “bus factor”?
• Are defects generated by the new feature development team that get fixed by the maintenance team?
• Are team members not finding value in attending a Daily meeting or complaining about getting micro-managed with agile?

Potential Solutions

Although each organization has it’s own culture, sources of influence, and characteristics there are a few potential solutions I have seen to enable more alignment of the “Team Member in Siberia” with the rest of the development team.

• Share maintenance responsibilities across team - Instead of placing all responsibility for system maintenance on a single individual or small group of people make the maintenance activities part of the whole team’s shared responsibilities. In fact, force mentoring on maintenance activities across the team by always having a person less familiar with the system shepherd the task and ask for mentoring when necessary from the person with more familiarity.
• Put maintenance activities into backlog of work - Rather than keep a separate list of work from the new features from business, have business prioritize the maintenance work along with the new features. This forces maintenance to get more attention over time and streamlines maintenance with better infrastructure in order to enable new feature development.
• Track the amount of maintenance work - Teams can keep a container and small pieces of paper in their Daily Standup area so they can capture how much time was used each day on unplanned maintenance work. Although this won’t fix the problem we can become more aware of how much work is coming in unplanned. You can do a couple of things with this information. Firstly, teams add up the amount of unplanned maintenance work they have each day or iteration and track trends over time to see if they are handling it more efficiently. Second, it can be used to inform business about the amount of unplanned work impacting the team. Lastly, the team can figure out how to streamline and minimize the unplanned work over time.
• Define product backlog items from the end user’s point of view - Some backlogs contain what could be thought of as tasks for the team to take on rather than describing what the end user wants. Instead of items such as “analyze monthly revenue report” describe a higher level need of the end user such as “As Finance I want to see the monthly revenue report”. The first item is something that a business analyst on the team will work on but the second item will involve multiple team members including the business analyst, tester, and programmer. When these team members come to their Daily Standup they will have common work items and be more interested in potential impacts on each other. This will cause the Daily Standup to feel less like micro-management.

There are other potential solutions but this list is a good start for those finding their teams in the “Team Member in Siberia” anti-pattern. Please share your potential solutions, suggestions, and questions in the comments of this blog entry. And get those team members working together for the good our products.

1 Comment »