What are metrics in software development?

Metrics are tools we use to measure our progress toward a goal. In software development, some teams attempt to use metrics to show that they are getting better at software development over time. Other teams use metrics simply to measure output each cycle, with the goal of averaging the outputs over time in order to predict the delivery time of features in future cycles. In the latter case, the metrics are not used to improve processes, just to predict processes.

Good metrics are often difficult to achieve, hence the trepidation with which many teams put them to use. Unfortunately, metrics are often used to reward and punish employees and teams and often in cases where the teams and individuals have little control over the metrics. When metrics are used to reward and punish, you can be almost certain that individuals will attempt to game the metrics to their advantage. For example, if developers are rewarded based on the number of bugs they fix over a time period, they will compete against each other to grab the easiest bugs that time the least time to fix. They will also overestimate or underestimate the cost and value of fixing some bugs to gain advantages. Teams have even been known to regularly introduce small bugs in the code base such as misspelled words in order to have easy bugs to fix the next cycle.

I propose the use of the SMART acronym for metrics, very similar to the SMART acronym for goals, though with slight variations in the meaning of some of the attributes.

SMART Metrics

1. Specific - A metric should be easy to understand. It should be clear what the metric metric represents.
2. Measurable - A metric should be measurable. The more objectively measured the better. A good metric needs to be consistent and comparable to previous cycles in a reliable way to make the metric comparison between cycles relevant. But everyone should understand that this is the most problematic attribute of metrics. Often the things we care most about, such as “How much value did we deliver this iteration?” are nearly impossible to measure reliably and consistently.
3. Actionable - You should be able to take action based on the metric. You should be able to use the metric to make changes with high confidence the changes will lead to better metric values in subsequent cycles. In finance, these are often called “leading indicators” because they have predictive value telling us where things are going. That is in contrast to “lagging indicators” in finance, that simply confirms that an event is happening, but doesn’t help us change the outcome. In metrics, lagging indicators are often also referred to as vanity metrics.
4. Relevant - A metric should be relevant to our improvement goals. “Lines of code” written or “number of code commits” rarely relate to software development speed or quality. Just as often fewer lines of code and fewer commits relates to quicker delivery and higher quality. We must avoid the trap of measuring something just because it is easy to measure. We need to pick measurements that are relevant to our goals.
5. Timely - We need to receive the metric values early enough that we can take action based on the metric if needed.

Are there different categories of metrics?

There are many categories of metrics, and also several ways to attempt to use each metric.

• Application metrics - these are usually built into the software and measure performance, usage of features, and so on.
• Team metrics - these metrics often attempt to measure the output of a team in order to compare them to other teams, or to see if the team is improving over time.
• Individual metrics - these metrics are the most dangerous and have been used less in recent decades, in team software development, as the drawbacks and negative morale impacts have been discovered. These attempt to gather metrics about one team member in order to compare them with other team members.
• Project metrics - these metrics measure how well we met the goals set out for us for a specific project, often related to being on time and on budget.
• Process metrics - DORA metrics are a subset of process metrics that attempt to determine of our processes are improving over time toward our goals.
• Customer satisfaction metrics - a lot of software developers simply want to make their customers and end users happy. This is difficult to measure objectively and subjective surveys are often used. With this category is an attempt to determine if the team is delivery value to the customers and end users at a good pace.

Metrics gathered each cycle may be used to compare individuals to other individuals, processes to other processes, teams to other teams, and solutions to other solutions.
Metrics gathered each cycle may be used to show improvement over time from one cycle to the next in each category.

Why use metrics in software development?

As mentioned above, metrics are primarily used to measure progress, or to collect data to be used to estimate future values. Hopefully, the metrics will confirm our changes are having the desired effect, or will allow us to plan/forecast more accurately.

Why not use metrics in software development?

Good, useful metrics are really difficult to discover and implement, and when metrics are not handled correctly they can have make things worse instead of better.

How does your context affect the success you will have with metrics?

*1. How does the Organizational Structure in your context impact the probability that metrics will add value to your process?

Some organizations believe all aspects of the company will perform better if metrics are in place. And some organizations used the metrics to evaluate the performance of individuals and teams. Whether or not that practice is good for your team is outside the scope of this article, but it is important to realize that the use of metrics will be forced upon some teams by the organization.
The structure of the organization along with the chosen metrics represents another challenge affecting the choice and value of metrics. Some metrics will be more valuable than others depending on whether team members spend all of their time working on a single solution or allocate time across several projects and solutions. Long term team metrics are harder to acquire when teams and projects have short durations. It is more challenging to aggregate and compare metrics in matrix organizations where team members have multiple bosses competing to emphasize what the team member should spend time on.

*2. How does the Development Team Culture in your context impact the probability that metrics will add value to your process?

The accuracy and value of your metrics will be greatly impacted by your team culture and the metrics chosen. When metrics are used to compare team members they create competition and decrease team cohesion. This can be good for the outstanding performers on the team, but bad for the software. When team members are allowed to participate in the development and selection of the measurement metrics, there is a better chance they may be useful. As with many practices, team morale and culture greatly impact the success of using metrics, and addressing cultural needs is probably more valuable when the culture is poor than implementing other new practices.

3. How does the Hiring Process in your context impact the probability that metrics will add value to your process?

Your organization’s hiring process probably has little impact on your team’s ability to gain value from metrics or the lack thereof.

*4. How does the Team Composition in your context impact the probability that metrics will add value to your process?

If your team members are allowed to create their metrics, then experienced developers can often contribute a lot of value to the creation of the metrics. The experience of delivering software many times, especially through your processes, helps developers see what metrics may be useful or could be harmful.

5. How does the Training Provided on Code in your context impact the probability that metrics will add value to your process?

The training provided on code probably has little impact on the value metrics may bring to your team.

6. How does the Training Provided on ‘Our Ways of Doing Things’ in your context impact the probability that metrics will add value to your process?

The training your team provides on how things are done likely has little impact on the value of metrics provides to your team.

7. How does the Environment in your context impact the probability that metrics will add value to your process?

The environment probably may have a little impact on the value metrics can add to your process, if the metrics are used to compare individual performance. In such a case, individuals with good hardware and environmental setup and few distractions may outperform their peers partially due to that environment.

*8. How does the Project Selection in your context impact the probability that metrics will add value to your process?

The way in which projects are selected may have a big impact on the value of your team’s metrics. As in every case, it depends greatly on the metrics you use and how they are implemented. If your team uses DORA metrics, then projects to improve DevOps are likely to show a lot of gain and value, but projects that merely add new features customers want, may show no improvement in the DORA metrics, which can tempt teams to pick a DevOps project over a “customer value” project because the team wants to show management improvement based upon the things they are measured upon. But this is not a criticism of DORA metrics, but rather a criticism of the application of metrics evaluation by the management team, which is a risk faced by every metric chosen.
No matter what metrics your team uses, they will find that the measurement of those metrics is simpler and more accurate for some projects than for others. This can create challenges for evaluating metric scores over time, as well as in determining which teams get which projects. It all depends on the metrics chosen and how management uses the metrics in ways that may impact team members behaviors.

*9. How does the Solution Purpose in your context impact the probability that metrics will add value to your process?

The purpose of your solution may have significant impact on the value of your metrics. This is most likely when your team works on different projects and solutions with different purposes. The metrics applied when a team or person creates a solution just to experiment with a new technology, or develop a prototype, may need to be different than the metrics you want to use our your regular line of business software solutions. The choice of metrics are also likely to be different for a public facing website, application, or phone app than for an internally used application developed for just a few people. Amazon.com probably should have different metrics than the developers of the TurboTax desktop application, and the Candy Crush phone app will also have different metrics. Metrics that work well for the Facebook website may be irrelevant for a company producing software deployed on premise by city governments to track city code violations. DORA metrics are very helpful for some solutions, but irrelevant for others.

*10. How do the Application Architecture Priorities in your context impact the probability that metrics will add value to your process?

Your architectural priorities should have a significant impact on metrics. Some metrics measure your processes of developing software, others may measure individual or team output. The metrics you choose for your software, and whether or not your software is progressing toward your goals, will depend upon those goals. If delivering quickly is your top priority, DORA metrics may be right for your team. If security or performance are most important, you may find other metrics to evaluate those goals. If your goal is to provide value with features to end users, you may be challenged to find good metrics to reflect that. The Net Promoter Score (NPS) may be valuable for some teams, but primarily by comparison to the NPS of their competitors, and even then it may be more valuable to public consumers of the software rather than the team building the software.

11. How do the Security Demands in your context impact the probability that metrics will add value to your process?

The security demands of your application may no impact on the value metrics may add to your process.

12. How do the Requirements, Features, and Priorities in your context impact the probability that metrics will add value to your process?

The way your team gathers requirements and prioritizes tasks probably has little impact on the value metrics bring to your process.

13. How do the Regulations in your context impact the probability that metrics will add value to your process?

The regulations your organization must comply with probably have no impact on the value metrics may bring to your process, though some regulations may be based on metrics that require tracking for compliance.

14. How does Who Decides What To Do in your context impact the probability that metrics will add value to your process?

Who decides what to do next on your teams probably has little impact on the value metrics may bring to your process, unless metrics measure individual outputs and those individuals have little input into the tasks they performe. In that context, metrics can be very damaging to morale.

*15. How does Project Management in your context impact the probability that metrics will add value to your process?

The way you manage projects probably has some impact on the value metrics may bring to your process. Primarily the impact relates to how well and accurately you can gather and track those metrics, depending upon the metrics tracked. However, just the gathering of metrics can negatively impact teams that desire to try changes to their process and project management practices but are prevented from making those changes because the change would disrupt the aggregation of metrics across teams. Some examples of this include:

• An organization prevents a team from switching from a two week iteration to a one week iteration because all the other teams use two-week iterations and this would disrupt comparison or aggregate metrics.
• An organization prevents one team from switching from Scrum to Kanban because they don’t know how to gather similar metrics from a Kanban team and wouldn’t know how to measure the team.

16. How do the Quality Processes in your context impact the probability that metrics will add value to your process?

The processes you have in place for improving quality have no impact on the value metrics may bring to your process.

*17. How does the Architecture Forecasting in your context impact the probability that metrics will add value to your process?

The plans you have for evolving the architecture of your application may have significant impact on the value metrics may bring to your process. Metrics can drive a change in architecture. For example, a team that wants to deploy faster may decide a different architecture is needed. To some degree, this is not a decision made as the result of metric collection though.
In other contexts, some architectures better facilitate the collection of application metrics than others. If you have decided your future solution needs to run faster, be deployed more quickly, be more secure, whether or not that decision is based on metrics, it can affect your future architecture.

18. How do your DevOps in your context impact the probability that metrics will add value to your process?

This depends greatly on the metrics your team chooses to implement. DORA metrics place a great emphasis on devops. If the delivery of your software from test environments to production is largely outside of the control of your team, then delivery frequency metrics may be useless. Measurements of team performance and customer satisfaction are probably little affected by your devops processes.

*19. How does the Architecture in your context impact the probability that metrics will add value to your process?

The architecture of your application may have some impact on the value metrics may bring to your process, particularly when you desire to gather application metrics. The architecture matters less or not at all for customer satisfaction metrics. Architecture an matter if you are attempting to compare the performance of two teams developing software with different architectures.

20. How does the Code in your context impact the probability that metrics will add value to your process?

The specific languages and tools used to write code probably have little impact on the value metrics may bring to your process, unless you want to add metrics gathering features into the code itself.

*21. How does the Code Quality in your context impact the probability that metrics will add value to your process?

The quality of your existing code probably has little impact on the value metrics may bring to your process, unless you want to add metrics gathering features into the code itself.

22. How does the Application Quality in your context impact the probability that metrics will add value to your process?

The current quality of your application probably has little impact on the value metrics may bring to your process.

23. How does the Deployment Environment in your context impact the probability that metrics will add value to your process?

The environment into which your software is deployed may have quite a bit of impact on the value metrics may bring to your process. As with every answer it depends on the metrics you desire to collect. It is much easier to collect usage and interaction metrics from customers on a public web site than it is to track similar metrics from desktop apps that your customers deploy internally.

24. How does the Product Training in your context impact the probability that metrics will add value to your process?

How your team trains people using your solution probably has no impact on the value metrics may bring to your process, but training and support both can implement customer satisfaction, which is often a desired outcome of any metrics program.

25. How does the Support in your context impact the probability that metrics will add value to your process?

How your team supports people using your solution probably has little impact on on the value estimates may bring to your process, although it may be difficult to gather some metrics without changes to some support processes. However, with the implementation of metrics, especially related to customer satisfaction and the delivery of value, your support processes could be impacted by things revealed by the metrics. Such things are generally the goal of good metrics, to drive valuable improvements.

What does ChatGPT think?

(May 2023)Hey ChatGPT: User
What are the benefits of using metrics to measure improvement in software development processes, and what are the dangers of using metrics to measure improvement in software development processes?

Benefits of Using Metrics in Software Development Processes:

1. Objective Evaluation: Metrics provide an objective way to measure and evaluate the progress and effectiveness of software development processes. They offer tangible data and insights that can help identify areas of improvement and track performance over time.
2. Performance Monitoring: Metrics allow teams to monitor various aspects of software development, such as code quality, productivity, efficiency, and defect rates. This helps in identifying bottlenecks, addressing issues promptly, and ensuring that the development process is on track.
3. Data-Driven Decision Making: Metrics provide data-backed evidence to support decision making. They help stakeholders make informed choices about resource allocation, process improvements, and prioritization based on quantitative analysis rather than subjective opinions.
4. Identifying Best Practices: By analyzing metrics, teams can identify successful patterns and best practices within their development processes. This enables them to replicate successful strategies, optimize workflows, and improve overall efficiency.
5. Continuous Improvement: Metrics serve as a feedback mechanism for continuous improvement. By regularly monitoring and analyzing metrics, teams can identify areas for enhancement, set realistic goals, and track progress towards those goals over time.

Dangers of Using Metrics in Software Development Processes:

1. Overemphasis on Quantity over Quality: Relying solely on metrics can lead to a focus on meeting numerical targets, potentially sacrificing the quality of the software. It’s important to strike a balance between metrics and qualitative assessment to ensure that the overall product is of high quality.
2. Misinterpretation and Misuse: Metrics can be complex and subject to misinterpretation. If stakeholders misinterpret or misuse metrics, it can lead to misguided decisions or actions that may not align with the actual improvement needs of the development process.
3. Gaming the System: When metrics become the sole focus, there is a risk of individuals or teams manipulating the data to achieve desired outcomes, rather than genuinely improving the development process. This can result in distorted metrics and undermine the effectiveness of using them for improvement.
4. Unintended Consequences: Overreliance on specific metrics may lead to unintended consequences, such as teams optimizing for those metrics while neglecting other important aspects. For example, optimizing for lines of code written per day may discourage code refactoring or hinder creativity and innovation.
5. Lack of Context: Metrics provide quantitative data but often lack the contextual understanding of the complex software development process. They may not capture qualitative factors, such as user satisfaction, collaboration, or creativity, which are crucial for overall success.
6. Discouragement and Demoralization: Setting unrealistic metrics or using them to compare individuals or teams can create a culture of excessive pressure, demoralization, and burnout. It’s important to use metrics as tools for improvement rather than as a means of judgment or punishment.

To leverage the benefits of using metrics while mitigating the dangers, it’s essential to carefully select relevant metrics, ensure their accurate interpretation, and combine them with qualitative assessments and feedback from stakeholders. Regular review and adaptation of metrics