fbpx

How We Measure Software Project Quality: Scrum Metrics We Use

May 6, 2021
13 min

Anthony W. Richardson, an author of ‘How to Grow Any Startup Without a Plan or a Clue’, once said:

That which cannot be measured cannot be proven.

Anthony W. Richardson.

These words perfectly suit the concept of project evaluation as only measurement leads to efficiency and success in the long run.

Today we want to share our best practices on how to effectively measure project quality with Agile metrics to keep your software development team on track.

In this post we share:

  • Scrum metrics we use for measurement of software quality and productivity
  • Formulas on how to measure quality in project management
  • Examples of metric charts and tables
  • Review of Agile Scrum methodology

Let’s get started with our favorite Agile productivity metrics we recommend using to measure project progress and performance efficiently.

1. Sprint Velocity

Sprint velocity is one of the most wide-spread Scrum metrics which allows you to get a historic overview of how much value you have delivered at every sprint.

In terms of one sprint, it is the amount of work (story points) your developers can complete under ideal circumstances.

Why sprint velocity is a crucial metric?

With this knowledge, you can plan projects and predict how much work can be completed in the next sprint. The more accurate you are with this estimate, the more realistic your plans will be.

How to estimate sprint velocity?

Everything you need is just to sum the story points completed at all sprints and divide them by the number of sprints.

We should mention that all examples in this article will start from the 2nd sprint. Our Senior Project Manager, Tatiana Vichkanova says:

Mostly, the first sprint of every project is dedicated to setting up the internal processes, adjusting the teamwork, preparing efficient communication channels, and so on.

- Tatiana Vichkanova, Senior Delivery Manager at Ascendix

For example, let’s calculate sprint velocity based on the below data:

  • Sprint 2: 4 user stories x 12 story points = 48
  • Sprint 3: 5 user stories x 12 story points = 60
  • Sprint 4: 6 user stories x 12 story points = 72
  • Total: [Sprint 2] + [Sprint 3] + [Sprint 4] = 180

So, your average sprint velocity is 180 / 3 = 60.

Grab a Freebie: Scrum Metrics to Evaluate Software Team Performance (with Formulas)

Download a list of metrics we use to evaluate the productivity and performance of our software development teams. We explain how to calculate and analyze each metric.

Name
Email

 

This way, you can now predict the number of sprints left to get all work done. If you have 180 story points remaining to be done, you can assume that your developers will need 3 sprints more to complete the project.

However, it is only an estimate which can be reached only if all other things constant.

One of the most effective ways to track sprint velocity is to draw visuals charts.

There are many cloud-based tools that allow you to draw multiple charts and share them with a Scrum team.

Below you can see an example of a velocity chart created at Ascendix Tech:

An Example of a Sprint Velocity Chart | Ascendix Tech

An Example of a Sprint Velocity Chart

This chart allows you to analyze the amount of work done during each sprint to get an overall picture of the project activity.

2. Effort Estimation Variance

Over- and under estimations have a huge impact on a team’s performance and project activity.

If you overestimate the tasks, it may lead to the appearance of new risks that can ruin your project plans.

It worth mentioning that only accepted user stories should be taken into account while evaluating an under/overestimation ratio.

In case you experience underestimation, then you just cannot correctly evaluate how many hours or story points needed to complete a specific user story and do not meet clients’ needs.

For example, you have estimated 12 hours for a certain user story, but the work was done within 24 hours. This variation of 12-hour overwork leads to extra expenses for clients and high dissatisfaction.

What are the core reasons for underestimation?

  • The team is not familiar enough with the product
  • The team experience difficulties with the tech stack
  • The team has low skills in estimation.

 

How to solve underestimation issues?

  • Take a training course on advanced estimation
  • Conduct a sharing session on the product to better discuss all nuances
  • Divide project work into small pieces.

 

What are the core reasons for overestimation?

  • Lack of trust and confidence so that the team is afraid to make mistakes.

 

Effort estimation variance is one of the most effective Agile Scrum reporting metrics.

It is a valuable metric that allows you to track actual efforts made over the estimated hours to improve future performance.

Let’s take a look at an example of how to calculate effort estimation variance.

Here is a table of sprints, estimated hours, and actual efforts in hours that were spent:

 

Sprint EstimatedActual Effort
Sprint 2 130 40 
Sprint 3 190 190 
Sprint 4 140 200 
Sprint 5 200 189 
Sprint 6 221 178 
Sprint 7 220 215 

 

As you can see, only the third sprint has the same amount of estimated and actual efforts put into software development. Other sprints were either over- or underestimated.

So, there are two simple formulas that allow you to calculate over/underestimation values:

  • Overestimate = (E-A)/E * 100
  • Underestimate = (A-E)/E * 100

, where A – Actual effort and E – Estimated effort.

 

This way, we can easily calculate the effort estimate variance for each sprint in a percent:

 

Sprint Variance 
Sprint 2 69% 
Sprint 3 0% 
Sprint 4 -43% 
Sprint 5 6% 
Sprint 6 19% 
Sprint 7 2% 

 

Based on the data above, you can safely assume, that Sprint 2 was excessively over-estimated (by 69%!) while Sprint 4 was underestimated.

If you see a negative variance, it is a sign that something is wrong.

Your team either doesn’t have enough knowledge to properly evaluate the complexity of the tasks or lacks details or you should check the individual productivity of team members.

Next, you can define the over- and underestimation limits in percent to draw an effort estimation variance chart and bring into sharp the defects of each sprint.

It allows you to analyze the gaps from sprint to sprint and improve your estimation skills based on the analytics data.

Below you can see an example of an effort estimation variance chart:

 

An Example of Effort Estimation Variance Chart | Ascendix Tech

An Example of Effort Estimation Variance Chart

3. Scope Change (SC)

Scope change is another valuable and complementary Agile project metric that allows you to analyze root causes of no meeting commitments.

Literally, scope change helps you track the number of stories added/removed within the project during spring or release.

This metric has 2 key indicators ‘descope’ and ‘scope increase’ that help you determine when and why any mistakes or changes were made.

Here are two formulas to calculate scope change indicators:

  • Descope: = (D/C) * 100
  • Scope increase = (A/C) * 100

, where D – removed SP, A- added SP, C – committed SP.

 

So, let’s take a look at our example input data and estimate the final variance:

SprintCommittedAddedDescoped
Sprint 2 29 10 
Sprint 3 34 
Sprint 4 16 23 
Sprint 5 30 15 
Sprint 6 45 
Sprint 7 35 

As you can see, sprints 2, 4, and 5 were modified by adding or de-scoping story points.

The key reasons for de-scoping stories are:

  • The Product Owner doesn’t prepare enough all required aspects for the upcoming sprint so he/she is forced to make changes down the road
  • The changes in the priorities of stories.

 

Based on the above table, we can calculate the scope change variance:

SprintVariance
Sprint 2 35% 
Sprint 3 0% 
Sprint 4 144% 
Sprint 5 50% 
Sprint 6 0% 
Sprint 7 0% 

Ultimately, you can build a scope change chart that more vividly demonstrates and allows you to track the variance of changes from sprint to sprint.

An Example of Scope Change Chart | Ascendix Tech

An Example of Scope Change Chart

4. Defect Leakage

As your software product grows, the number of bugs reported will naturally increase and influence the final estimation, client happiness, and team satisfaction.

Is there any way to track the bug ratio within the project scope and improve on? Sure, take advantage of defect leakage.

So, it is one of the key Agile metrics which allows you to track the summarized number of defects detected when a sprint or release is done.

Primarily, this metric is about the efficiency of quality assurance processes as most bugs become transparent at the stage of testing the delivered software.

At Ascendix, we have established a company RAG (red, amber, green) status which allows us to get a relation coefficient/variance to measure and track a project’s activity in a fast and easy way.

Different projects require different Scrum metrics due to the fact that net indicators cannot and shouldn’t be analyzed identically across multiple projects.

So, defect leakage is a good example of using RAG status to categorize indicators’ values and measure them more efficiently.

Let’s take a look at how we categorized possible variance values based on our RAG status:

 

LevelFromToTotal
Green 0% 5% 5% 
Amber 6% 16% 11% 
Red 17% 100% 84% 

 

Now let’s consider the example input data table where we summarize the detected sprint defects:

 

SprintDefects During SprintDefects Post Sprint
Sprint 2 20 
Sprint 3 10 
Sprint 4 30 

 

In order to calculate the defect leakage value, we need to use the following formula:

Defect leakage = (Post sprint defect count / defects detected during sprint) * 100

This way, we can measure the variance value for each sprint:

 

Sprint NumberVariance
Sprint 2 5.00% 
Sprint 3 0.00% 
Sprint 4 13.33% 
Sprint 520.00%
Sprint 633.33%
Sprint 727.27%

 

Ultimately, you can build a defect leakage chart that vividly demonstrates the volume of defects at every sprint completed:

Scrum-An-Example-of-Defect-Leakage-Chart-Ascendix Tech

An Example of a Defect Leakage Chart

Good or bad, all products have defects. However, you can keep a record of defects found and repaired prior to release to maintain a high-quality standard in your software.

Equally important is to make sure that all defects found after a release or sprint are integrated back into the developers and quality assurance processes.

So, try defect leakage metric for your project and improve the software quality you deliver.

Why Use Scrum Metrics?

Agile productivity metrics provide Product Owners, Scrum Masters, and developers with the visibility they need to assess quality and track productivity throughout the software development lifecycle (SDLC).

Metrics derived from Agile workflows measure agile processes such as velocity and task board activity.

So, Agile project metrics first help evaluate the project development process and indicate positive and negative points that influence the Scrum team, process, and overall performance.

What’s more, when you detect some problems that slow down the development process, Scrum performance metrics will help you determine the root cause so that you can eliminate it quickly.

However, you should clearly realize that NOT all Scrum metrics and reporting can be efficiently implemented for any kind of a project.

As Scott M. Graffius, an author of ‘Agile Scrum: Your Quick Start Guide with Step-by-Step Instructions’, once said:

If you don’t collect any metrics, you’re flying blind. If you collect and focus on too many, they may be obstructing your field of view.

- Scott M. Graffius

This means that different metrics work for different projects and there may be no sense in implementing additional Agile KPI metrics if they do not provide value.

Let’s check the key principles you should follow to make metrics effective to evaluate and improve your project management:

#1 A metric should be used by the team

Metrics should not be implemented just to follow the Agile Scrum methodology. They should be used by the team to learn, improve, and deliver a more valuable product to customers.

#2 A metric should be based on a conversation

Again, use Agile software development metrics wisely to start a discussion about the process and roadblocks that negatively affect a team’s performance. Metrics are NOT about numbers; they are about changes and analysis.

#3 A metric should answer concrete questions

All Scrum metrics that you implement in a project should answer specific questions so that you can understand what should be improved.

#4 A metric should be used together with other metrics

If you use a single metric to evaluate a project, you may bump into a tunnel vision that obfuscates the reality due to the lack of analytics data. This way, we recommend using several Agile productivity metrics together to get a balanced picture of your project activity.

#5 A metric should be easy to calculate and understand

In case you find it difficult to use a metric, then it most likely will not become useful in guiding daily activities, even if it provides great insights into your team’s work.

What is Agile and Scrum?

The word ‘agile’ has the following meanings in the Oxford English Dictionary (OED):

  • able to move quickly and easily
  • able to think quickly and in an intelligent way.

Simply speaking, these are the cornerstones of your business success. If you are not flexible to obstacles and do not make decisions quickly – you are most likely on the competitors’ heels.

Agile mindset is totally about flexibility, simplicity, and high speed. Agile stands for a complex of software development methodologies that are based on an iterative approach. It allows you to deliver software incrementally instead of all at once.

Agile is about any development process that corresponds with the Agile Manifesto principles. This is a collection of software development concepts established by 14 leading tech specialists back in 2001.

To be short, Agile Manifesto values are as follows:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan.

 

Key Agile Manifesto Principles | Ascendix Tech

Key Agile Manifesto Principles

What Are The Benefits of Agile Software Development?

#1 Prime quality product

Agile software development with Scrum is divided into multiple sprints during which clients/product owners can make changes to the requirements and functionality.

This way, the chances to get a software product fully compliant with clients’ needs are much higher than following traditional methodologies when on-going changes are impossible.

#2 High customer satisfaction

As the clients are constantly involved in the decision-making and development processes throughout the entire project delivery, they become fully satisfied with the process, result, and reduced time-to-market period.

The latter becomes possible through high involvement from clients and close interaction during the development process.

#3 Controllability

Agile project management provides high transparency, feedback integration, and daily progress reports that help managers and stakeholders better control the project development process.

#4 Better project predictions

It becomes easier to indicate and predict risks when the project development is divided into multiple sprints with continuous improvements and alterations. This way, managers can easily predict sprint-to-sprint changes and plan efficiently.

#5 Low risks

Advanced risk management is a hallmark of Agile software development with Scrum as small sprints and continuous delivery allow to better predict the upcoming development phases and reduce potential risks.

#6 Continuous improvement

As one of the 12 key concepts of the Agile Manifesto, continuous improvement means that managers and developers perform error correction sessions from sprint to sprint and enhance the delivery quality and results of the Agile software development project.

 

What is Scrum?

According to Wikipedia:

Scrum is an Agile framework for developing, delivering, and sustaining complex products, with an initial emphasis on software development, although it has been used in other fields including research, sales, marketing and advanced technologies.

- Wikipedia

Put it simply, the Agile Scrum framework a lightweight methodology which helps a cross-functional and self-managing team to deliver value. It’s also scalable and can be used by 2 or more Scrum teams using Scaled Agile frameworks for more complex projects.

 

Besides, Scrum has a system of roles:

  • A Product Owner

An apologist who perfectly knows the Product Goal. Being a part of a Scrum team, the Product owner keeps the needs of many stakeholders in the Product Backlog, which should represent the Product Goal. So, the main accountability of the Product Owner is to keep Product Backlog items clear, highly transparent, and well communicated.

  • A Scrum Master

A person who helps to establish and maintain a Scrum framework for the team. This person helps Developers accomplish and deliver sprint increments without disturbances and distractions.

Scrum Master’s main accountability is to keep the Scrum team effective. This is achieved through coaching the team to be self-managed and cross-functional, keep the focus on goals, resolve roadblocks and maintain Scrum events.

  • Developers

A self-managing and cross-functional team of specialists that have the required skill set to deliver value each sprint. In the context of software development, Developers may carry out such activities as prototyping, UI/UX design, software development, testing, etc. They approach user stories and always follow their priority to deliver the required functionality first.

Agile Scrum Roles | Ascendix Tech

Agile Scrum Roles

Apart from roles, Scrum also has four essential events (ceremonies):

  • Sprint Planning

The first meeting includes all Scrum team members (Scrum Master, Product Owner, and Developers) to collaborate, discuss the work required to achieve Sprint Goal.

Most often, a Scrum Master facilitates this meeting, where the Product Owner and developers ask/answer questions about execution and acceptance criteria.

  • Daily Scrum

A day-to-day meeting during a sprint often takes around 15 minutes. It is organized to schedule the day’s work program towards the Sprint Goal.

Also, Daily Scrum allows developers to discuss current user stories in order to inspect and adapt their working processes on a day-to-day basis.

  • Sprint Review

A demonstration meeting at the end of a sprint to show off the product developed during the last sprint to customers and stakeholders.

Simply put, sprint reviews allow stakeholders to ask clarifying questions to better understand what was done in detail and adjust the Product Backlog to better suit the Product Goal.

  • Sprint Retrospective

A final meeting before a new sprint where a Scrum team discovers the process, what went right, and how to improve communication, tools, and the process itself to increase the delivery quality and effectiveness.

4 Key Scrum Events | Ascendix Tech

4 Key Scrum Events

 

Key Agile Scrum Artifacts

The framework includes three main artifacts that refer to a kind of work that should be done to finish sprints and the project. They help make internal information transparent for all team members.

Before we start reviewing them, we should mention that each artifact includes a commitment.

According to The Scrum Guide:

Commitment provides information that enhances transparency and focuses against which progress can be measured. It exists to reinforce empiricism and the Scrum values for the Scrum Team and their stakeholders.

- Scrum Guide

So, they are as follows:

  • Product Backlog

The list of required actions that should be completed during the entire project development process to deliver a high-quality product.

The commitment of Product Backlog is a Product Goal.

  • Sprint Backlog

All the tasks created for a particular sprint should be done within one iteration. A Sprint Backlog is created based on Product Backlog during a Sprint Planning meeting.

The commitment of Product Backlog is a Sprint Goal.

  • Increment

A sum of all Product Backlog elements that were completed during a sprint and the value of all previous sprints. By the end of a sprint, an Increment should be “Done” which means that it is functioning and meets the acceptance criteria of a Scrum team.

The commitment of Product Backlog is a Definition of Done.

Main Agile Scrum Artifacts | Ascendix Tech

Main Agile Scrum Artifacts

Final Thoughts

With the help of the Ascendix Tech Project Management team, we shared with you the Scrum metrics we use when developing software. They can help you better analyze and measure your team’s performance and project progress to improve on.

So, try these Agile metrics for your projects and find those that help you enhance the final software quality and value delivered.

What ways do you use to measure software quality? Do you use any metrics? Share your experience in the comments below.

If you are looking for a reliable software development company with expertise and experience in both creating top technology solutions and effectively managing projects, feel free to contact us.

We will be glad to discuss your project, capture your goals, and transform your idea into a successful custom software on time and on budget.

Share:

Leave a comment

Your email address will not be published. Required fields are marked *

Comment
First name *
Email *

Subscribe to Ascendix Newsletter

Get our fresh posts and news about Ascendix Tech right to your inbox.