Beef up your flow, flex your monitoring muscles and eliminate waste from your diet (or tech delivery!). Metrics can help you become a lean mean delivery machine by tracking your performance and tweaking your routine to get results.
Like in the gym, the world of software delivery has an ever-increasing drive to work faster, smarter, and more efficiently with a bigger impact. For this metrics are your friend. But which ones? What should you measure and what does it tell you? In the IT world of Agile delivery, there are a number of different focus areas you can track to help yourself learn and improve. I’ve broken them down into the following categories:
- Ways of working metrics
- Quality metrics
- Non-functional metrics
- Business Key Performance Indicators (KPIs)
- Project Management metrics
Some can be measured in a clear numerical way by counting, time or story points. Others are more qualitative or categorised. Before delving a little more deeply into each, it feels a good time to express the importance of creating a balanced scorecard when targeting areas to improve. If your focus is purely aimed at one single metric, it could be at the expense of another, with the overall outcome being the detriment of your business. From a continuous improvement perspective, if you balance your targets across a range of different focus areas and repoint that focus once you have mastered each area, then you will be set up for success. Some good headings to ensure a range of perspectives are Customer, Operations, Ways of Working,
Quality and Commercial
In the words of Arnold Schwarzenegger, “If you never do leg day at the gym you end up looking like a chicken” –completely missing the overall goal! Another issue with too much focus on one specific area is that it’s easy to turn improving into gaming, or basically cheating. You then lose the potential benefits and lose sight of the bigger picture.
Ways of Working Metrics
These are those that give insight into how well your IT delivery processes, and your Agile teams are working. They will help you to reduce bottlenecks and eliminate wasted time. They will also improve the predictability of your delivery capabilities.
Delivery Process Metrics
These are your agile KPIs:
- Sprint burndown chart - tracks sprint progress or can be used at the next level up at initiative or project.
- Velocity - Productivity in amount done per sprint measured in story points or time. Should take account of capacity.
- Lead time - includes the time from when a task is accepted into the backlog while waiting to get in-progress and then to done status.
- Cycle time - time from in-progress to completion.
- Flow - Shows over time the number of tasks in each delivery phase. This is a Lean metric that can help identify and reduce bottlenecks, blockers and waiting time. It can indicate areas where things queue up and go over the ‘Work In Progress’(WIP) limit. A good way to visualise this information is with a cumulative flow diagram. Flow is a key Agile metric for both Kanban teams and Scrum teams.
- Waste - Another Lean metric, waste tracks time spent without progress, such as context switching, blockers, rework, failed deployments.
- Estimate accuracy – reflecting on estimate quality in story points, delivery estimates, and business case estimates. Helps to improve predictability.
- Average age of backlog – similar to lead time, can help to manage stakeholder expectations.
People Metrics
- Team Satisfaction/Morale
- Personal Development time spent
- Skills matrix and competency framework score against Target Operational Minimum (TOM)
Quality Metrics
Quality of a deliverable can represent a number of different aspects – how well does it work? Does it do what it’s meant to? Is it quick enough, safe enough or compliant enough to serve your customers?
Delivery Quality
- Post-live issues - any issues missed during testing are worth tracking and analysing to prevent similar issues from getting through in the future.
- Customer satisfaction scores - try XLAs - the new improved SLA. Experience Level Agreements (XLA) help ensure the customer experience is positive both within Service Delivery and Software Delivery.
- Product Owner satisfaction scores – any missed requirements? Is it fit for purpose?
- Vulnerabilities – Security issues, can be identified with code-checking tools, penetration testing or other security scanning tools.
- Technical Debt levels – when projects are rushed, coding can be done the quick easy way leaving problems that could slow you down in the future. If left unmanaged, this will likely catch up with you at an inconvenient time and get in the way.
Automation
Although automation could also be considered ‘ways of working’ by enhancing efficiency, I’ve included it under quality as it undoubtedly improves quality through removing the risk of human error. Measure adoption levels of the following:
- Test automation code coverage - for example unit testing, integration testing or regression testing
- DevSecOps - Automating Security throughout delivery life cycles
- DevOps - Release and deployment automation with automated environment management
- Automated service management, monitoring, alerting and error handling - many cloud providers have services to support this capability
- Automation of any operational tasks or Service Delivery tasks - Artificial Intelligence (AI) and Robotic Process Automation (RPA) can automate many repeated tasks saving time
- Metrics - don't forget to automate your metric creation, data crunching can be extremely time-consuming!
Technical best practice
It is good practice to have best-practice guidelines! These can support and educate your delivery teams, whilst ensuring the quality of your deliverables. At Airwalk Reply, we model our deliveries against the systems and software quality guidelines in ISO 25010 which breaks down quality into eight categories that include architectural elements alongside usability and functionality. Other examples include principles such as cloud well-architectured frameworks. Scoring your quality against these can support tracking of technical debt, but also indicate the quality of the delivery as it improves over the iterations released. It can feed into Acceptance Criteria and support your overall outcome tracking.
- Coding best practice
- Security best practice
- High Availability assessments and best practice
- Cost Optimisation including FinOps
- Operational best practice – e.g. good housekeeping such as storage, retention or backup routines
Non-functional Metrics
Non-functional capabilities are often deprioritised when time is tight, however, you could pay dearly once your change has gone live. It may function as intended but if it is not robust then it may not serve you for long. Introducing metrics in these areas can help you monitor for a reduction in the quality of your operations.
- Performance - how fast is the service?
- Capacity - how many customers can be served at once?
- Availability - 24/7 or only at lunchtime on a Wednesday?
- Monitoring coverage - Do you know the health of your system?
- Error handling - If it goes wrong can it fix itself? Does it call for help?
- Maintainability - Has it grown arms and legs that need handling every time a change is needed?
- Risk management - How secure is it, how reliable, how resilient?
- Operational resilience capability – service continuity, disaster recovery and data replication
Business KPIs
Business Key Performance Indicators are those which specifically talk to the performance or success of your product. It could be customer numbers, profit or growth. These will be specific to your business and should be driven by your overall strategy to feed the ‘what’ into the previous sections that talk about the ‘how’.
Project Management Metrics
What is the best measure of progress on an Agile project? With Agile methodologies, work is broken down into smaller incremental deliverables, with each intending to add value, and enable feedback to be received on the success of the change. It can be hard to reconcile this method of change with the Waterfall project management methodologies that deliver projects with key milestones over a planned timescale. With Agile delivery or product delivery methodologies, the focus is on the outcomes not the output. For example the outcome is ‘to increase customer numbers by 50%’ the output might be a marketing campaign. Agile delivery allows the output to change as feedback is received from Minimum Viable Products (MVPs) that are put live to customers. Feedback enables the outcome to be achieved as the best output is discovered, throughout the change process. The best metrics are therefore those that are focussed on progress towards the desired outcomes and demonstrate the incremental value being delivered from each iteration.
How to use Metrics
Understand your metrics – In order to solve problems you must first understand them. Through analysing your metrics you can identify areas where there look to be problems, be it inefficiency, blockers or poor scores against targets. There are many ways to perform Root Cause Analysis on issues that can help you identify the core problem. For example the following:
- The 5 whys
- The Fishbone Analysis
- Pareto Chart
- Scatter Diagram
- Failure Mode and Effects Analysis (FMEA)
In Scrum development teams this could be routinely done as a part of your retrospective meetings, or it could be introduced as analysis brought into the sprint. It is beneficial to share the analysis and learnings so that everyone can improve from the findings.
Continuous Improvement actions can then be added to your next delivery cycle to resolve the issue or improve the metric.
Celebrate Success - Use your metrics to show the team members and your stakeholders how well you’re progressing and the benefits that have resulted. Recognition will enhance motivation and help you prioritise further improvements.
How to Get More for Less by Prioritising Effectively Read more
