Zero Bug Policy
At Zoopla, Quality is at the forefront of everything we do. When it comes to bugs, we handle them as quickly as possible, to focus on quality rather than storing and managing them. Every bug is important to us, and we treat them fairly in terms of their impact on the business. We believe in addressing all bugs as they occur and making a plan for them right away so they don't get lost in backlogs for these reasons:
- With too many bugs, important and relevant information gets lost in the noise and housekeeping work increases in terms of managing bugs.
- Bugs tend to become obsolete quite rapidly in frequently updated systems.
- The steps to reproduce become irrelevant, the functionality changes, or the impact of the bug often gets lower.
We regularly monitor backlogs to ensure that bugs are always factored into sprints for resolution, and prioritised accordingly. Since we are talking here about bugs, before moving on, here are some definitions and their meanings:
Bug
In simple terms, any functionality which is not working as expected and affects the user experience is a bug. A bug could also be classified as defect, error, fault, issue or failure.
Zero Bug Policy
Zero Bug Policy is to be proactive when it comes to bugs. As soon as the bug is reported in the backlog, the life cycle of the bug is determined to be either ‘fix' or ‘don't fix'. In this way, you can focus on quality compared to managing bugs that often lead to tedious tasks.
Assessment of bugs is the key towards a zero bug policy. Therefore, triaging bugs have the greatest effect in terms of its success.
Bug Triage
Triaging is the process of reviewing bugs to ensure they are valid, reproducible, and have accurate information that allows them to be resolved and tested. After a bug is triaged, it should be sequenced according to the priority for fixing it.
Regular bug triage is important, especially before sprint planning to ensure all important bugs are fixed as part of the sprint.
It's always a good idea to quantify the bug’s impact on users or customers. An example would be ‘Bug X impacts 5% of the user’ has a lower priority compared to ‘Bug Y impacts 50% of the user’.
Approach towards 'A Zero Bug Policy'
Once a bug is reported in the backlog, after a thorough assessment of the impact with the team, bugs may fall into any of the three categories such as:
- Fix it Now
- Will Not be Fixed
- Converting bug to Story or Task
Here, the team is defined as the technical lead, engineer, quality engineer, product manager or any other relevant stakeholder(s).
Bugs raised can be referenced at any time so here we do not remove/delete bugs, we simply decide the status of bugs.
Fix it Now
If the bug has an impact on business, and you are able to prioritise it. Bring it to the sprint and fix the bug.
This bug could result in a loss of revenue or potentially a loss of customers due to a bad functional experience. These bugs need to be addressed immediately.
Will not be Fixed
If there is no certainty regarding the bug and the impact of the bug is not determinable.
Think of a bug where there is no loss of revenue or a small percentage of the users are impacted.
Consider converting a bug to Story or Task
Think about a situation where you decide that this bug has the potential to be fixed but not just now, or you decide that the scope of a fix is more than a bug, for example:
- Accessibility/ Performance/ Security, one cannot resolve these, part of a single bug. This must be corrected incrementally and requires a broader discussion and estimation of the team.
- Architecture or Design Bugs, not in a position to fix the bug straight away as it requires architecture or design change which is more than a bug fix.
- CI/ CD pipeline improvements.
Measure success towards ‘Zero Bug Policy’
How can we measure this? It’s easy! First, subtract the total number of open bugs after the zero bug policy from the total number of open bugs before adopting zero bug policy, which will provide you with the Bug Reduction figure.
Here is the formula:
- Bug Reduction = Total number of open bugs before adopting the policy - Total number of open bugs after adopting the policy
- Bug Reduction % = Bug Reduction * 100 / Total number of open bugs before adopting the policy
Before Zero Policy
You’ll notice there’s always a bug backlog in existence, which is harder to manage.
After Zero Policy
You can notice there has been a sharp reduction in the number of bugs after adopting the policy.
Benefits of 'Zero Bug Policy'
- Increased Product Quality - This prevents teams from becoming desensitised to bugs and therefore more sensitive to the user experience.
- Increased Customer Satisfaction - By addressing bugs quickly, we are prioritising our customers and their experiences as high priority.
- Increased Agility - This allows us to be proactive rather than reactive regarding product improvements.
- Increased Collaboration & Communication - The team engages all relevant stakeholders in deciding the lifecycle of bugs
Best Practices/ Zoopla Recommendations
- The team is responsible for shifting focus from bug management to quality.
- Collaborate with Product Managers, Delivery, Engineers and all relevant stakeholders to communicate the "Zero Bug Policy" process from the beginning so that they have clear visibility.
- Work with the team and stakeholders to determine the appropriate percentage (e.g, 5 - 10%) of the sprint capacity for bug fixes.
- Raise bugs with clear problem statements, success criteria for work to be done, attachments or videos are always helpful in replicating bugs.