How to Create a Quality Strategy and Testing Model

I spent the latter part of 2022 working with my team on a new Quality Strategy. It was such a great experience. Creating the strategy allowed us to capture all of our existing good practices, and refine them. It also allowed us to look forward to what good looks for us like in 2023. Something I am really proud of is the testing model that came about almost as a by-product to our process.

Created by me and my team, we call it the Zoopla Holistic Testing Model. It is designed to give a high-level overview of how Quality and testing forms part of every stage of the software development lifecycle. In doing so, it also acts as a quick reference for What, How, When and Where to test.

We were also able to make it non-specific to Zoopla. As such, it can help other teams in other organisations. Whether it’s treated as a standard to work to, or something more aspirational. I genuinely believe it creates a great starting point for a Quality Strategy for any business or team wanting to “shift left”.

If you’d like to read a deeper dive into the Zoopla Holistic Testing Model itself, you can do so on the Zoopla.blog. On that blog I have given a stage by stage breakdown of the model. It provides a more detailed insight into the good practices we believe contribute to excellent Quality practices.

Rather than repeat that for this blog I am going to focus on creating a Quality Strategy instead. I won’t share the strategy in full. Rather, I will talk about the process we followed to create our Quality strategy. Beyond that, I will share some section headings and the key considerations we had when writing each section.

Creating the Zoopla Holistic Testing Model

When it was time to create a Quality strategy, there was one thing I was sure of. I wanted all of my team to have the chance to contribute. That very first step of team collaboration ultimately resulted in the creation of the Zoopla Holistic Testing Model.

I knew early on that I wanted a core visual aid that would form part of the Quality Strategy. I also know that it would take inspiration from  Dan Ashby’s Continuous Testing Model and Janet Gregory’s Holistic Testing Model. Not only do they well define the way we work as Quality Engineers, we already regularly reference them. As great as they are for Quality Engineers, I felt they lacked the detail needed for other disciplines about How, What, When, and Where to test.

To get started then, I set up a Miro board for my team to start collecting ideas. We started off with a rough looking DevOps loop.

Then started adding notes around the loop about the different ways we can test at each stage of the Software Development Lifecycle. As we did so, we assigned them to stages of our DevOps loop. We also considered where the testing would take place at each stage and added labels for that too. After much collaboration, discussion and refinement, we were happy with what we had.

Before committing ourselves to what we had created, we gathered feedback from the other domains within Zoopla. In this context, a domain describes a part of Zoopla that works on a specific product type, or value stream.  Although we are all part of the larger organisation, due to the way in which Zoopla grew, we often have slightly differing ways of working, tool sets etc. Not only did we value their feedback, but by this point, we had begun to call it the Zoopla Holistic Testing Model. We couldn’t let that name stick if the rest of the Quality discipline didn’t buy into it. Thankfully, they did.

After our final round of refinement with the wider Quality Engineering team, this is what we ended up with.

It was far from the nicest looking thing and not the easiest to read. It had captured our thoughts though, and that was key for the next step. I took this wonderful Miro, and worked with one of our designers. I helped them to understand the purpose and message we wanted to convey at each step. We discussed ideas for the style of the diagram and we worked through those ideas. The end result was the diagram you saw at the top of the page. To save you scrolling up, I am going to share it again here.

The final model. It clearly demonstrates how each discipline can test at each stage of the Software Development Lifecycle. It’s even useful as a quick reference guide. Not only that, but it shows how testing exists at each stage, which helps with breaking down the notion that “QA does the testing”.

How the Zoopla Holistic Testing Model helped us create a Quality Strategy

Creating the Zoopla Holistic Testing Model was just the first part of the journey. We had consolidated our thoughts on What, How, When, and Where we test. Now we had to convert those thoughts into a full blown Quality Strategy. The strategy had to be about more than just testing. It had to convey the idea of a holistic approach to Quality. To do this we broke down our strategy into these main topics:

• What is Quality?
• How we work
• Definitions of Ready and Done
• Measuring Success
• Recommendations for test tooling
• Guidance on test environments

Below I will explain what each of these sections is about and how we decided on the right content for each.

What is Quality?

Quality is not throwing something ‘over the fence’ to be tested; rather it is a goal that the entire team share, from idea to delivery and beyond

Stuart Thomas

This first quote appears right at the top of our Quality Strategy, rather than in the What is Quality? section. It is a phrase I have repeated many times to many people. We wanted to call it out top and centre as it is an obstacle many Quality Engineers face. If you are a Quality Engineer who is facing this, remember you are not alone, and you can overcome it!

When it came to the “What is Quality?” section, there were two quotes that stood out. The first, I am ashamed to admit, is something else I had said.

Good quality practices provide the confidence and ability to develop quickly and release frequently; resulting in a better product and happier customers.

Stuart Thomas

The idea of including this quote was to set a basis from the outset that Quality wasn’t just about testing. Instead that Quality is about making everything better. When I say everything, I really mean that too. Quality isn’t just about the product, it’s about the experience of everyone involved. From the Product Manager, to the Designer, the Software Engineers, the Data Engineers, the Sales person, the Support team, and of course, our end users.

Talking of end users. We couldn’t have a section defining Quality without the below quote from David Williams. This quote was included in a special Zoopla printing of Leading Quality by Ronald Cummings-John and Owais Peer. Side note, if you haven’t read Leading Quality, I highly recommend it!

Quality is “goodness” and “correctness” (just because something is ‘correct’, it doesn’t necessarily mean it’s ‘good’); it’s the value that a customer or user feels while using the product or service, and it is entirely unique to that customer.

So, how do we make things with ‘goodness’ at the centre of them, and make our customers and clients feel great about Zoopla?

Although testing plays a huge part in our perceived quality, testing alone can never guarantee software with goodness at the centre of it; we also need to create and drive a culture of quality, where quality is at the forefront of everything we do, from ideation to product design, to engineering, to release processes, to customer support, etc

David Williams

Between them, those two quotes really capture what Quality is all about – Making things better for all. Not just in the product, but in our processes and tooling too.

How we work

This felt like a bold title. It covers such a large range of topics. Despite this, we felt it was essential. The key consideration for us was that, being such a large section, we had to ensure people could locate information easily. It also had to be simple to read and understand. To achieve this we broke it down further into subsections.

• Quality Principles – Addressed some of the common hurdles that appear when trying to “shift-left”. Such as the assumption that Quality Engineers are “just” testers. As well as launching what may be new concepts for some. Like Quality applying to tools and processes as much as it does the product.
• Approach to Quality – This focused more on the Who, What, Where, When of Quality and Testing. It included general guidance as well as guidance specific to our existing codebase and product.
• How we Shift Left – Here we introduced the Zoopla Holistic Testing Model.
• When and Where to include Quality Engineers – I loved this idea. We broke it down by different disciplines so that it was easier to find examples relevant to a person’s role. For each discipline we provided examples of how they can and should work with Quality Engineers.
• Desirable Quality Gates – Primarily suggestions for automated checks and processes that can be used depending on the code in question. Ultimately intended to reduce manual effort and provide confidence. We also included guidance for our recommended use of CODEOWNERS files.

Definitions of Ready and Done

Definitions of Ready and Done can be really valuable tools to help a team have a consistent approach to creating stories and considering the work “done”. They can also be dangerous if they introduce unnecessarily strict gating. Mike Cohn covers this well in his blog, The Dangers of a Definition of Ready.

The approach we have taken is to provide a basic set of what we consider minimum standards for both Definition of Ready and Done. With the caveat that we accept not everything is relevant to every story and ticket. This way teams are free to build on top of our definitions while not feeling like they have to fulfil everything, every time. That last point we felt was especially important to avoid the Definition of Ready dangers that Mike Cohn talks about.

If Definition of Ready or Definition of Done are new concepts to you, then there are some great descriptions as part of knowledgehut’s Scrum Tutorial.

Measuring Success

Measuring success is an important part of Quality that can be overlooked. It is important that we understand that we have built the right things. Making sure they work correctly in production is a good idea too. There are a huge number of ways we can measure success, and the finer detail of how you do it is dependent on your product and team. A few things we felt were important to consider are:

• Observability – You can use observability tools to measure machine based metrics that indicate product health, but also run regular tests/checks. These regular tests are often called synthetic monitors, they run a script or UI driven test that asserts a particular behaviour. A fantastic tool to have in the arsenal.
• Logging – A staple of software development. Ensuring that you log meaningful and relevant data is the most critical. That way you can be sure to not only have the information you need in case of an incident, but also have meaningful feedback at all times.
• Net Promoter Score (NPS) – I love this graphic for so had to include it rather than typing it out

There are many other ways you can measure success. The last one I want to mention is one that is relevant to all teams, the DevOps Research and Assessment team metrics, or DORA metrics. The DORA metrics consist of 4 metrics:

• Deployment Frequency
• Lead Time for Changes
• Mean Time to Recover
• Change Failure Rate

DORA metrics are an entire topic on their own, and so I won’t write more about them here. To learn more about what they are check out The Ultimate Guide to DORA Metrics by Will Lam

Recommendations for test tooling

It seems obvious, but providing guidance to teams on test tooling is extremely important. Not only does it help to encourage consistency across teams, it also presents an opportunity to recommend tools that aren’t currently in use. This can be especially useful where there is a type of testing you don’t currently have in place but have a preferred tool that you would like to use when it is introduced. If a team has a need for the tool before you put the framework together, and they are willing to invest the time in doing so, they know where to start.

Ultimately, this should also just be guidance. There will always be edge case scenarios where a chosen tool isn’t ideal for all situations. This is where you should help teams to identify the best solution, whether that’s using a different tool or creating a solution to the problem in the existing tooling.

Guidance on test environments

All of the engineers know how you use your different test environments, right? Why would you need to provide guidance on this? Teams change. The strategy should be as much about supporting the existing staff as it is an aid to onboarding new staff. As obvious it may seem now, new staff don’t know how you use your test environments and so including this kind of information in a Quality Strategy can be immensely valuable.

Core Quality Strategy vs Living Documentation

One last thing to consider. Not everything has to form part of the core Quality Strategy document. In fact I would encourage you not to attempt to do that. Instead, keep the core principles and decisions within the Quality Strategy document. From there provide links out to further reading within what I would call your “Living Documentation”.

The key concept here is that your Quality Strategy document should rarely change. The Living Documentation that supports it though, will. These are the documents that you update as you improve and refine your processes etc. I find this way that your core Quality Strategy remains true to the original intent and the things that should evolve are free to do so.

Remember, a good strategy takes time

A Quality Strategy serves multiple purposes. 

• It outlines how Quality and Testing should be treated across the board. Supports onboarding new joiners to our ways of working.
• Provides a reminder about our practices and processes.
• Acts as a central point for discovering the relevant living documentation that supports it.
• That’s a lot of things!

Don’t underestimate the amount of work that goes into creating a good Quality Strategy. When you start you might think you’ll be able to knock something up quickly. Once you start really thinking about your existing practices, and where you want to go, it will quickly build. The more you work on it, the more you remember topics you need to cover, and there will be more than you expect.

Creating all of the supporting documentation takes time too. Likely longer than creating the core strategy. Then just as you think you are getting close to the end, you’ll want to seek feedback from key stakeholders prior to a full rollout. This in itself takes time. It will also likely result in further iterations. So when you start, make sure you give yourself plenty of time to get it right. The results will be worth it.

Once you have it produced, be sure to introduce it to the entire tech team, product, designers, engineers, etc. For a strategy to be successful you need everyone to buy into it. A well constructed presentation where you and your team get to share your passion for Quality and what you have produced is a great way to get that initial buy in.

Summary

So there you have it, an overview of how we created the Zoopla Holistic Testing Model and the process we followed for creating our Quality Strategy. Please feel free to use the Zoopla Holistic Testing Model as part of your own strategies. If you do, let me know how it goes down with your teams. I would love to hear their thoughts, and yours.

Further Reading

If you enjoyed this post, then you may also enjoy my recent post on Why you need to (not) use Story Points. I discuss why I am not a big fan of them, but also how to use them in a way that has worked for many teams I have worked with.

Main Photo by Patrick Perkins on Unsplash

Subscribe to The Quality Duck

Did you know you can now subscribe to The Quality Duck? Never miss a post but getting them delivered direct to your mailbox whenever I create a new post. Don’t worry, you won’t get flooded with emails, I post at most once a week.