but which one is of higher perceived quality?

Simon Prior recently shared this question as a challenge on LinkedIn.

There are 2 mobile apps which provide similar functionality but which one is of higher perceived quality?

App 1 - Lots of bugs, but customers love the app despite it's quirks and it's user base continues to grow

App 2 - No defects, all requirements met, but despite being in the field for as long as App 1, no one is using it and it has poor reviews on the app stores

Those of you who saw the post will likely have seen I wrote quite a lengthy response. I ran out of space in a single LinkedIn reply though and had to write a shorter summary. So I decided I’d steal the question and write a blog post about it.

To answer the question fully, we really need to break it down further. The first question we need to ask ourselves is, what is quality?

What is Quality?

There are so many answers to this. They tend to vary based on who is answering, and what it is they are trying to measure. To me, Quality is fundamentally about how we experience “things”. The definition that always resonated the most with me is this one written by David Williams in a special Zoopla printing of Leading Quality.

Quality is “goodness” and “correctness” (just because something is ‘correct’, it doesn't necessarily mean it's ‘good’); it's the value that a customer or user feels while using the product or service, and it is entirely unique to that customer.

So, how do we make things with 'goodness' at the centre of them, and make our customers and clients feel great about Zoopla?

Although testing plays a huge part in our perceived quality, testing alone can never guarantee software with goodness at the centre of it; we also need to create and drive a culture of quality, where quality is at the forefront of everything we do, from ideation to product design, to engineering, to release processes, to customer support, etc

I reference this often. It is my go to way of resetting thinking about Quality, whether in the context of an application, like in this question. Or whether in the context of processes, tools, ways of working etc. We experience all of those “things” and the more they make that experience good with high levels of correctness, the better our experience and the higher we consider the quality.

A chance to talk about video games

I loved Simon’s question. It goes against what you would expect to be the case. We see it all the time though and I want to share some real examples.

When Cyberpunk 2077 launched 3 years ago, it was a broken mess¹. Despite this, it was a massive success. Within twelve hours after its release, it had over one million concurrent players on Steam² and received generally favourable reviews³.

Conversely, when No Mans Sky launched in 2016, it was missing functionality, but worked beautifully. Yet it received mixed reviews⁴. It also received backlash because of the missing content⁵.

Over the coming years, both games had to release huge updates. Cyberpunk had to fix the bugs. No Mans Sky continued to extend the content. Today? Both games are considered excellent.

Before any gaming fans come for me, I appreciate this is a massive over-simplification and both games had their fans and critics at all stages. Still, it made a good comparison.

So how does this apply to the apps?

App 1 – Lots of bugs, but customers love the app despite it’s quirks and it’s user base continues to grow

App 1, like Cyberpunk2077, has “goodness” but not “correctness”. The apps users like something about it that makes it a “good” app, despite its quirks. Eventually though, just like the developers of Cyberpunk2077 they will have to fix those quirks. Goodness alone is not enough and they will need to address the lack of correctness.

App 2 – No defects, all requirements met, but despite being in the field for as long as App 1, no one is using it and it has poor reviews on the app stores

App 2 has “correctness” but not “goodness”. That is, it does exactly what it was designed to do, as it was designed to do it. Unfortunately the designs lacking something that makes its users consider it “good” and so it has poor reviews.

What can we learn from this?

This is a great demonstration for the value of taking an holistic approach to quality and testing.

If app 1 had better quality and testing processes within the build and development stages it would probably have less bugs. Any issues that exist in the app could have been caught earlier. Providing engineers with the fast feedback they need to address them before they reach production.

If app 2 had better quality and testing processes within the design, ideation, and planning phases, it would have been able to gather feedback about what would give it goodness. This would have meant earlier communication with potential users to uncover what good looked like to them so that they can be sure they are building the right things.

Isn’t this all just hypothetical? What about the real world?

It sure is. This is just a nice thought experiment. It got me thinking about the perception of Quality and how a well implemented holistic approach to quality and testing has many benefits. No matter what systems you have in place, defects will escape and features wont have the expected level of success.

What an holistic approach to quality and testing does is minimise that happening and improve the experience of everyone involved with your product.

The implementation will look different for each company, product, and team. You have to build the processes right for you. However, the basic framework and thought exercise is the same, no matter what you do.

Subscribe to The Quality Duck

Did you know you can now subscribe to The Quality Duck? Never miss a post by getting them delivered direct to your mailbox whenever I create a new post. Don’t worry, you won’t get flooded with emails, I post at most once a week.

Newsletter