Follow These 6 Steps and Your SaaS Product Quality Will Rock n Roll

You are a true test driven evangelist in your organization. You think you figured it all out - you have your developers writing unit tests and have sonarqube to prove it. You have automation tests covering all possible end-points. You have built glorious UI tests implementing every story DoD as defined by the Product Owners. But your quality remains low. Customers consistently complain about severe bugs, broken functionality and system’s instability, and you have no clue how you got here.

While working in Agile, be it Scrum or Kanban, most startups’ R&D teams will implement automation as follows: with every Story defined, the PO (i.e. product owner) will have a discovery session with the team’s technical point of contact. That can be a Team Leader, a Technical Leader (aka tech-lead) or an expert on the subject matter who provides feedback, asks challenging questions and comes up with Story estimations. All compiles into a well-defined estimated story.

Done right, this is an on-going process done just in time. Working in Scrum, that process typically occurs during the current Sprint as a discovery session, done before the next Sprint starts. Working in Kanban, that process is done continuously. The team will pull the highest “baked stories” in priority from the backlog, and jointly groom them.

What typically happens next is that the team begins working on groomed Stories as part of the new Sprint or by priority when in Kanban — breaking them down to both development and QA related tasks and assigning them to the relevant developers or testers.

That’s when the developers start working on their tasks - quick napkin or whiteboard designing/modeling phase, peer design reviews, code writing, unit tests, pull request, green build and PUSH!

Related: Can I Become a VP R&D Unicorn Applying the 10,000 Hours Rule?

Testers, on the other hand, will typically start with breaking down the Stories into test-cases. If the PO was thorough, the tester’s life may be a bit easier, as the story DoD already considers many for the test-cases. Where feasible, most of the test-cases will be automated leaving manual testing for only few (this adds technical debt that we should try to avoid, and a topic for an entirely new post).

Now, when automation actually runs, it really depends on your startup’s specific continuous integration (i.e. CI) implementation.

Well, “not bad” won’t cut it.

Let me explain.

A technical-leader working on a story will typically try to see the broader picture and try to understand how that single story falls into place with all other stories. How that additional piece of code written, falls into place with all other moving parts and pieces of the system. How she can design it to match the overall architecture and standards. How she maintains a clean API. How she reduces the overall system entropy with every line of code written.

She would even have a peer review on architecture and design and consult with the architect (or VP R&D if one doesn’t exist) to ensure that no concepts or standards have been broken.

Rarely have I seen the same paradigm applied to testing in startups.

In most cases, that unique point of view is lacking or overlooked.

As VP R&Ds it’s our responsibility to ensure that the same mechanisms are followed across both our production code and the code that tests the production code. CODE is CODE is CODE.

QA should adhere to the same standards. Given a story, and before writing a single line of automation code, we should take a broader view and consider the Epic (i.e. series of Agile stories) as a whole — both the stories we have already automated and those we haven’t yet. As well as what business use-cases we are trying to deliver our customers, and the ways in which customers be using the product.

A Story by definition adds incremental value and increases the overall value proposition. In regards to Quality Assurance, one Scrum or Kanban story doesn’t tell the entire “story”.

Similar to the processes applied by developers, we have to educate our organization to follow the same principles and concepts when it comes to testing. Otherwise, you may have hundreds of tests coming out green every single time, but still have low customers’ use-cases coverage.

Personally, I don’t believe that any type of testing can reach 100% coverage, but it does have a nice tune to it when you write a blog post title. But seriously, even if there were such thing as 100% test automation coverage, I would never set it as a goal.

What does 100% coverage mean anyway?

Does covering 100% of every code line written with unit-tests falls into that definition? Does covering all public API end-points and all possible JSON contract transformations with integration tests apply? Should your team have 100% unit-tests, system-tests, integration-tests and UI tests coverage all at the same time?

The answer is clearly no.

As our main focus here is setting the right automation testing mindset in the R&D organization, I would say that; on a very limited time, high pressure to meet deadlines, scarce resources and constant change priorities, perfect is the enemy of the good, and aiming for that 100% coverage objective is a total waste of your teams’ time, effort and focus.

Assuming all automation development, provisioning, monitoring and reporting tools are already in place, here is what I suggest you do:

Related posts:

Do you use Google Voice to stay in touch with your customers? The first impression with new customers is important, and a message on an answering system is sometimes the only impression they’ll get…

Written by Donald Wooller, Customer Protection/Business Management Lead of Cyber Security Team at HMRC Digital Hi, I’m Donald Wooller, part of HMRC’s Cyber Security Team. Our work on SMiShing…

A positive community is a wonderful thing. It’s fun to read and almost irresistible to join. You instantly feel welcomed and quickly make new friends. Carefully managed communities tend to be…