Jamie also blogs about Testing, Lean and other things at GeetarWords

I've been working with the talented Stuff product technology team as Test Chapter Lead for most of 2015.

This is the first in a series of articles on how we've approached testing at Stuff. This article will look at how we've captured our Test Strategy in a 1-page visualisation of our guiding principles. The next article will look at some of the experiments we've run based on the strategy and what we've learnt from them.

Background

Fairfax NZ Product Technology is the in-house development shop responsible for much of the stuff.co.nz platform (NB: there are various 3rd parties looking after other parts of the platform, such as Mags4Gifts).

We're currently grouped in two agile delivery teams - Dream Team and Team Lamp. The teams came together in a self-selection exercise in the middle of the year, based around team members aligning themselves to specific business goals.

Dream Team have been focused on upgrading the Stuff video platform and on experiments to increase Stuff membership. Team Lamp (the name a team who can't come up with a name gives themselves!) have been developing the new mobile Stuff article page and implementing Facebook Instant Articles.

There are 2 Testers working in each of the two development teams, plus one focused on release testing and myself working as the Test Chapter Lead. My Chapter Lead role involves coaching and supporting the Testers and the wider development teams. It has also included a strong focus on automation, driving implementation of our first BDD automation framework and ensuring we concentrate on developing high value automated tests over large quantities of automation.

In my time working here, the growth of the Testers has been absolutely outstanding. They have developed into collaborative quality advocates and assistants, capable of working with whatever tools and approaches best fit their context. They had open mindsets and a strong desire to learn from the start; with coaching and support from many corners and a real appreciation of their value from their teams, they have become more confident and willing to lead their teams towards higher quality products.

Visualising a Test Strategy

One of my first tasks with the Stuff team was to implement a test strategy for our story / feature development. The strategy was aimed at ensuring testing and a quality-focus is integral to every stage of development.
The testing needed for our teams can range from features which require detailed test documentation due to technical or functional complexity to other testing which suits a more exploratory approach based on ideas, heuristics and testing experience. I felt that trying to document a detailed test strategy or process would be of little value because so much of our approach to testing depends on context.

Through conversations with the Testers and wider development teams and getting feedback from Assurity colleagues (Aaron Hodder in particular offered a lot of valuable feedback and advice) I refined the strategy to a set of principles to guide our open-minded and collaborative teams in Testing.

Once I had pared the strategy back to guiding principles, I was inspired by some Lynne Cazaly drawings I'd seen to try and visualise the strategy. Visualising the strategy would be communicating it in an engaging, easy-to-understand way that could be on the wall as reference in stand-ups and help guide the conversations that would determine the story-specific test approach.

I am however limited in my visualisation skills - I can sketch out ideas, but I certainly can't draw at a level which would get the engagement I was after. Luckily, Stuff has Jaume Durany, an Agile coach with a passion for visualisation and strong drawing skills! Discussing options for visuals and working with my rough sketches and wording for the strategy, Jaume produced the visualisation below - version 1 of the Stuff product tech test strategy:

A Test Strategy Guide

The visualisation is only the starting point for a discussion on testing. Supporting the visualisation is more detail on what the guiding principles mean to us.

Centre: Testing as a Team

“Testing as a Team” is the central concept to the Test Strategy. Testing happens throughout the development cycle and all team members are involved from collaborative test modelling to automation to exploratory testing. Testing is a collaborative process involving many skill sets

• Some tasks may require specialist testing skills
• Others may require technical skills better suited to Developers
• Others may require specialist business or domain knowledge
• and some could be covered by any team member

Stories shouldn't be held up because a Tester is not available to do the testing.

Top Left: Shifting Left

Testing from the start - we talk testing from design stage to improve quality up front and plan the most efficient use of automation and manual testing efforts

• Testers facilitate collaborative test modelling
• Test modelling is focused on clarifying desired outcomes, determining test coverage and working out how to best cover the testing tasks.
• Testing tasks may include exploratory testing, developing automated tests or peer review.

Top Right: Test Lean

Reduce waste by understanding what is being tested, where it’s being tested, and how.

• We improve flow by collaborating as a team on testing
• Our collaboration identifies and reduces waste and duplication in test effort We automate the tests that are valuable to automate, using the Testing Pyramid to guide where and how to automate them.
• Manual test effort does not duplicate automated test coverage. It focuses on exploratory testing and checking test conditions which cannot be easily automated Test documentation should be at a level to enable re-use (where applicable) by other team members

Bottom Right: Shift Right

Testing doesn't finish at release

• Validate ideas with controlled experiments in production
• Look at what analytics, feedback and monitoring tell you, adapt accordingly. We regularly review our tests in the light of production data to ensure we are focusing on the areas with the highest business impact and archiving those that are no longer relevant or valuable.

Bottom Left: Quality Assistance

Our Testers are specialists who facilitate, guide and support testing, enabling the team as a whole to be responsible for quality.

Has it worked?

The Test Strategy visualisation has been a partial success so far. It's been visible on a shared team wall where we used to have our standups, but with our product owners based in different cities from the development teams, we've moved to having meetings over Hangouts in meeting rooms instead. I am working with the agile coaches to make it visible in other ways, including an experiment to create a micro-learning video which will have Jaume drawing the visualisation and me narrating.

While I believe we've had some success with all the principles in the strategy, the real success has been our development teams buying into the idea of "Testing as a Team". From the Testers doing nearly all the Testing earlier in the year, we now have all team members actively involved in some way in testing.

We are much more willing now to ask our teams who can test a story, rather than just looking to the Testers. We are discussing the skills and experience required for testing stories as teams and identifying where our Testers specialist skills can best be applied. This includes both the Testers doing testing themselves and them supporting other team members doing testing.
While they are executing less testing, our Testers are more available to drive quality across the whole development process and are consulting and being consulted throughout development for their specialist skills and experience.

Most of our wider team members - from Developers to Designers, Agile Coaches to Development Leads - have tested stories over recent months. This testing has been in a variety of scenarios:

• Assisting with / doing more straightforward functional testing for a story
• Picking up technical verification aspects of testing for relevant stories
• Crowd-sourced mobile testing, getting a wide range of stakeholders and team members to help increase device, OS and browser coverage
• In small cross-functional teams ("mobs") focused on specific solutions, all team members are actively working in testing (and Testers are involved in development).

I will be going into experiments we've run with these, and more, scenarios in the next blog article.