You Don't Need a Test Strategy

The Problem (Speaking Plainly)

 

In agile testing, you use the whole team approach to maximize the likelihood that you’re building the right thing and that you’re building it right by first distributing the responsibility for quality - whatever that means to the product owner and team - to the entire team, customer included. Requirements collated, communicated and understood just in time and articulated as executable examples yields the best chance at building the right thing. Similarly, fine-grained and well-focused automated unit tests provide the idioms for describing and communicating the intended design and best of all, it’s articulated in source code in an ever-growing repository of “this is how it works.” Developers make changes with less fear because they know that the tests will guide them; they will learn quickly if they have broken anything because there are failed tests. Life is good, and after an appropriate sprint where the minimally viable product will yield discernible benefits to the customer, a suitable set of activities take place that nurture the system through final end-to-end testing and acceptance/sign-off so that deployment happens. The testing is thinner at this stage because the end users/customers have been engaged all along. Deployment is a rehearsed action, ideally automated as well, so it’s not that big of a deal. It’s live. 

 

And it’s not done.

 

We are used to thinking that being done means going live. Go-live, after all, represents a huge milestone, no doubt about it. It’s not done because it has not yet yielded the intended benefits. Although we have set it up for use, laid the framework for achieving those benefits, it hasn’t happened yet. Unused software just doesn’t magically realize those intended benefits - people need to start using the product, and that usage has to be monitored and measured to back up claims that it is achieving those benefits. This usually does happen, to a degree. It might be orchestrated, it might not be. It absolutely needs to be.

 

The Solution (Speaking Hopefully)

 

A “test strategy” is aimed at achieving a goal related to testing. Based on it’s name alone, it seems that we have accomplished our goal and fully implemented the strategy when we have completed the testing. I want to toy with that word “strategy” for a sec ...

 

Sales strategy - maximize the likelihood of more sales - i.e., setting prices to maximize sales/profitability 

Marketing strategy - maximize the likelihood of more sales - i.e., bringing more eyes to what is being offered

Business strategy - maximize the likelihood of achieving business goals, whatever those might be

Test strategy - ???

 

What purpose does a test strategy fulfill? To maximize the effectiveness of testing? To squeeze as much testing as possible into a fixed budget/time window? To make sure that the product is tested? We test because we want answers to questions about the solution, we want to know if there are bugs, we want to know that we’ve built the right thing, etc. We want information. Testing is “the headlights of the project” [Kaner et al] after all. So it’s tempting to say that the test strategy is all about testing - how we will know when we’re done testing, get everyone on the same page as to how the solution will be tested, etc. 

 

I’m suggesting that this is not enough. It exposes us to the risk that we’ve fulfilled the test strategy and yet somehow, the solution is not considered a success. We want to make a call on the solution’s adoptability - will the customer/user/business be able to adopt the solution and make it part of their day? If every user type has a micro-goal (place an order, generate an invoice, etc.) using the solution, can they accomplish that goal more easily than they could without the solution?

 

I’m proposing that an effective test strategy maximizes the likelihood of solution adoption and benefits realization, in so far as suggesting that we rename it the “adoption strategy” instead. It’s more clearly what we need.

 

Tested is a milestone but isn’t the end state that you are looking for.

Deployed is a milestone but isn’t the end state that you are looking for.

Used. Adopted. That’s the end state that we are looking for.

 

What I’ve seen in most businesses where I’ve worked is that the “test strategy” really only includes the first item - getting to this ambiguous state called “solution tested.”

 

If we were to include ALL THREE elements above - tested, deployed, and used/adopted, then our test strategy sways towards including elements of a marketing strategy - the tested solution being what it is, how can we bring people around to its benefits and start using it? What other non-software items should we take care of in order to maximize the likelihood of a smooth adoption? We need a rapid, repeatable deployment process so that we can adapt quickly to feedback. We need automated tests to support checking for regressions. We need to monitor usage in production and measure adoption and accrued benefits. This is part of the overall “headlights” vision of testing albeit extended into production, a place where at least in some environments and for some testing people, testing is not allowed. Marketing people don’t hesitate to use A/B testing on an e-commerce site. I’m suggesting that an acceptance/adoption strategy would include A/B testing whether or not there is a marketing element in the target business domain. I’m suggesting that adoption statistics be automated/implemented alongside the target business functionality. It’s what distinguishes an adoption strategy from a traditional test strategy. It may also include, depending on the context, testing the effectiveness of the training, testing the job aids that are rolled out along with the solution, identifying meaningful business process measures, etc.

 

For example, In my most recent project we used a testing framework to monitor financial transactions as they were fed into the financial systems. Since it logged every assertion as it ran, the resulting logs were useful as a means of measuring overall error rates. Clearly errors in finances is something that would be a barrier to acceptance and adoption. In acceptance testing, it was relatively easy to establish an error-free set of transactions. But that would only be a sampling of the combinations and permutations that could happen in production. Given it was automated, we promoted it to production and then we had the means to provide actual error rates to stakeholders, making it easier for them to accept that the solution wasn’t actually a random number generator in disguise.

 

That’s a small example of the kind of information that is helpful to adoption/benefits realization that people with testing skills can provide. It’s really just another feedback loop that we could be exploiting to improve the product or service under development.

The Calculus of Acceptance Testing

It’s tempting to believe that acceptance testing is the straight-forward process of comparing the as-built solution to the as-required solution. Any differences discovered during testing are highlighted in defect reports and ultimately resolved through a defect resolution process. You’re done when you’ve had the business test everything and when enough of the differences have gone away for the business to believe that they could use the solution as part of their regular business activities.

 

It’s tempting to believe this statement because it’s simple and it seems reasonable. Looking closely, however, it’s an over-simplification that relies on three assumptions that are difficult and complex to uphold. 

 

First, the assumption that there is agreement on what the “as-required solution” really is. The various people involved may not share the same mental model of the business problem, the solution to that problem, or the implementation of that solution. There may be a document that represents one point of view, or a even a shared point of view, but even then it only represents a point in time - when the document was published. In a large enough management context where multiple workgroups interact with the solution on a daily basis, the needs and wants of the various parties may be in conflict. In addition, business and technical people will come and go on and off the project. This too leaves gaps in the collective understanding.

 

Second, the assumption that the business testers can test everything. Defining ‘everything’ is an art in itself. There are story maps, use case models, system integration diagrams, or even the Unified Modeling Language to help define ‘everything’. Comprehending ‘everything’ is a huge undertaking, even if it’s done wisely so that the essential models are identified organically and then grown, not drawn. Finding “just enough” testing to complete to get to the point of acceptance - by that standard - is a black art. It’s a calculus of people’s perspectives, beliefs and biases - because it’s people that accept the functionality - and of technical perspectives, beliefs, and biases - because there are technical elements of testing. Even acceptance testing.

 

Third, the assumption that the target of the test is the software. In reality the target of an acceptance testing is the intersection of software (multiple layers and/or participating components), business process (new, changed, and unchanged), and people’s adoption skills. To make this even more complex, consider that often the business testers are not the only users; sometimes they represent their workgroup with a wide range of technology adoption skills. So they’re not even testing at this critical section with solely their own needs in mind - they have to consider what other solution adopters might experience using the software.

 

For these and other reasons that I will explore in this blog, acceptance isn’t an event as much as a process, and acceptance testing isn’t about software quality as much as it is about solution adoptability. Of course those two things are related because you can’t address adoptability without quality. The core of the calculation is a gambit - spend the time assessing quality and hope for the best on adoptability, or spend the time exploring that intersection mentioned above - software, business process, and the likelihood of adoption.

 

That puts a different spin on the term “acceptance testing”. Instead of evaluating software against requirements, what we do in the last moments before go-live is test acceptance. Acceptance testing.

Test Envisioning

Starting a project seems to be one of the most under-rated steps when it comes to describing critical success factors for projects. A project well-launched is like a well-designed set play that comes together on the field during a game. It's not launch effort but launch effectiveness that is important. In Agile Testing and Quality Strategies: Discipline Over Rhetoric Scott Ambler specifically describes requirements envisioning and architecture envisioning as key elements of initiating a project (he specifies this for launching an agile project, and I've come to believe now that it's true for any project with stakeholders).

In a classic standing on the shoulders of giants manner, "yes, and" ... we could also spend some of that project initiation time envisioning what the testing is going to look like. It is Iteration 0, after all so there is a large landscape of possible conversations that we could have. And increasingly I've seen misconceptions about how the project testing will be conducted as one of those "wait a second" moments. People just weren't on the same page, and they needed to be.

It's a fundamental part of the working agreement that Iteration 0 is intended to create - who will test what, when? Will the developers use test-first programming? Will they also use higher level test automation such as one of the BDD tools? Will they do build verification testing beyond the automated tests? If the solution provider is a remote, out-sourced team, what testing will the solution acquirer organization do? When will they do it? How will they do it? Will they automate some tests too? Is there formal ITIL release management going on at the acquiring organization that will demand a test-last acceptance test phase? Will the contract require such testing?

You see my point. There are a lot of alternate pathways, a lot of questions, and it's an important conversation. Even if some of the answers are unknown and require some experimentation, at least everyone should agree on the questions. Context matters.

I come back to the point about test envisioning. The result of such envisioning is a working agreement on how testing might be conducted. That working agreement might well be called a test strategy, albeit a lean one. That's why I promote it as a poster first, document second (and only if it helps to resolve some communication need). What you put on that poster is driven by what conversations need to take place and what uncertainties exist.

To build on Scott's description of Iteration 0 then, the work breakdown structure for Envisioning may include the following:

Iteration 0: Envisioning

• Requirements Envisioning
• Architecture Envisioning
• Test Envisioning

and the result might very well be three big visible charts - one for each. Talking about testing upfront lets everyone listen, understand and contribute to the mental model melding that must take place for the team to be effective.

UAT as a Gateway to Enterprise Agile

Let's admit it. Projects that deliver enterprise solutions have barriers to being agile, especially those that are based on purchased packages (commercial off-the-shelf packages, or COTS). In my experience I've wrestled with at least two of those - the technical barrier of test automation capability and the organizational barrier of having a single Product Owner that truly speaks for the whole of the enterprise.

In this post, 'user acceptance testing' represents that last-gasp critical-path test activity that enterprise projects typically run just before going live. It is generally an entry condition for migrating the solution into the production environments. The theory is generally that the 'business unit' needs to understand and approve of what they will be using post go-live and that this is the last time to check that the combined new/revised/existing business processes are supported by the new solution, and vice-versa as the case may be.

The 'test last' bit and acceptance not being done during the delivery sprint(s) doesn't feel 'agile'. So then how could user acceptance testing be the gateway to agile for enterprise solution teams besides the general observation that a UAT window in an enterprise project is usually 2-4 weeks, that is, the same duration you might expect of a typical scrum project.

Gateway Characteristic #1: Whole-team Approach.

Briefly, a UAT phase run in a critical-path manner is an 'all hands on deck' phase. The project or test manager usually rallies the team around the UAT cause in some sort of a launch meeting, where the people that are doing the testing meet the people that are supporting them doing the testing and everyone in between. Everyone participates in this phase - trainers, the business units that will receive/use the new solution, IT personnel that will support the solution post go-live, etc.

A wise test manager will in fact plan UAT this way as well by hosting workshops with the business to identify and otherwise order the business process scenarios that would be in-scope of the UAT. To avoid creating test scripts that repeat the content of either training material or business process descriptions, the intent of these workshops is to identify the testing tasks/activities that need to be done and in what order they should be done in for maximum effect.

Does that sound a bit like sprint planning to you?

The whole team also comes into play in the problem/issue workflow, where there is generally a daily triage of discovered problems that need to be described, classified for severity, and assigned to a resolution provider. Generally everyone participates in these triage sessions for cross-communications purposes. The problems are not to be resolved in this meeting, just raised, classified and then picked up by someone that can do something about it.

Could this meeting be extended by 15 minutes so that everyone also mentions what they tested yesterday and what they will test today, that is, in addition to what problems/barriers/issues they are running into?

Last point in this section - most of the companies that I've managed UAT on behalf of establish a "UAT testing room" that is a co-opted training facility or meeting room so that all the business testers can be co-located when they are actively testing. Hmmm. Sounds like what we generally arrange for an agile team.

Gateway Characteristic #2: Intensive Business Involvement and Leadership.

It is common to warrant that business involvement and leadership exists by getting the UAT plan signed-off before the phase and a report of the results signed-off afterwards. That might be necessary, but it's not sufficient for any UAT. It is critical that there be business people actively testing - in fact they are the only ones that can perform the testing and acceptance according to the generally-accepted governance umbrella that the UAT operates under. Since there are always problems detected in UAT, the business leadership needs to pay attention and get involved when their people might get discouraged because of what they find or believe they are finding, as the case may be.

It's a nervous time for the business people involved in testing and for their leadership to dump all the responsibility and accountability onto them for making the acceptance decision is counter-productive and harmful. First, these aren't professional testers - they are people from the business units seconded to do this testing and second - they might not even like testing and third - they might still have to do their regular jobs during the testing. The right mix of leadership from the business unit and from the project team implementing the solution is required to support these individuals.

The common characteristic is that both agile projects and UAT require that business leadership touch to be successful.

Gateway Characteristic #3: There is a Backlog.

Most of the time - but I suppose not all the time - there is a list of problems/defects that were discovered in earlier test phases that will be resolved at some point during UAT. Those problems are not severe enough to have prevented UAT from starting, but at the same time, they do still need to be resolved. In addition, the UAT test plan lists business scenarios that need to be run over the course of UAT. These two lists are combined and generally otherwise ordered so that the time spent in UAT is efficient and effective.

This is a backlog and I've seen many teams successfully use burn-down/burn-up charts to communicate their test progress. Information radiators such as dashboards are commonly posted to either project room walls, project repositories, or both. Discovered problems/issues are incorporated into the backlog using a process similar to the way that an agile team deals with problems/issues discovered within a sprint.

Gateway Characteristic #4: The Backlog is Ordered.

When a problem/issue is discovered and goes through triage, it's been ordered along with everything else that has to be done within the planned UAT window. The agile characteristic of this is that everyone has participated in the prioritization - business and technical personnel together make these decisions.

Gateway Characteristic #5: Business Acceptance of Detected Problems Happens Within the Phase.

Detected problems go through their entire life cycle within the UAT window. They are identified, triaged, fixed, re-tested, resolved and closed before the phase ends. If not, then  the problem is serious enough to either extend UAT or to stop it outright and halt the go-live. Consequently, there is enormous effort put in to make sure that there are adequate resources on call to get any problem resolved - training materials, software, business process, environment - any root cause whatsoever will generally get all the attention it needs to be taken care of.

As a gateway characteristic, this is exactly what the business people involved in an agile sprint are asked to do - identify what they need, compare what they are given to what they need, back-and-forth, reach consensus on what the final product is, accept it and move on to the next item. Business units therefore DO have experience with this kind of acceptance regimen if they've been through a UAT.

Gateway characteristic #6: The Result is a Potentially-shippable Version of the Solution.

At the end of both a sprint and a UAT phase, there is a potentially-shippable version of the solution.

Concluding Thought Experiment

Generally a COTS package is selected after some period of analysis and review - a gap analysis per se - and then the implementation is planned based on closing those gaps. What if the needs and gaps were planned to be closed in a series of say 10 UAT-like phases? The two highest barriers remain the test automation problem - since many of these products do not automate well - and the difficulty filling the Product Owner role in large projects.

 

One Way

There is no one way to do things. One of my favourite quotes about technology that highlights this is from Robert M. Pirsig:

Technology presumes there's just one right way to do things and there never is.

In my current management context (I phrase I’ve adopted via Cem Kaner) I’m the quality control guy for an enterprise program that consists of five projects. Sub-projects, really. Each one of them a commercial-off-the-shelf (COTS) implementation. Each one of them a critical component to long-term technology adoptions plans by the acquiring organization. Each one of them completing the complexity trifecta: people, process, technology.

Checklists to the rescue. Like this one for the sub-project test plans, it’s effectively a functional decomposition of the test activities on a project. The intent is that each sub-project knows what they are doing in each of the following major categories and that each of them also has evidence backing up their claim that they know what they are doing.

• Test Environment Management
• Test Data Management
• Test Management
• Acceptance Management

Remember my context. Enterprise. Commercial-off-the-shelf implementation. We’re not testing the product but the installation/configuration of the product. We’re investigating potential barriers to solution adoption and to realizing the benefits that were planned.

Diving deeper…

Test Environment Management

BTW the colours are a risk assessment. In the enterprise, shared test environments are more complex to deal with than those you control exclusively. The closer you are to acceptance testing, the more likely you will have a shared environment to work in. Systems are inter-connected nowadays, eh. If the system itself is net new, it’s going to connect to something that isn’t new. And so, there be capability gaps that have to be closed, tested in isolation, then tested as part of regular business operations (end-to-end) testing.

Test Data Management

Again not as big a deal if you control the environment exclusively. If you don’t, and chances are that you don’t if you’re working with an enterprise system, then you have to devise how you are going to work with test data.

Test Management

Many people think of the role of a test manager and start/stop at test management. And that’s one of my major motivations behind building this test plan checklist.

I differentiate “design testing” from “design tests”. Word play, perhaps, but in my mind creating a test strategy and formulating a plan that implements that strategy is designing a system to complete the testing. That’s “designing testing”. Designing tests, on the other hand, is the activity of identifying/crafting and executing tests.

Really important: None of these checklist items demand test scripts/procedures. How much (or how little) you need scripts is part of planning and designing testing.

Acceptance Management

This may not be an activity for the test manager, but in most enterprises there is a concept of getting “sign off”. As I’ve mentioned before on this blog (at least I think I did) there are significant advantages to designing the sign-off process with the management context at hand in mind.

Test results play a role in requesting/achieving sign-off, and that’s why it’s here – even if the project managers choose to assume that role for themselves – as a test manager, you’re involved.

The Checklist

My test plan template for the sub-projects consisted of a cover page (enterprise art provided by the enterprise of course), the list of processes in full mind map glory, and a set of test planning principles to uphold (such as “plan to exercise the risky things first”).

The only other twist is that each leaf node on the mind map has a spot for a “if not now, then when” data point; just in case there were things yet to be discovered.

It’s a thinking tool, I think. And hope.

A Mandated Tool

In my current context I’m required to use a certain tool for managing the test effort; you probably know the tool. I won’t mention it by name unless the vendor pays me to, but it’s the first on the list when non-testers choose to purchase a test tool.

My approach is to use this tool as a checklist manager and it works really well for that purpose IF you don’t over-engineer your use of it.

Forget Requirements – List Test Targets

This means avoiding the requirements management trap, that is, using the requirements section for anything but helping you to organize the test effort. A test target is something that needs testing. A requirement, user story, feature, benefit, risk, performance criterion, security criterion, etc.

Test targets are in checklist format and organized in a way that makes it easy to do the next step, create the playbook.

In my current project, I have device specifications (temperature operating range, voltage operating range, etc.) that are clear and unambiguous test targets; I’m lucky. We will employ the same concept when we need to test business processes though.

Forget Test Plans – Create a Playbook

Think of this section as designing/creating a playbook. When we get a new release – we do this. When we test this feature right here, we do that. You should organize the plays in the playbook according to the conditions that exist before you run the play.

Back to my current project – when we receive a device firmware update or upgrade – the set of tests (the play) that we should run to accept that new configuration should be easy to find.

The Test Lab is for Test Sessions

Think of this section as recording the results of test sessions. For any given release or time period, pull plays from the playbook, run the play, discover what more you can/should do, update the play book, run the new/revised plays, rinse, repeat, wipe hands on pants.

At the end of the session, upload your session record so that you have evidence of having tested. Revisions of the playbook are evidence of having learned something about the product and continual improvement.

Current project example again – the February 12 release firmware tests were run over a period of a few days by two engineers and their session record (a Word document) was uploaded into this section in the test management tool. We will create a new folder for the March 29 release and go back to the source – the playbook – to define the firmware test session that will be held at that time. We will not re-run the checklist in the February 12 release section.

Principles

DRY – Don’t repeat yourself. This works in code, and it works in checklists. Don’t organize your test targets by release since that limits your checklist’s value when used three years from now to test a new release or upgrade. If your list of test targets and your list of set plays look the same, then you really haven’t done any design work and you’re not helping your future self as much as you could be.

Ignore the temptation to put a release/time/date anywhere but as metadata for test sessions. Nothing else needs a date since they should survive every release of the solution until they are decommissioned. Test targets and the plays in the playbook are for all time. Well, at least as long as the solution itself survives. I would organize the test sessions

Forget you’re on a project. You’re creating the list of test targets and a playbook for all releases, for the duration of the solution’s useful life. Without this goal – why are you using this tool again?

Requirements Management and Test Tools

When modeling requirements in English, one of the heuristics is to avoid ‘weak verbs’ such as ‘to process’, ‘to handle’ and ‘to manage’. The weak verb is subject to broader interpretation than other, more constrained (strong) verbs and in writing software, broad interpretations are more likely to yield building the wrong thing.

Some test management tools help us ‘manage requirements’. What exactly does that mean? It’s ironic that we use a weak verb in describing a tool used for manipulating (ideally) pairs of strong nouns and strong verbs.

Increasingly, I find that I’m evaluating test management tools based on how they help me work and collaborate around those requirements and how effectively they help me use the requirements as test targets.

• Create/edit/share checklists of test targets (including requirements).
• Version-control and label checklists so I can associate them with other version-controlled artifacts like source code. I don’t need per-item version control, just  version control on the checklist.
• Layer profiles onto the checklist to highlight themes: the high-risk theme unites high-risk items, the new-construction theme unites items that are new, … and I get to invent the layers/profiles that suit my context.
• Create/edit/share exploratory testing session charters based on a selection of items from a checklist
• Create/edit/share exploratory testing session records
• Compile checklists/session records into information radiators / dashboards for myself, for team members, and for external stakeholders.
• Create/edit/share a script based on a selection of items from a checklist (could be one, could be many); creating a script for a given checklist item shouldn’t be the default.

That’s what “manage requirements” means to me, a test manager.