The Calculus of Acceptance Testing

It’s tempting to believe that acceptance testing is the straight-forward process of comparing the as-built solution to the as-required solution. Any differences discovered during testing are highlighted in defect reports and ultimately resolved through a defect resolution process. You’re done when you’ve had the business test everything and when enough of the differences have gone away for the business to believe that they could use the solution as part of their regular business activities.

 

It’s tempting to believe this statement because it’s simple and it seems reasonable. Looking closely, however, it’s an over-simplification that relies on three assumptions that are difficult and complex to uphold. 

 

First, the assumption that there is agreement on what the “as-required solution” really is. The various people involved may not share the same mental model of the business problem, the solution to that problem, or the implementation of that solution. There may be a document that represents one point of view, or a even a shared point of view, but even then it only represents a point in time - when the document was published. In a large enough management context where multiple workgroups interact with the solution on a daily basis, the needs and wants of the various parties may be in conflict. In addition, business and technical people will come and go on and off the project. This too leaves gaps in the collective understanding.

 

Second, the assumption that the business testers can test everything. Defining ‘everything’ is an art in itself. There are story maps, use case models, system integration diagrams, or even the Unified Modeling Language to help define ‘everything’. Comprehending ‘everything’ is a huge undertaking, even if it’s done wisely so that the essential models are identified organically and then grown, not drawn. Finding “just enough” testing to complete to get to the point of acceptance - by that standard - is a black art. It’s a calculus of people’s perspectives, beliefs and biases - because it’s people that accept the functionality - and of technical perspectives, beliefs, and biases - because there are technical elements of testing. Even acceptance testing.

 

Third, the assumption that the target of the test is the software. In reality the target of an acceptance testing is the intersection of software (multiple layers and/or participating components), business process (new, changed, and unchanged), and people’s adoption skills. To make this even more complex, consider that often the business testers are not the only users; sometimes they represent their workgroup with a wide range of technology adoption skills. So they’re not even testing at this critical section with solely their own needs in mind - they have to consider what other solution adopters might experience using the software.

 

For these and other reasons that I will explore in this blog, acceptance isn’t an event as much as a process, and acceptance testing isn’t about software quality as much as it is about solution adoptability. Of course those two things are related because you can’t address adoptability without quality. The core of the calculation is a gambit - spend the time assessing quality and hope for the best on adoptability, or spend the time exploring that intersection mentioned above - software, business process, and the likelihood of adoption.

 

That puts a different spin on the term “acceptance testing”. Instead of evaluating software against requirements, what we do in the last moments before go-live is test acceptance. Acceptance testing.

Test Envisioning

Starting a project seems to be one of the most under-rated steps when it comes to describing critical success factors for projects. A project well-launched is like a well-designed set play that comes together on the field during a game. It's not launch effort but launch effectiveness that is important. In Agile Testing and Quality Strategies: Discipline Over Rhetoric Scott Ambler specifically describes requirements envisioning and architecture envisioning as key elements of initiating a project (he specifies this for launching an agile project, and I've come to believe now that it's true for any project with stakeholders).

In a classic standing on the shoulders of giants manner, "yes, and" ... we could also spend some of that project initiation time envisioning what the testing is going to look like. It is Iteration 0, after all so there is a large landscape of possible conversations that we could have. And increasingly I've seen misconceptions about how the project testing will be conducted as one of those "wait a second" moments. People just weren't on the same page, and they needed to be.

It's a fundamental part of the working agreement that Iteration 0 is intended to create - who will test what, when? Will the developers use test-first programming? Will they also use higher level test automation such as one of the BDD tools? Will they do build verification testing beyond the automated tests? If the solution provider is a remote, out-sourced team, what testing will the solution acquirer organization do? When will they do it? How will they do it? Will they automate some tests too? Is there formal ITIL release management going on at the acquiring organization that will demand a test-last acceptance test phase? Will the contract require such testing?

You see my point. There are a lot of alternate pathways, a lot of questions, and it's an important conversation. Even if some of the answers are unknown and require some experimentation, at least everyone should agree on the questions. Context matters.

I come back to the point about test envisioning. The result of such envisioning is a working agreement on how testing might be conducted. That working agreement might well be called a test strategy, albeit a lean one. That's why I promote it as a poster first, document second (and only if it helps to resolve some communication need). What you put on that poster is driven by what conversations need to take place and what uncertainties exist.

To build on Scott's description of Iteration 0 then, the work breakdown structure for Envisioning may include the following:

Iteration 0: Envisioning

• Requirements Envisioning
• Architecture Envisioning
• Test Envisioning

and the result might very well be three big visible charts - one for each. Talking about testing upfront lets everyone listen, understand and contribute to the mental model melding that must take place for the team to be effective.

UAT as a Gateway to Enterprise Agile

Let's admit it. Projects that deliver enterprise solutions have barriers to being agile, especially those that are based on purchased packages (commercial off-the-shelf packages, or COTS). In my experience I've wrestled with at least two of those - the technical barrier of test automation capability and the organizational barrier of having a single Product Owner that truly speaks for the whole of the enterprise.

In this post, 'user acceptance testing' represents that last-gasp critical-path test activity that enterprise projects typically run just before going live. It is generally an entry condition for migrating the solution into the production environments. The theory is generally that the 'business unit' needs to understand and approve of what they will be using post go-live and that this is the last time to check that the combined new/revised/existing business processes are supported by the new solution, and vice-versa as the case may be.

The 'test last' bit and acceptance not being done during the delivery sprint(s) doesn't feel 'agile'. So then how could user acceptance testing be the gateway to agile for enterprise solution teams besides the general observation that a UAT window in an enterprise project is usually 2-4 weeks, that is, the same duration you might expect of a typical scrum project.

Gateway Characteristic #1: Whole-team Approach.

Briefly, a UAT phase run in a critical-path manner is an 'all hands on deck' phase. The project or test manager usually rallies the team around the UAT cause in some sort of a launch meeting, where the people that are doing the testing meet the people that are supporting them doing the testing and everyone in between. Everyone participates in this phase - trainers, the business units that will receive/use the new solution, IT personnel that will support the solution post go-live, etc.

A wise test manager will in fact plan UAT this way as well by hosting workshops with the business to identify and otherwise order the business process scenarios that would be in-scope of the UAT. To avoid creating test scripts that repeat the content of either training material or business process descriptions, the intent of these workshops is to identify the testing tasks/activities that need to be done and in what order they should be done in for maximum effect.

Does that sound a bit like sprint planning to you?

The whole team also comes into play in the problem/issue workflow, where there is generally a daily triage of discovered problems that need to be described, classified for severity, and assigned to a resolution provider. Generally everyone participates in these triage sessions for cross-communications purposes. The problems are not to be resolved in this meeting, just raised, classified and then picked up by someone that can do something about it.

Could this meeting be extended by 15 minutes so that everyone also mentions what they tested yesterday and what they will test today, that is, in addition to what problems/barriers/issues they are running into?

Last point in this section - most of the companies that I've managed UAT on behalf of establish a "UAT testing room" that is a co-opted training facility or meeting room so that all the business testers can be co-located when they are actively testing. Hmmm. Sounds like what we generally arrange for an agile team.

Gateway Characteristic #2: Intensive Business Involvement and Leadership.

It is common to warrant that business involvement and leadership exists by getting the UAT plan signed-off before the phase and a report of the results signed-off afterwards. That might be necessary, but it's not sufficient for any UAT. It is critical that there be business people actively testing - in fact they are the only ones that can perform the testing and acceptance according to the generally-accepted governance umbrella that the UAT operates under. Since there are always problems detected in UAT, the business leadership needs to pay attention and get involved when their people might get discouraged because of what they find or believe they are finding, as the case may be.

It's a nervous time for the business people involved in testing and for their leadership to dump all the responsibility and accountability onto them for making the acceptance decision is counter-productive and harmful. First, these aren't professional testers - they are people from the business units seconded to do this testing and second - they might not even like testing and third - they might still have to do their regular jobs during the testing. The right mix of leadership from the business unit and from the project team implementing the solution is required to support these individuals.

The common characteristic is that both agile projects and UAT require that business leadership touch to be successful.

Gateway Characteristic #3: There is a Backlog.

Most of the time - but I suppose not all the time - there is a list of problems/defects that were discovered in earlier test phases that will be resolved at some point during UAT. Those problems are not severe enough to have prevented UAT from starting, but at the same time, they do still need to be resolved. In addition, the UAT test plan lists business scenarios that need to be run over the course of UAT. These two lists are combined and generally otherwise ordered so that the time spent in UAT is efficient and effective.

This is a backlog and I've seen many teams successfully use burn-down/burn-up charts to communicate their test progress. Information radiators such as dashboards are commonly posted to either project room walls, project repositories, or both. Discovered problems/issues are incorporated into the backlog using a process similar to the way that an agile team deals with problems/issues discovered within a sprint.

Gateway Characteristic #4: The Backlog is Ordered.

When a problem/issue is discovered and goes through triage, it's been ordered along with everything else that has to be done within the planned UAT window. The agile characteristic of this is that everyone has participated in the prioritization - business and technical personnel together make these decisions.

Gateway Characteristic #5: Business Acceptance of Detected Problems Happens Within the Phase.

Detected problems go through their entire life cycle within the UAT window. They are identified, triaged, fixed, re-tested, resolved and closed before the phase ends. If not, then  the problem is serious enough to either extend UAT or to stop it outright and halt the go-live. Consequently, there is enormous effort put in to make sure that there are adequate resources on call to get any problem resolved - training materials, software, business process, environment - any root cause whatsoever will generally get all the attention it needs to be taken care of.

As a gateway characteristic, this is exactly what the business people involved in an agile sprint are asked to do - identify what they need, compare what they are given to what they need, back-and-forth, reach consensus on what the final product is, accept it and move on to the next item. Business units therefore DO have experience with this kind of acceptance regimen if they've been through a UAT.

Gateway characteristic #6: The Result is a Potentially-shippable Version of the Solution.

At the end of both a sprint and a UAT phase, there is a potentially-shippable version of the solution.

Concluding Thought Experiment

Generally a COTS package is selected after some period of analysis and review - a gap analysis per se - and then the implementation is planned based on closing those gaps. What if the needs and gaps were planned to be closed in a series of say 10 UAT-like phases? The two highest barriers remain the test automation problem - since many of these products do not automate well - and the difficulty filling the Product Owner role in large projects.

 

One Way

There is no one way to do things. One of my favourite quotes about technology that highlights this is from Robert M. Pirsig:

Technology presumes there's just one right way to do things and there never is.

In my current management context (I phrase I’ve adopted via Cem Kaner) I’m the quality control guy for an enterprise program that consists of five projects. Sub-projects, really. Each one of them a commercial-off-the-shelf (COTS) implementation. Each one of them a critical component to long-term technology adoptions plans by the acquiring organization. Each one of them completing the complexity trifecta: people, process, technology.

Checklists to the rescue. Like this one for the sub-project test plans, it’s effectively a functional decomposition of the test activities on a project. The intent is that each sub-project knows what they are doing in each of the following major categories and that each of them also has evidence backing up their claim that they know what they are doing.

• Test Environment Management
• Test Data Management
• Test Management
• Acceptance Management

Remember my context. Enterprise. Commercial-off-the-shelf implementation. We’re not testing the product but the installation/configuration of the product. We’re investigating potential barriers to solution adoption and to realizing the benefits that were planned.

Diving deeper…

Test Environment Management

BTW the colours are a risk assessment. In the enterprise, shared test environments are more complex to deal with than those you control exclusively. The closer you are to acceptance testing, the more likely you will have a shared environment to work in. Systems are inter-connected nowadays, eh. If the system itself is net new, it’s going to connect to something that isn’t new. And so, there be capability gaps that have to be closed, tested in isolation, then tested as part of regular business operations (end-to-end) testing.

Test Data Management

Again not as big a deal if you control the environment exclusively. If you don’t, and chances are that you don’t if you’re working with an enterprise system, then you have to devise how you are going to work with test data.

Test Management

Many people think of the role of a test manager and start/stop at test management. And that’s one of my major motivations behind building this test plan checklist.

I differentiate “design testing” from “design tests”. Word play, perhaps, but in my mind creating a test strategy and formulating a plan that implements that strategy is designing a system to complete the testing. That’s “designing testing”. Designing tests, on the other hand, is the activity of identifying/crafting and executing tests.

Really important: None of these checklist items demand test scripts/procedures. How much (or how little) you need scripts is part of planning and designing testing.

Acceptance Management

This may not be an activity for the test manager, but in most enterprises there is a concept of getting “sign off”. As I’ve mentioned before on this blog (at least I think I did) there are significant advantages to designing the sign-off process with the management context at hand in mind.

Test results play a role in requesting/achieving sign-off, and that’s why it’s here – even if the project managers choose to assume that role for themselves – as a test manager, you’re involved.

The Checklist

My test plan template for the sub-projects consisted of a cover page (enterprise art provided by the enterprise of course), the list of processes in full mind map glory, and a set of test planning principles to uphold (such as “plan to exercise the risky things first”).

The only other twist is that each leaf node on the mind map has a spot for a “if not now, then when” data point; just in case there were things yet to be discovered.

It’s a thinking tool, I think. And hope.

A Mandated Tool

In my current context I’m required to use a certain tool for managing the test effort; you probably know the tool. I won’t mention it by name unless the vendor pays me to, but it’s the first on the list when non-testers choose to purchase a test tool.

My approach is to use this tool as a checklist manager and it works really well for that purpose IF you don’t over-engineer your use of it.

Forget Requirements – List Test Targets

This means avoiding the requirements management trap, that is, using the requirements section for anything but helping you to organize the test effort. A test target is something that needs testing. A requirement, user story, feature, benefit, risk, performance criterion, security criterion, etc.

Test targets are in checklist format and organized in a way that makes it easy to do the next step, create the playbook.

In my current project, I have device specifications (temperature operating range, voltage operating range, etc.) that are clear and unambiguous test targets; I’m lucky. We will employ the same concept when we need to test business processes though.

Forget Test Plans – Create a Playbook

Think of this section as designing/creating a playbook. When we get a new release – we do this. When we test this feature right here, we do that. You should organize the plays in the playbook according to the conditions that exist before you run the play.

Back to my current project – when we receive a device firmware update or upgrade – the set of tests (the play) that we should run to accept that new configuration should be easy to find.

The Test Lab is for Test Sessions

Think of this section as recording the results of test sessions. For any given release or time period, pull plays from the playbook, run the play, discover what more you can/should do, update the play book, run the new/revised plays, rinse, repeat, wipe hands on pants.

At the end of the session, upload your session record so that you have evidence of having tested. Revisions of the playbook are evidence of having learned something about the product and continual improvement.

Current project example again – the February 12 release firmware tests were run over a period of a few days by two engineers and their session record (a Word document) was uploaded into this section in the test management tool. We will create a new folder for the March 29 release and go back to the source – the playbook – to define the firmware test session that will be held at that time. We will not re-run the checklist in the February 12 release section.

Principles

DRY – Don’t repeat yourself. This works in code, and it works in checklists. Don’t organize your test targets by release since that limits your checklist’s value when used three years from now to test a new release or upgrade. If your list of test targets and your list of set plays look the same, then you really haven’t done any design work and you’re not helping your future self as much as you could be.

Ignore the temptation to put a release/time/date anywhere but as metadata for test sessions. Nothing else needs a date since they should survive every release of the solution until they are decommissioned. Test targets and the plays in the playbook are for all time. Well, at least as long as the solution itself survives. I would organize the test sessions

Forget you’re on a project. You’re creating the list of test targets and a playbook for all releases, for the duration of the solution’s useful life. Without this goal – why are you using this tool again?

Requirements Management and Test Tools

When modeling requirements in English, one of the heuristics is to avoid ‘weak verbs’ such as ‘to process’, ‘to handle’ and ‘to manage’. The weak verb is subject to broader interpretation than other, more constrained (strong) verbs and in writing software, broad interpretations are more likely to yield building the wrong thing.

Some test management tools help us ‘manage requirements’. What exactly does that mean? It’s ironic that we use a weak verb in describing a tool used for manipulating (ideally) pairs of strong nouns and strong verbs.

Increasingly, I find that I’m evaluating test management tools based on how they help me work and collaborate around those requirements and how effectively they help me use the requirements as test targets.

• Create/edit/share checklists of test targets (including requirements).
• Version-control and label checklists so I can associate them with other version-controlled artifacts like source code. I don’t need per-item version control, just  version control on the checklist.
• Layer profiles onto the checklist to highlight themes: the high-risk theme unites high-risk items, the new-construction theme unites items that are new, … and I get to invent the layers/profiles that suit my context.
• Create/edit/share exploratory testing session charters based on a selection of items from a checklist
• Create/edit/share exploratory testing session records
• Compile checklists/session records into information radiators / dashboards for myself, for team members, and for external stakeholders.
• Create/edit/share a script based on a selection of items from a checklist (could be one, could be many); creating a script for a given checklist item shouldn’t be the default.

That’s what “manage requirements” means to me, a test manager.

Personal Kanban for Exploratory Testing–Part 2

Part 1 | Part 2 | Part 3 | Part 4

The sensibility of done is something that testers struggle with. Testing is, after all, like a gas. It will fill whatever volume you give it. As testers we know that we can’t exhaustively test enterprise solutions. Yet our project managers and program managers and product owners and most harshly of all some of our test managers ask us to tell them with no measure of uncertainty that the product works. That we are done testing.

Not so fast.

Uncertainty that a solution ‘works’ and is usable/adoptable by the intended user community is a tester’s friend and foe.Friend, since it’s where we go when we want to understand what to test next. Foe, since we can’t really ever eliminate it even though people would like us to. Like limits in calculus approaching zero, we can test and test but ultimately we can’t get that uncertainty to zero. Not from anyone’s perspective.

So ‘done’ for a tester is a hard place to find.

Here’s a happy discovery about personal kanban: you as the performer only define ‘done’ when you yourself are the beneficiary of the work.

• Washing the car, you do a lap to check your work. Then declare done. You might be the only one that cares about the cleanliness of your car so you don’t have to ask anyone else if you’re really done or not.
• A personal development task like hiking to the top of a mountain for the first time has an explicit criterion, but only you can define the mountain that qualifies; it’s your personal development and someone else’s standards for what constitutes a mountain may or may not matter to you.

In those examples there is no flow of consideration between people in making that leap to done.

In testing, and in many other things, there is a flow of something in behind the scenes. My assertion is that when there is an underlying flow of consideration, you don’t define done on your own. You ask. You stop and talk. You listen. You breathe the same air your customer breathes, put yourself in their shoes, walk a mile. To quote a favourite person of mine, you “slide” into their world perspective and observe.

Good thing good testers come with advanced skills of observation.

In my exploratory testing and personal kanban mashup then, it’s not the testers that define done. We stop and talk about what we did. We complete a miniature retrospective on that exploratory testing session that was just completed with the people that we are testing for.

As I alluded to in Part 1, this was something that we failed to execute consistently in our project. As testers we talked to each other and to the respective area business leads but we didn’t account for the types of things that new users would encounter given the training that they were offered. Our testers were already experts by the time we reached the major testing milestones. As a result, we had issues post go-live that I bet we could have avoided if we had let in more new users into our test room. Lesson learned.

Who you talk to depends on what you are testing. If you’re testing a minimum beneficial feature (MBF) then you need to understand who benefits from that feature and talk to them before you move that MBF into the ‘Done’ column. In an enterprise project, the list of potential beneficiaries is:

• direct and indirect end-users for the relevant MBF
• the solution delivery personnel that configured/implemented that MBF
• the decision makers on the project team

We only had one place to stop and talk to the beneficiaries of a feature, and it was at the end in a buffer before the ‘Done’ column. I changed the name of our ‘Done’ column to ‘Communicated’. Thinking back now, that’s too late and our definition around ‘Ready to be Explored’ wasn’t strong enough; it should have included a criterion around understanding the context of the MBF before testing. I assumed we would be OK because we had business testers that were testing within the confines of their role but because of their ‘expert’ status, this was a shaky assumption.

This is the board that I maintained. I would change the ‘Ready to be Explored’ column description to have clearer entry conditions:

- somebody who would know says the MBF is ready to test
- as a tester, I understand the benefits and context of that feature

The last two columns are all about making sure that we’re communicating our test results – positive and negative – to decision makers. Testing is an information service, after all. If you keep the results to yourself for any length of time that’s – er – inventory. Don’t want that.

Here are some questions and concepts around ‘done’ for personal kanban exploratory testing mashups:

• Avoid letting panicking project managers in the ‘done’ conversation.
• There is always something more you could test.
• The risk of the MBF not working and the risk tolerance of those affected are your guides; be wary of posers. I’ve seen posers inflate and deflate risk tolerance levels to suit their own needs. Stop and talk to those truly affected if you can.
• In an enterprise solution delivery scenario as this one was, the list of beneficiaries is finite so you won’t have to dig up new people/groups to talk to for every MBF; you will get to know them and that’s a good thing. Still, don’t pretend to know them so well that you end up trying to represent them.

I felt like this was progress since it helped me understand and communicate where we were with our user acceptance testing effort and I believe that it ultimately helped the project.

Not satisfied though.

I’m anxious to try out new columns next time: “Live” and “In Active Use” so that usage data can be included in the feedback cycle. When the feature is adopted and you’re learning how that MBF is actually being used and adjusting that MBF based on that feedback in subsequent releases seems like a happy place to be.

In our case, we thought operating user acceptance testing like a “business simulation” would help and maybe it did – but as I mentioned we had problems post go-live because our testers were too good with the product/platform. The usage data we collected wasn’t representative of the true "go-live” usage data.

What I Done Told Ya

• For a tester, ‘done’ is a hard place to find.
• Use a self-declaration of done only when there isn’t a flow of consideration between you (the performer) and another person
• When there is a flow of consideration between you and someone else, find the community of people that are truly affected by ‘done’ and explore the definition of done with them.
• In our case, ‘Explored’ wasn’t done. Communicating the results was an important aspect of ‘Done’.
• I look forward to adding ‘Live’ and ‘In Active Use’ columns to the right-hand side of the board in future uses of kanban for exploratory testing.

What’s Next

In Part 3 I’ll talk about the work-in-process (WIP) limits and the experiments I used to come up with them. Like any system, it takes time to come to an equilibrium and there is typically so much going on in large enterprise solution implementations that thinking about WIP limits can be easily crowded out or clouded.

Thanks for getting to here.

Personal Kanban for Exploratory Testing–Part 1

Part 1 | Part 2 | Part 3 | Part 4

I’m going to assume that you’re familiar with exploratory testing and personal kanban for this post. There are many good references out there and I encourage to get out there and read them and at the same time, keep your mind fresh for bringing your own definition into focus.

I have a couple of twisted definitions for you to use so you don’t have to rush around finding links. Just in case.

Exploratory Testing

For my purposes here, exploratory testing is test-driven testing. Weird, huh? It’s like going meta. And being proud of it.

When I explore I don’t start with a script. I run the system under test, so it is a form of dynamic testing; yet I don’t pretend to know what tests are best. So I discover them by running tests.

Test-driven developers elaborate requirements and specify designs by writing tests and use running the tests as an indicator of what’s done. It’s emergent because the number and extent of the examples and specifications required is unknown ahead of time. It’s like the first expression of the requirements is a skull and through continued refinement, the developers fill in the facial features as they work. No one on the team pretends to know it all upfront, that is, no one knows what the face will actually look like until they go through the process.

Test-driven testers elaborate tests by running tests. Their drive is to find a better test than the one they just ran. Repeatedly. Once they discover a gem of a test, they look back at their journal and perhaps ‘Save As…’ so that someone else knows how they got there and how to run it again. I’m ignoring the session-based aspect of the exploration and test chartering on purpose -  don’t need those details for the rest of this post. At least I don’t think you do. Aside: In the big picture they are important.

Personal Kanban

I use personal kanban as a mechanism for visualizing the ways that I could improve the work that I do. Visualizing the work is the key since it forces you to ask questions about your work:

• what am I delivering (work types, work items)
• what is the life cycle I use as I complete work items (board design)
• who am I trying to help by doing this work (who defines done)
• what is my capacity (work in process limits)

These are all good questions, and the kanban board (left-to-right visualization of the work in play) is the radiator for the visualization. I envision each work item as one of those glowing orbs that changes colour. Green if the exploration is going well, red if I’m stuck. Yellow and orange if I’m swamped with uncertainty.

A Mashup of Exploratory Testing and Personal Kanban

Notice it says, “a mashup”. Not “the mashup”. This is a personal thing. What you end up with might be different, and that would be OK. The more you use your board, the more you will want to fiddle. So fiddle.

Let’s go through those questions.

What Am I Delivering – Work Types and Work Items

A great debate ensued. Fortunately I was the only debater. Once I realized I was talking to my own foot, I relented and just got to work.

If I’m exploratory testing, what should flow across the board? Candidates: test scripts (no, they don’t exist until the very end, if at all). test cases (yeah, but I don’t know what they are ahead of time PLUS I could cheat and make these as trivial as I needed to in order to demonstrate hyper-productivity to my bosses.

Requirements. Scenarios. Minimum Viable Product. These are better for one reason: I’m not the only one that defines them. Their definition is (ideally, usually?) a team thing, or at least there is some consensus on what is being delivered. I tried these for a bit, and sure, they worked. What I soon discovered though was that I could put a lot through my board and nobody cared. What was important to me was cadence – I needed to achieve something and if the work item was too big, I lost that sense of achievement. If the work item was too small, it seemed like I was inventing progress.

So I came up with a definition for myself of a “minimum beneficial feature” or MBF as a work item. I adapted the definition so that I could focus on a feature at a time that someone actually cared about. Seems congruent with definitions of quality that I had read. At the same time, it needed to be small enough to give me that cadence. I like moving 2-5 things per day as a tester. In acceptance testing, when the testers are business testers, this would be per user.

In one session I gathered the business leads for the work areas that were expected to adopt the solution and most affected by its smooth operation. Got out the index cards and markers. Described what I needed. Gave them time to write on cards. Role-played the test process using a three-column kanban board as the backdrop (to test, being tested, done).

And then I typed up all the index cards and used that as our collective backlog.

I didn’t immediately see a need for work types when I used minimum beneficial feature as the work item. So I didn’t push it. And that worked for me. Let the work types evolve (so I’ll address those later in a subsequent post).

An interesting thing that happened was that over the course of time, a work item that we thought was a natural candidate – a defect or issue – didn’t emerge. We kept needing to retest a corresponding minimum beneficial feature, so we moved it backwards on the board from ‘tested’ to ‘to be tested’ and attached the reasons why this moved backwards to the card.

The other concept that we had to deal with was our desire to practice using the solution under test. Our mission was to find barriers to solution adoption in all affected areas of the enterprise, so acceptance testing was a business simulation – a miniature version of the entire enterprise, all in one testing room. One MBF could therefore intentionally be run through the board several times. Even if there weren’t any issues detected on an earlier run. Do we move the item backwards? Do we define that as a separate work item with the same name? We chose the latter. Worked well.

As an aside: This meant I had some late nights/early mornings understanding what we needed to re-cycle through the board. A lot of talking to the business people during the course of the day, gauging our risk level and their tolerance of it (yesterday’s weather). Every morning during testing I created a fresh daily backlog that was informed from yesterday’s weather and where we had planned to be ahead of time. Management liked that I could report on the number of MBF’s that we expected to test versus the number that we were able to test every day.

As another aside: I wish we had done more of this with business people other than the ones we had seconded for the project. There were still post go-live adoption issues because our testers had become so darn good at using the system. They weren’t good at picking out new user adoption problems by the time our acceptance testing wrapped.

Who Defines Done

I’ll give you a hint. Not you. More coming in Part 2.

Evolving Approaches

One approach is “first I will build n then I will test n”. Test-last thinking. Replace “n” with feature/story/requirement/behaviour/system/minimum viable product/product as required for your context.

Another approach is “first I will describe/run tests for n, then I will build n”. Test-first thinking. Increasingly we’ve seen the benefits of evolving n towards meaning “behaviour” at both micro- and macro levels. Not surprisingly, words other than “describe/run tests” have been suggested, most notably “describe/run examples of” and “describe/run executable specifications for” the desired behaviour. The desired behaviour has been – I believe in most people’s minds – a characteristic of the system under development/test. Testers ask questions like “how will the system respond if I do (this)”?

Many testers stop here.

Let’s not stop here.

Another approach is “if I want to test n, then I will build m” and to extend the behaviour to things other than the system under development. Like people. Or any other part of the whole system.

Not-so-traditional test-first thinking and a subtle difference. We identify a behaviour that we want to test, and ask, given that desirable behaviour, what do I need to build?

• I want to test potential orders turning into real orders on my web site. What do I need to build? (good)
• I want more potential orders to turn into real orders on my web site. What do I need to build? (better)

The second is highlighted as (better) since it’s more specific and provides insight as to what measures/monitors might be required. We need the function to create a real order, we need the monitor, and third – implied – we need the ability to parameterize aspects of this capability so that we can continue to optimize and improve the real order rate on an on-going basis, and do that easily.

There’s a pattern to this: (capabilities, monitors, parameters).

I want more people to sign up to do volunteer work. What capability do I need to build? What monitors do I need to build? What needs to be parameterized so that we can continue to improve the capability?

I want field office personnel to stop using email attachments when sending completed work orders back to head office for billing. What capabilities, monitors, improvement parameters do I need to build?

This is where we need to apply testing skills. Use our agile testing skills for getting the capabilities to done-done. Use our analysis skills for getting the right measures and monitors in place. Use our test design and execution skills to run the experiments that get the feedback to the right people in a timely way. Use our test management skills to smooth over that whole nested set of feedback cycles.

Exciting times.

Beyond the Agile Testing Quadrants

You can build the right product and you can build it right, and still not deliver value to the customer/user. For any number of reasons, they don’t adopt it easily, completely, or on time.

You can blame them. Luddites.

You can  blame others. Microsoft.

You can blame yourself. Incompetent. Need a hairshirt.

Or you can evolve your testing to include user adoption and benefits realization. Ask questions aimed at finding user adoption issues and explore risks to benefits realization.

I remember reading a Twitter post from Joshua Kerievsky about usage data and how evolving a product without it now seemed so inadequate once he/they started using it. In my mind, this extends the agile testing quadrants to finding adoption issues in addition to helping discover the right product.

A/B testing is another example. This is using real customer behaviour to guide the evolution of the product. Imagine what a tester could do – with their skills tuned towards asking the important questions at the right time – to this field. I think it’s filled with opportunity.

In the enterprise domain (where I spend all my time) the agile testing quadrants are not enough. You can follow them bravely and assuredly and still not deliver value because of:

• an inadequate end-user communications plan
• an inadequate end-user training program, or having a gap between training and when the end-users need to start using the solution
• using the same testers in every test cycle or continuously, making them more expert and less skilled in finding adoption issues
• not translating pre-release performance testing to post-release performance monitoring
• …

Consequently, because I love the agile testing quadrants for getting the right product built the right way, I thought to extend the idea to include supporting adoption and supporting benefits realization.

Q1: Discover barriers to solution adoption by the support/sustainment/operations group(s). For example, assess the communications and training for support/sustainment personnel.

Q2: Discover barriers to solution adoption by end-users/customers. For example, assess the communications and training for end-users/customers, use A/B testing to determine/optimize features/interactions the customer cares about.

Q3: Discover risks to benefits realization that originate with the end-users/customers using the solution. For example, monitor the benefits to see if there is progress towards their realization.

Q4: Discover risks to benefits realization that originate in support/sustainment/operations. For example, monitor the solution performance on an on-going basis so that performance bottlenecks don’t become barriers to effective usage that would in turn lead to the planned benefits being realized (already a common practice).

It’s a start. I almost put the agile testing quadrants themselves in Q2 and I might yet put them back there. For now, I’m hoping this conveys the way that testers need to start thinking:

• write a test strategy that lasts forever, not merely to the end of the project and/or release; deployment is not done 
• create checklists that the solution delivery team can evolve forever, not to just get the solution shipped/in production but one they can use to continually increase the likelihood of user adoption and benefits realization; deployment is not done 
• explore barriers to solution adoption; expand the list of test targets to include end-user communications and training; deployment is not done 
• explore risks and barriers to benefits being realized; deployment is not done 
• do this continually; deployment still isn’t done

Maybe you detected the theme that deployment is not done in the above points.

You can’t afford to think like this if there are product defects that decrease the likelihood of user adoption. You’ll be too busy writing bug reports. So yeah, this is complementary to agile testing (and the agile testing quadrants concept). These new quadrants are the next step, at least in the enterprise.

Vacation Smoke

As test managers, we ought to coach others on giving and receiving feedback with grace since creating feedback cycle is so much a part of testing. In that line of thinking, Scott Berkun’s latest book Mindfire contains an essay on feedback that should be required reading for test managers and testers alike.

I have a story about giving/receiving feedback that has stuck with me for many years.

---------------------------------------

A boss from a previous life time told me this story. It was from an ancient time period when you could still smoke on airplanes.

Brian and his wife were going on a vacation to a sunny destination from their home in Winnipeg, Canada – I think it was Cuba. Festive and feeling good, Brian lit up a small cigar just after they got seated and looked out the window, on top of the world.

His wife was beside him in the three-seat aisle – he being in the window seat - and asked him to put out the cigar in her usual way. He made a special point when he was repeating this story to me that her usual way was loving and warm. Not a demand in any sense of the word. They were both feeling great about the vacation, there was no tension there. A simple request.

“It’s bothering me in this space. I don’t care when we’re at home because I can open a window or leave the room but here I can’t leave my seat. Do you mind?”

In relaying his response, Brian choked up a tiny bit. He told me that his response was swift and direct. It was not mean, but it was without hesitation, “No way. I’m on vacation. This is something that I do on vacation.”
She sat back. Disappointed, but committed to enjoying every bit of the vacation including the flights there and back. She closed her eyes, cuddling his arm with both of her hands.

A few minutes later, a woman sat in the aisle seat next to Brian’s wife, settled in, read the in-flight magazine for a bit and then leaned forward, asking Brian if he would mind putting out his cigar. Another polite, non-demanding request (these are Canadians after all). Brian put it out immediately.

This was when Brian really choked up. “There was the woman beside me, the one I married, the beautiful woman that I have planned my life with asking for a simple favour and I refuse her. Yet a stranger asks the exact same favour and I don’t hesitate to comply.” He paused for effect. “Never again.”

To his credit, Brian told the story to all his staff. There are many lessons here; the one about serving those closest to you as you would treat a stranger, and a second about teaching others what you’ve learned. The third is more obscure but clear once I tell the rest of the story.

Brian looked at his wife. She opened one eye at him, a small, sly smile curling the edges of her mouth upward. He was looking at her with an embarrassed smile on his face.

“Sorry honey. That should have been for you.”

“I know.”

The third lesson is his willingness to stop and talk and to do a micro-retrospective on his own behaviour and adapt.

Then there is her response. The graceful acknowledgement and gentle teaching she chose to give him. She could have been mad, she could have embarrassed him further, and she could have spent the rest of the flight making fun of his mistakes, perhaps even with the woman sitting beside her. She chose none of those options; instead she respectfully and gracefully showed him the way.

As test managers, we ought to coach others on giving and receiving feedback with grace. A feedback cycle is an important part of testing. Turning the feedback cycle negative is generally harmful and rarely, if ever, needed. Sure graceful feedback takes energy, but it is worth it.

Benefits Modeling

The user story template helps us build the right thing by identifying why we want a particular capability in the product we are working on.

As a <blank>
I want to <blank>
so that <blank>

Interpreted from a benefits modeling perspective and mapped directly into the structure as it stands, the template is

As a <stakeholder>
I want to <be able to perform or have performed some enabling action>
so that <I get some specific benefit>

The order seems odd to me, that is, identifying the enabling action before we identify the benefit. Most businesses drive initiatives from benefits first, and determine the most appropriate actions second. Maybe it should be

As a <stakeholder>
I want <this benefit>
so I need to <be able to perform or have performed some enabling action>

The best place for this kind of thinking? In the project charter – the vision section. Why have a vision statement when you can have a whole set of benefits and enabling actions? The fit nicely into the concept of story mapping too:

As a <stakeholder>
I want <this benefit>
so I need <this user story>

Now that’s a vision worth sharing.

The Tourist Trap

Recently at a conference I didn’t attend there was a talk on exploratory testing “tours” and that by adopting various personas, you might change up the “tour” you were on to fit the context.
Seems reasonable. The problem is, I don’t like to travel that way.
When I travel I use a tour as a last resort – as in, you aren’t going to see that attraction any other way than to take a tour. Like the Vatican. But Rome, the city? A tour isn’t for me, even if I’m the one making up the tour’s stops and activities.
When in Rome, we stayed in a quiet residential area and walked to local restaurants that only had single-language menus. I had an even better experience traveling to Argentina where I could live there and experience what “there” is and who “they” are.
Traveling in a tour, even one of my own invention, I have never experienced cultural authenticity beyond being an observer. Seth Godin describes this harshly in his book Graceful when he says, “if you go as a tourist, you will come back as you went”. Being a tourist is being there but being insulated, an observer but not someone participating in something bigger.
Here's the trap. A 'tour' in the sense I'm thinking about, is akin to a scripted experience. Restricting me in certain hallways at certain times. Keeping me away from an arrival city so that I only see certain highlights, insulated from ugliness, hustled away from the imperfect.
That’s not the way I want to test. I want the ugly bits, quickly. I want to smell the garbage and follow my nose to the source. I want to appreciate the structure, feel it, breathe it, “get it” in the gut and then see the ugliness in that context.

“You never really understand a person until you consider things from his point of view […] until you climb into his skin and walk around in it.” – Harper Lee in To Kill a Mockingbird spoken by the character Atticus.

So avoid interpreting 'tour' as the happy path tour in tourist brochures. Certainly sometimes this changes my mind about something. Or someone. Opening myself up to someone else’s mental model is a complex experience, needing skills ranging from humility/compassion/patience to technical/problem-solving/communication skills. Understanding is a non-competitive way of gaining credibility with someone else; show them you understand their model, their way, their view. Paraphrase, then… shift. Them, or you.Travel changes you, if you let it.
Because then, when you do find a problem, you already understand before you have to help someone else understand (sorry St. Francis, you’re irresistible again). You can anticipate their fears with what you’re about to say, and help them through that fear. I’m using the fear word in the same sense as in Fearless Change. The fear of change, the fear to change. Because sometimes even a small bug shatters someone else’s view of the way things are. This gives you the insight and ability to lead them. Not push them, cajole them, or bully them into making that change. Lead. Yes, lead. Leadership.
So take your tours, sure. Just don’t stay on the happy path tour. Take the other tours, like commenter Philk points out - the garbage collector tour, the red light tour, ... Don’t do things just to check them off the checklist. Pivot. Shift. Change your mind about the ways things work. Start a new tour without planning ahead.
Be like my Dad - in tears at the Roman Coliseum from grasping the immensity of it all. He let the place in. He hasn’t been the same since.

A Measure of Trust

When we complain about metrics, it seems that we focus on the potential for abuse. A team that will optimize whatever measure that is put into place. A consultant that considers yak-shaving as billable, even if that means their projects experience an overrun.

But what about metrics when the environment is a high-trust environment?

What if the team truly wanted to improve and their management truly wanted them to improve, and their clients were bought in as well? If the potential for abuse is eliminated (or low likelihood) then why not use simple, non-invasive measures to rally around? Why not celebrate that .5d-improvement in cycle time?

Work on trust first, then metrics.

UAT as a Lean Startup

User acceptance testing (UAT) – that misunderstood human activity that is squeezed from front to back by late delivery and set-in-stone go-live dates – is an opportunity to use lean tactics.

The user acceptance team is a cross-functional team. An enterprise solution will have many different flavours of technical folk and an equally diverse range of business folk. If it were a coach on the sidelines, Lean would say to bring those diverse individuals together early to craft an approach together.

The time allocated to UAT is fixed, and might shrink. Our coach Lean would say better to identify those things that really matter and get to them first. An excellent example of a context for relentless prioritization.

The technologies you use to set up, track, and communicate with don’t have to be the expensive ones. Lean would say you can be capital-efficient by favouring simple tools – checklists over test management systems, burndown charts over tool-generated coverage reports, a kanban board for discovered issues over a defecting tracking system (issue severity is a natural attribute to use for defining classes of service).

Then there’s the communication piece. As lean is transparent to and inclusive of the customer, UAT is transparent to and inclusive of its customers – developers and those people making the acceptance decision. Constant communication and adjustment should be expected, and given. The coach would suggest that since a decision is required, asking what information those decision-makers need in front of them to make that decision is prudent. And do that at the beginning.

Ready, Set, Fail

You’re a test manager in a waterfall-style solution delivery project. UAT is coming; it’s in the project plan and the delivery team is working hard to get the solution ready for it. Really hard. The project manager from the humungous-company vendor is coming down hard on them.

The test environment, the test data, the master test data, the people lined up ready to test, the scripts written and distributed. You’ve done your part.

If UAT goes well, you’re the hero of the day.

If UAT uncovers issues, well, you’re telling bad news to a large audience that doesn’t want to hear it. No one wants to hear it. Not the delivery people, not the implementing company, and certainly not the project manager. The aggressive, classically-trained, command-and-control project manager.

They fight to lower the issue severity. They challenge your evidence that the issue isn’t new but rather re-introduced in the rush. They censor your status report and work hard to make sure you are isolated and shielded from the project stakeholders. A subtle censure, sometimes not so subtle. You feel terrible, you feel like a failure for having caused so much trouble.

It’s go-live. Cutover to production. The deployment script is run, the users are notified, stakeholders sign off. And then there are issues. Issues that come in all sizes of pyjamas and none of them can be easily put to bed.You are certain that those scenarios were tested, you know it. You have the evidence. And yet there are issues, some subtlety that wasn’t checked or some difference between the test environment and production got exposed or some scenario varies from what was trained. You feel terrible, you feel like a failure for having caused so much trouble.

--

You’re a test manager on a traditional, waterfall-style delivery project with a command-and-control project manager. You will feel terrible, you will feel like a failure for having caused so much trouble.

Unless.

Unless you know this is coming and prepare everyone, including the project manager, for those issues. Scrap the scripts and replace them with checklists built by people that know the business and convert the checklists into information radiators for those not close to the project. Lower the barriers to issue resolution by avoiding root cause analysis in lieu of finding the fastest route to the individuals that can resolve the issue. Don’t think deployment is done.Use an information radiator for issues that everyone can access and update it daily. Publish the link so that anyone, everyone, can see it from anywhere.

Transparency is the only way to get around the censure, and if you set that up from the beginning, it will be embraced by even the project manager. I’m willing to bet that the project status report will ultimately contains parts of or links to the information radiators you set up.

Working with Business Testers

From Nancy Kelln’s presentation at STPCon,business testers

• are not testers
• do not want to be testers
• require guidance and support
• require their expectations to be managed
• may be working part-time on the project

and all of these things need to be considered when creating the test strategy/approach on a project that will utilize them.

My last two projects as QA/test lead launched for 3500 and 7500 users respectively, and I can say that we only used a smattering of professional testers on those projects (not counting the professional testers that might have been working for the vendors). The end-of-cycle testing was all done by business testers, that is, people that would be classified as full-time end-users of the solution.

I can add the following to Nancy’s list:

• business testers tend towards happy-day scenarios; adjust your coaching so they get a sense of what exploring means
• avoid technical testing terms like boundary value analysis, equivalence class partitioning, etc. in your coaching; introduce the concepts but using examples
• removing obstacles to their testing is critical, especially since they rarely differentiate between a test environment problem and a problem with the solution – it’s all the same to them
• you can use session-based testing and call sheets as a way of managing part-time testers; a call sheet tells them what to investigate, when and gives them space to report back in. I populate the call sheets with items from a main checklist everyone is working from.
• as a test lead, avoid becoming a test enforcer – you won’t be able to convince them to test in a timeline that actually helps you. Instead, make them aware that they are driving the investigation in their area and that how much they test is their decision; then communicate, communicate, communicate what they have told you and what they end up doing.
• business testers can absolutely use exploratory testing even in projects with stringent audit and control requirements; you do need to state that this is the process you will follow, and then be able to demonstrate that you followed that process. You will need to instruct the testers on what evidence they need to gather for you.
• another risk is what I refer to as cloister vision – the business testers get too good at using the system under test and are no longer adequate proxies for the end-users that will soon be required to adopt the solution; mix it up a bit if you can. A core of business testers is a good idea when they are providing test data for downstream workflows but for investigating solution adoption concerns, work with the training people to get some additional testers that have been trained on the solution involved so that you can observe the “unboxing” first-hand.

That last one – cloister vision – was largely responsible for go-live complications that involved call centre software. Yeah. My customer’s customers noticed, and that hurt, big-time. It’s why I keep writing about solution adoption considerations and have started using “deployment is not done” as a mantra.

Whatever In a Flash

If you enjoyed Agile in a Flash: Speed-Learning Agile Software Development by Jeff Langr and Tim Ottinger and were burning to create your own whatever-in-a-flash as a result, then here’s a template for you in Word 2010 format.

I print on card or cover stock and then if it’s low volume, cut it myself. For volume cutting, it’s easier to take it to a print shop with one of those guillotine cutters. For my 35-card stack of 15 sets, it cost me less than $5 for this service and it would have taken me hours to do myself…

Strategy vs. Plan

The strategy survives multiple iterations, multiple releases, multiple projects for the same, albeit evolving, solution.

The plan implements the strategy by naming names, specifying dates, and identifying the risks that guide the investigation for that iteration or  release.

This way, we can use the lessons learned during each investigation to continually improve the strategy and not get lost in the concern about deadlines.

Checklist Formatted as a Dashboard

 

Context: This format was used on a number of user acceptance initiatives; in one of those cases, one of the main components of the solution was delivered by an agile team that exploits both automated unit tests and automated functional tests.

The concept of a “session” was not consistent throughout; on one project, there was one of these per day and the “session” was a 1-4 hour session that an end-user or group of end-users completed. This was useful for scheduling part-time testers into the test effort. I also gave those user-testers a session record template to journal their test results into. On another project, the “session” was really a test cycle and not a session at all, but the format was still useful as an information radiator.

Everything, session records and checklist/dashboard, was posted into a project team site so that everyone could see the latest updates when they wished.

There is another variation of this format that I’ve used that replaced the traffic light metaphor with a weather indicator (5 different images from stormy to sunny) so that there were more than just three choices available.

Sample UAT Checklist in One Page Project Manager format

More details on the one page project manager format for other purposes is available at oppmi.com (my apologies for posting an incorrect URL to the forum earlier).

This dashboard and an associated burn-down chart that showed actual progress as compared to ideal progress are consistently ranked highly in retrospectives (I host a test retrospective separate from the project retrospective for initiatives like these).

One of the oft-hidden benefits of this checklist is that it is useful for the next release of the solution, especially if you revise it every release to reflect what the organization has learned during the previous one. Refactoring, per se.