One Way

There is no one way to do things. One of my favourite quotes about technology that highlights this is from Robert M. Pirsig:

Technology presumes there's just one right way to do things and there never is.

In my current management context (I phrase I’ve adopted via Cem Kaner) I’m the quality control guy for an enterprise program that consists of five projects. Sub-projects, really. Each one of them a commercial-off-the-shelf (COTS) implementation. Each one of them a critical component to long-term technology adoptions plans by the acquiring organization. Each one of them completing the complexity trifecta: people, process, technology.

Checklists to the rescue. Like this one for the sub-project test plans, it’s effectively a functional decomposition of the test activities on a project. The intent is that each sub-project knows what they are doing in each of the following major categories and that each of them also has evidence backing up their claim that they know what they are doing.

• Test Environment Management
• Test Data Management
• Test Management
• Acceptance Management

Remember my context. Enterprise. Commercial-off-the-shelf implementation. We’re not testing the product but the installation/configuration of the product. We’re investigating potential barriers to solution adoption and to realizing the benefits that were planned.

Diving deeper…

Test Environment Management

BTW the colours are a risk assessment. In the enterprise, shared test environments are more complex to deal with than those you control exclusively. The closer you are to acceptance testing, the more likely you will have a shared environment to work in. Systems are inter-connected nowadays, eh. If the system itself is net new, it’s going to connect to something that isn’t new. And so, there be capability gaps that have to be closed, tested in isolation, then tested as part of regular business operations (end-to-end) testing.

Test Data Management

Again not as big a deal if you control the environment exclusively. If you don’t, and chances are that you don’t if you’re working with an enterprise system, then you have to devise how you are going to work with test data.

Test Management

Many people think of the role of a test manager and start/stop at test management. And that’s one of my major motivations behind building this test plan checklist.

I differentiate “design testing” from “design tests”. Word play, perhaps, but in my mind creating a test strategy and formulating a plan that implements that strategy is designing a system to complete the testing. That’s “designing testing”. Designing tests, on the other hand, is the activity of identifying/crafting and executing tests.

Really important: None of these checklist items demand test scripts/procedures. How much (or how little) you need scripts is part of planning and designing testing.

Acceptance Management

This may not be an activity for the test manager, but in most enterprises there is a concept of getting “sign off”. As I’ve mentioned before on this blog (at least I think I did) there are significant advantages to designing the sign-off process with the management context at hand in mind.

Test results play a role in requesting/achieving sign-off, and that’s why it’s here – even if the project managers choose to assume that role for themselves – as a test manager, you’re involved.

The Checklist

My test plan template for the sub-projects consisted of a cover page (enterprise art provided by the enterprise of course), the list of processes in full mind map glory, and a set of test planning principles to uphold (such as “plan to exercise the risky things first”).

The only other twist is that each leaf node on the mind map has a spot for a “if not now, then when” data point; just in case there were things yet to be discovered.

It’s a thinking tool, I think. And hope.

A Mandated Tool

In my current context I’m required to use a certain tool for managing the test effort; you probably know the tool. I won’t mention it by name unless the vendor pays me to, but it’s the first on the list when non-testers choose to purchase a test tool.

My approach is to use this tool as a checklist manager and it works really well for that purpose IF you don’t over-engineer your use of it.

Forget Requirements – List Test Targets

This means avoiding the requirements management trap, that is, using the requirements section for anything but helping you to organize the test effort. A test target is something that needs testing. A requirement, user story, feature, benefit, risk, performance criterion, security criterion, etc.

Test targets are in checklist format and organized in a way that makes it easy to do the next step, create the playbook.

In my current project, I have device specifications (temperature operating range, voltage operating range, etc.) that are clear and unambiguous test targets; I’m lucky. We will employ the same concept when we need to test business processes though.

Forget Test Plans – Create a Playbook

Think of this section as designing/creating a playbook. When we get a new release – we do this. When we test this feature right here, we do that. You should organize the plays in the playbook according to the conditions that exist before you run the play.

Back to my current project – when we receive a device firmware update or upgrade – the set of tests (the play) that we should run to accept that new configuration should be easy to find.

The Test Lab is for Test Sessions

Think of this section as recording the results of test sessions. For any given release or time period, pull plays from the playbook, run the play, discover what more you can/should do, update the play book, run the new/revised plays, rinse, repeat, wipe hands on pants.

At the end of the session, upload your session record so that you have evidence of having tested. Revisions of the playbook are evidence of having learned something about the product and continual improvement.

Current project example again – the February 12 release firmware tests were run over a period of a few days by two engineers and their session record (a Word document) was uploaded into this section in the test management tool. We will create a new folder for the March 29 release and go back to the source – the playbook – to define the firmware test session that will be held at that time. We will not re-run the checklist in the February 12 release section.

Principles

DRY – Don’t repeat yourself. This works in code, and it works in checklists. Don’t organize your test targets by release since that limits your checklist’s value when used three years from now to test a new release or upgrade. If your list of test targets and your list of set plays look the same, then you really haven’t done any design work and you’re not helping your future self as much as you could be.

Ignore the temptation to put a release/time/date anywhere but as metadata for test sessions. Nothing else needs a date since they should survive every release of the solution until they are decommissioned. Test targets and the plays in the playbook are for all time. Well, at least as long as the solution itself survives. I would organize the test sessions

Forget you’re on a project. You’re creating the list of test targets and a playbook for all releases, for the duration of the solution’s useful life. Without this goal – why are you using this tool again?

Requirements Management and Test Tools

When modeling requirements in English, one of the heuristics is to avoid ‘weak verbs’ such as ‘to process’, ‘to handle’ and ‘to manage’. The weak verb is subject to broader interpretation than other, more constrained (strong) verbs and in writing software, broad interpretations are more likely to yield building the wrong thing.

Some test management tools help us ‘manage requirements’. What exactly does that mean? It’s ironic that we use a weak verb in describing a tool used for manipulating (ideally) pairs of strong nouns and strong verbs.

Increasingly, I find that I’m evaluating test management tools based on how they help me work and collaborate around those requirements and how effectively they help me use the requirements as test targets.

• Create/edit/share checklists of test targets (including requirements).
• Version-control and label checklists so I can associate them with other version-controlled artifacts like source code. I don’t need per-item version control, just  version control on the checklist.
• Layer profiles onto the checklist to highlight themes: the high-risk theme unites high-risk items, the new-construction theme unites items that are new, … and I get to invent the layers/profiles that suit my context.
• Create/edit/share exploratory testing session charters based on a selection of items from a checklist
• Create/edit/share exploratory testing session records
• Compile checklists/session records into information radiators / dashboards for myself, for team members, and for external stakeholders.
• Create/edit/share a script based on a selection of items from a checklist (could be one, could be many); creating a script for a given checklist item shouldn’t be the default.

That’s what “manage requirements” means to me, a test manager.

Personal Kanban for Exploratory Testing–Part 2

Part 1 | Part 2 | Part 3 | Part 4

The sensibility of done is something that testers struggle with. Testing is, after all, like a gas. It will fill whatever volume you give it. As testers we know that we can’t exhaustively test enterprise solutions. Yet our project managers and program managers and product owners and most harshly of all some of our test managers ask us to tell them with no measure of uncertainty that the product works. That we are done testing.

Not so fast.

Uncertainty that a solution ‘works’ and is usable/adoptable by the intended user community is a tester’s friend and foe.Friend, since it’s where we go when we want to understand what to test next. Foe, since we can’t really ever eliminate it even though people would like us to. Like limits in calculus approaching zero, we can test and test but ultimately we can’t get that uncertainty to zero. Not from anyone’s perspective.

So ‘done’ for a tester is a hard place to find.

Here’s a happy discovery about personal kanban: you as the performer only define ‘done’ when you yourself are the beneficiary of the work.

• Washing the car, you do a lap to check your work. Then declare done. You might be the only one that cares about the cleanliness of your car so you don’t have to ask anyone else if you’re really done or not.
• A personal development task like hiking to the top of a mountain for the first time has an explicit criterion, but only you can define the mountain that qualifies; it’s your personal development and someone else’s standards for what constitutes a mountain may or may not matter to you.

In those examples there is no flow of consideration between people in making that leap to done.

In testing, and in many other things, there is a flow of something in behind the scenes. My assertion is that when there is an underlying flow of consideration, you don’t define done on your own. You ask. You stop and talk. You listen. You breathe the same air your customer breathes, put yourself in their shoes, walk a mile. To quote a favourite person of mine, you “slide” into their world perspective and observe.

Good thing good testers come with advanced skills of observation.

In my exploratory testing and personal kanban mashup then, it’s not the testers that define done. We stop and talk about what we did. We complete a miniature retrospective on that exploratory testing session that was just completed with the people that we are testing for.

As I alluded to in Part 1, this was something that we failed to execute consistently in our project. As testers we talked to each other and to the respective area business leads but we didn’t account for the types of things that new users would encounter given the training that they were offered. Our testers were already experts by the time we reached the major testing milestones. As a result, we had issues post go-live that I bet we could have avoided if we had let in more new users into our test room. Lesson learned.

Who you talk to depends on what you are testing. If you’re testing a minimum beneficial feature (MBF) then you need to understand who benefits from that feature and talk to them before you move that MBF into the ‘Done’ column. In an enterprise project, the list of potential beneficiaries is:

• direct and indirect end-users for the relevant MBF
• the solution delivery personnel that configured/implemented that MBF
• the decision makers on the project team

We only had one place to stop and talk to the beneficiaries of a feature, and it was at the end in a buffer before the ‘Done’ column. I changed the name of our ‘Done’ column to ‘Communicated’. Thinking back now, that’s too late and our definition around ‘Ready to be Explored’ wasn’t strong enough; it should have included a criterion around understanding the context of the MBF before testing. I assumed we would be OK because we had business testers that were testing within the confines of their role but because of their ‘expert’ status, this was a shaky assumption.

This is the board that I maintained. I would change the ‘Ready to be Explored’ column description to have clearer entry conditions:

- somebody who would know says the MBF is ready to test
- as a tester, I understand the benefits and context of that feature

The last two columns are all about making sure that we’re communicating our test results – positive and negative – to decision makers. Testing is an information service, after all. If you keep the results to yourself for any length of time that’s – er – inventory. Don’t want that.

Here are some questions and concepts around ‘done’ for personal kanban exploratory testing mashups:

• Avoid letting panicking project managers in the ‘done’ conversation.
• There is always something more you could test.
• The risk of the MBF not working and the risk tolerance of those affected are your guides; be wary of posers. I’ve seen posers inflate and deflate risk tolerance levels to suit their own needs. Stop and talk to those truly affected if you can.
• In an enterprise solution delivery scenario as this one was, the list of beneficiaries is finite so you won’t have to dig up new people/groups to talk to for every MBF; you will get to know them and that’s a good thing. Still, don’t pretend to know them so well that you end up trying to represent them.

I felt like this was progress since it helped me understand and communicate where we were with our user acceptance testing effort and I believe that it ultimately helped the project.

Not satisfied though.

I’m anxious to try out new columns next time: “Live” and “In Active Use” so that usage data can be included in the feedback cycle. When the feature is adopted and you’re learning how that MBF is actually being used and adjusting that MBF based on that feedback in subsequent releases seems like a happy place to be.

In our case, we thought operating user acceptance testing like a “business simulation” would help and maybe it did – but as I mentioned we had problems post go-live because our testers were too good with the product/platform. The usage data we collected wasn’t representative of the true "go-live” usage data.

What I Done Told Ya

• For a tester, ‘done’ is a hard place to find.
• Use a self-declaration of done only when there isn’t a flow of consideration between you (the performer) and another person
• When there is a flow of consideration between you and someone else, find the community of people that are truly affected by ‘done’ and explore the definition of done with them.
• In our case, ‘Explored’ wasn’t done. Communicating the results was an important aspect of ‘Done’.
• I look forward to adding ‘Live’ and ‘In Active Use’ columns to the right-hand side of the board in future uses of kanban for exploratory testing.

What’s Next

In Part 3 I’ll talk about the work-in-process (WIP) limits and the experiments I used to come up with them. Like any system, it takes time to come to an equilibrium and there is typically so much going on in large enterprise solution implementations that thinking about WIP limits can be easily crowded out or clouded.

Thanks for getting to here.

Personal Kanban for Exploratory Testing–Part 1

Part 1 | Part 2 | Part 3 | Part 4

I’m going to assume that you’re familiar with exploratory testing and personal kanban for this post. There are many good references out there and I encourage to get out there and read them and at the same time, keep your mind fresh for bringing your own definition into focus.

I have a couple of twisted definitions for you to use so you don’t have to rush around finding links. Just in case.

Exploratory Testing

For my purposes here, exploratory testing is test-driven testing. Weird, huh? It’s like going meta. And being proud of it.

When I explore I don’t start with a script. I run the system under test, so it is a form of dynamic testing; yet I don’t pretend to know what tests are best. So I discover them by running tests.

Test-driven developers elaborate requirements and specify designs by writing tests and use running the tests as an indicator of what’s done. It’s emergent because the number and extent of the examples and specifications required is unknown ahead of time. It’s like the first expression of the requirements is a skull and through continued refinement, the developers fill in the facial features as they work. No one on the team pretends to know it all upfront, that is, no one knows what the face will actually look like until they go through the process.

Test-driven testers elaborate tests by running tests. Their drive is to find a better test than the one they just ran. Repeatedly. Once they discover a gem of a test, they look back at their journal and perhaps ‘Save As…’ so that someone else knows how they got there and how to run it again. I’m ignoring the session-based aspect of the exploration and test chartering on purpose -  don’t need those details for the rest of this post. At least I don’t think you do. Aside: In the big picture they are important.

Personal Kanban

I use personal kanban as a mechanism for visualizing the ways that I could improve the work that I do. Visualizing the work is the key since it forces you to ask questions about your work:

• what am I delivering (work types, work items)
• what is the life cycle I use as I complete work items (board design)
• who am I trying to help by doing this work (who defines done)
• what is my capacity (work in process limits)

These are all good questions, and the kanban board (left-to-right visualization of the work in play) is the radiator for the visualization. I envision each work item as one of those glowing orbs that changes colour. Green if the exploration is going well, red if I’m stuck. Yellow and orange if I’m swamped with uncertainty.

A Mashup of Exploratory Testing and Personal Kanban

Notice it says, “a mashup”. Not “the mashup”. This is a personal thing. What you end up with might be different, and that would be OK. The more you use your board, the more you will want to fiddle. So fiddle.

Let’s go through those questions.

What Am I Delivering – Work Types and Work Items

A great debate ensued. Fortunately I was the only debater. Once I realized I was talking to my own foot, I relented and just got to work.

If I’m exploratory testing, what should flow across the board? Candidates: test scripts (no, they don’t exist until the very end, if at all). test cases (yeah, but I don’t know what they are ahead of time PLUS I could cheat and make these as trivial as I needed to in order to demonstrate hyper-productivity to my bosses.

Requirements. Scenarios. Minimum Viable Product. These are better for one reason: I’m not the only one that defines them. Their definition is (ideally, usually?) a team thing, or at least there is some consensus on what is being delivered. I tried these for a bit, and sure, they worked. What I soon discovered though was that I could put a lot through my board and nobody cared. What was important to me was cadence – I needed to achieve something and if the work item was too big, I lost that sense of achievement. If the work item was too small, it seemed like I was inventing progress.

So I came up with a definition for myself of a “minimum beneficial feature” or MBF as a work item. I adapted the definition so that I could focus on a feature at a time that someone actually cared about. Seems congruent with definitions of quality that I had read. At the same time, it needed to be small enough to give me that cadence. I like moving 2-5 things per day as a tester. In acceptance testing, when the testers are business testers, this would be per user.

In one session I gathered the business leads for the work areas that were expected to adopt the solution and most affected by its smooth operation. Got out the index cards and markers. Described what I needed. Gave them time to write on cards. Role-played the test process using a three-column kanban board as the backdrop (to test, being tested, done).

And then I typed up all the index cards and used that as our collective backlog.

I didn’t immediately see a need for work types when I used minimum beneficial feature as the work item. So I didn’t push it. And that worked for me. Let the work types evolve (so I’ll address those later in a subsequent post).

An interesting thing that happened was that over the course of time, a work item that we thought was a natural candidate – a defect or issue – didn’t emerge. We kept needing to retest a corresponding minimum beneficial feature, so we moved it backwards on the board from ‘tested’ to ‘to be tested’ and attached the reasons why this moved backwards to the card.

The other concept that we had to deal with was our desire to practice using the solution under test. Our mission was to find barriers to solution adoption in all affected areas of the enterprise, so acceptance testing was a business simulation – a miniature version of the entire enterprise, all in one testing room. One MBF could therefore intentionally be run through the board several times. Even if there weren’t any issues detected on an earlier run. Do we move the item backwards? Do we define that as a separate work item with the same name? We chose the latter. Worked well.

As an aside: This meant I had some late nights/early mornings understanding what we needed to re-cycle through the board. A lot of talking to the business people during the course of the day, gauging our risk level and their tolerance of it (yesterday’s weather). Every morning during testing I created a fresh daily backlog that was informed from yesterday’s weather and where we had planned to be ahead of time. Management liked that I could report on the number of MBF’s that we expected to test versus the number that we were able to test every day.

As another aside: I wish we had done more of this with business people other than the ones we had seconded for the project. There were still post go-live adoption issues because our testers had become so darn good at using the system. They weren’t good at picking out new user adoption problems by the time our acceptance testing wrapped.

Who Defines Done

I’ll give you a hint. Not you. More coming in Part 2.

Evolving Approaches

One approach is “first I will build n then I will test n”. Test-last thinking. Replace “n” with feature/story/requirement/behaviour/system/minimum viable product/product as required for your context.

Another approach is “first I will describe/run tests for n, then I will build n”. Test-first thinking. Increasingly we’ve seen the benefits of evolving n towards meaning “behaviour” at both micro- and macro levels. Not surprisingly, words other than “describe/run tests” have been suggested, most notably “describe/run examples of” and “describe/run executable specifications for” the desired behaviour. The desired behaviour has been – I believe in most people’s minds – a characteristic of the system under development/test. Testers ask questions like “how will the system respond if I do (this)”?

Many testers stop here.

Let’s not stop here.

Another approach is “if I want to test n, then I will build m” and to extend the behaviour to things other than the system under development. Like people. Or any other part of the whole system.

Not-so-traditional test-first thinking and a subtle difference. We identify a behaviour that we want to test, and ask, given that desirable behaviour, what do I need to build?

• I want to test potential orders turning into real orders on my web site. What do I need to build? (good)
• I want more potential orders to turn into real orders on my web site. What do I need to build? (better)

The second is highlighted as (better) since it’s more specific and provides insight as to what measures/monitors might be required. We need the function to create a real order, we need the monitor, and third – implied – we need the ability to parameterize aspects of this capability so that we can continue to optimize and improve the real order rate on an on-going basis, and do that easily.

There’s a pattern to this: (capabilities, monitors, parameters).

I want more people to sign up to do volunteer work. What capability do I need to build? What monitors do I need to build? What needs to be parameterized so that we can continue to improve the capability?

I want field office personnel to stop using email attachments when sending completed work orders back to head office for billing. What capabilities, monitors, improvement parameters do I need to build?

This is where we need to apply testing skills. Use our agile testing skills for getting the capabilities to done-done. Use our analysis skills for getting the right measures and monitors in place. Use our test design and execution skills to run the experiments that get the feedback to the right people in a timely way. Use our test management skills to smooth over that whole nested set of feedback cycles.

Exciting times.