Checklist Formatted as a Dashboard

 

Context: This format was used on a number of user acceptance initiatives; in one of those cases, one of the main components of the solution was delivered by an agile team that exploits both automated unit tests and automated functional tests.

The concept of a “session” was not consistent throughout; on one project, there was one of these per day and the “session” was a 1-4 hour session that an end-user or group of end-users completed. This was useful for scheduling part-time testers into the test effort. I also gave those user-testers a session record template to journal their test results into. On another project, the “session” was really a test cycle and not a session at all, but the format was still useful as an information radiator.

Everything, session records and checklist/dashboard, was posted into a project team site so that everyone could see the latest updates when they wished.

There is another variation of this format that I’ve used that replaced the traffic light metaphor with a weather indicator (5 different images from stormy to sunny) so that there were more than just three choices available.

Sample UAT Checklist in One Page Project Manager format

More details on the one page project manager format for other purposes is available at oppmi.com (my apologies for posting an incorrect URL to the forum earlier).

This dashboard and an associated burn-down chart that showed actual progress as compared to ideal progress are consistently ranked highly in retrospectives (I host a test retrospective separate from the project retrospective for initiatives like these).

One of the oft-hidden benefits of this checklist is that it is useful for the next release of the solution, especially if you revise it every release to reflect what the organization has learned during the previous one. Refactoring, per se.

Feature Advocacy

Take everything you know about bug advocacy – the art of maximizing the likelihood that any given bug is fixed as per its impact on those who care about it – and direct it towards feature advocacy – the art of maximizing the likelihood that any given feature might be adopted as per its value to those who would use it.

In a world where the concept of a ‘project’ as a way of working is being eaten away by pull systems and continuous delivery models, one possible side effect is the death of the bug report.

Features (user stories, etc.) are either ready to adopt by the user community, or they are not. As complex and subjective as that decision might be, it is still ultimately a decision that a person makes. As testers, we investigate and explore and communicate so that the decision is an informed one. The people employing a pull system might require a few more columns than just two – but fundamentally, there is ultimately a state that a feature must reach that says, “as far as we can tell, we can adopt this.”

Everything we gather during testing is data about that particular feature – successes, failures, shortcomings, omissions; its beauty, economy, utility – and all of that information should be available in making that decision. Whether or not that feature plays well with other features is still information about that feature, especially if that other feature has already been in active use. We either move the kanban for the feature to the right, or we do not.

So why create a bug report? Why introduce a whole new ‘thing’ to create, resolve, test, re-open, close, etc.? Why not just add to the information that is known about that feature and ask the appropriate authority to either move it to the right, or send it back to the left?

I understand that there is the capacity for kanban to include a ‘bug’ work type, and a class of service that gets serious ‘bugs’ resolved sooner. The detachment of that ‘bug’ that prevents the adoption of another work item on the board – the feature that was being explored when the ‘bug’ was identified' – is a problem to me and I would vote for splitting the feature if and only if some of the original feature could still be adopted (or considered for acceptance, whatever the end state of the kanban is). I suggest the problem preventing acceptance be kept with the item being considered for said acceptance.

Feature splitting doesn’t generate a bug report, it generates another feature, presumably worded the same way that any other feature is worded. Still no bug report.

And we would again advocate for its adoption given the value it brings to the user community in the context of all the other features being considered.

Checklists, Refactored

Part 1 – Background

Guided exploratory testing is using checklists to support and guide the exploration. In addition, it also means using those checklists as the backbone for communicating to other stakeholders on the project/initiative.

I’ve started to advocate the term ‘testlists’ to replace ‘checklists’ when they are used this way, that is, when checklists are used to support testing and the communications on and about a test initiative/mission (credit goes to Michael Bolton for positing the difference between checking and testing).

Supporting the Acceptance Decision

The procurement process for a new solution includes an acceptance decision, well-described by Gerard Meszaros, Grigori Melnick and Jon Bach in Acceptance Test Engineering Guide – Volume 1 (http://testingguidance.codeplex.com/).

The acceptance decision is usually undertaken late on the project life cycle in a critical-path “acceptance test phase”. This is true even if agile testing, acceptance test driven development (ATDD), or behaviour-driven development (BDD) is the development style of the team delivering the solution. I can’t emphasize that point enough. Ron Jeffries put it succinctly in a tweet:

“don't care if they are ATDD'd. I'm a user of this stuff. Needs to be in the farking manual. if they had a farking manual” – Ron Jeffries

Ron was talking about adopting the software in question, the step that comes after the development and deployment. In short, what happens after ‘done’.

I’m advocating exploratory testing for finding the information needed to make the acceptance decision, as do many others. The tweak that I propose is the use of testlists to support collaboratively planning the exploration, for understanding where the explorers currently are, and for reporting to outsiders on the results, outsiders like internal/external auditors, steering committee members, board members, etc.

Checklists and Testlists

The difference between a checklist and a testlist is the following:

• A checklist lists things to check; a testlist lists things to explore in and around.
• To most people, a checklist is a means of tracking to completion. Check these items and you are done. A testlist is a list of places to start. Where you “end” depends on what you find in the micro-context of each item on the list.
• A testlist has attributes such as risk, business value ranking, persona attached to the items so that explorers can use that information to decide what to explore first, given a limited amount of time.
• A testlist might be hierarchical to support communicating to different audiences. A testlist might also evolve into a mind map depending on the usage and in that case be called a testmap.

We use testlists in the format of the one-page project manager (http://oppmi.com/) so that we can also provide testlist audiences with an indication of status and when/who will do the exploration/investigation. But really any format will do. The content of the testlist has been business processes/workflows/functions from the business user’s perspective, or if one exists, the starting point for the testlist content can be the story map that guided the solution development.

More on format, building testlists, and refactoring testlists in upcoming posts.

A.

Play Me a Song

 

My sister was visiting us a few weeks ago so that she might meet her new niece in person. She lives about 1500km away from us, so it’s just far enough away to make these sorts of visits infrequent.

I learned a lot.

I watched her reverse-engineer a song that I’d been singing to my son and reduced it to its essentials so that I might learn to play it on the piano. In about a minute and a half, she identified the major chord and said, “just play around with that chord with the right rhythm and you’ll get it”.

I didn’t believe it.

That is, until I started playing the song, a couple of weeks after she left. Her gift – countless hours of practice as a performer and teacher that made it possible notwithstanding – was to show me the essentials, guide me with just enough to get started, and then to let me go experiment for myself.

I didn’t get a lesson plan.

She didn’t give me a script, a transcribed version of the song into notes on the paper. She didn’t tell me the rhythm, she let me feel that out for myself. She didn’t tell me what to do when.

Turns out this is what checklists should do for testers, especially those business testers that are not necessarily software test professionals in the first place. They know their business. They know how to use a computer. They need guidance, but as I didn’t need a script to learn a song on the piano, they don’t need a script to test effectively.

In my experience, the more we tell our testers what to do, how to do it, in what order, how often, etc., the more likely it is that they will do just that. And only that. Sticking to script is rarely a good thing in testing.

Checklist items are chords, not notes. If you believe the tester is a person, a sentient being capable of putting emotional labour and thought into their work, then identifying the chords is enough. They will play the notes without you going through the effort of transcribing them.

Give them room to create their own art.

Terminology

There’s an excellent discussion initiated by Gojko Adzic on terminology used in association with acceptance test driven development (ATDD). It’s excellent because it’s playing with people’s mental models of what’s really going on. Getting to the essence of things. I support what’s going on there.

I need to get something out of my head though.

My problem (because I’m not convinced that anyone else shares it) is that we too often change the ‘nouns’ when we are describing what’s going on.  Work items transition from ‘idea’ to ‘feature’ to ‘requirement’ to ‘story’ to ‘example’ to ‘specification’ to ‘test’ with too much fluency for my comfort, like there is implied magic in behind the scenes. It seems we transform work items into completely different things by doing something.

Maybe we do, maybe we don’t. What I need to get out of my head is – what if we just used one noun, and described everything else as a state change?

My first candidate noun was minimal marketable feature (MMF). The challenge, however, was that an ‘automated MMF’ is, well, the solution we’re trying to specify. Didn’t work.

My second candidate was, as in the diagram above, specification. I transcribed Gojko’s terminology into a statechart. I used ‘Refined’ instead of ‘Distilled’ based on a twitter conversation I overheard between Gojko and Elisabeth Hendrickson. This seems to work better.

The ‘Executable’ state is actually a composite state that I almost labelled ‘Automated’. I only prefer ‘executable’ since it’s seems to be used more often in conjunction with my preferred noun (‘specification’) and the term ‘automated’ is more often used with a different noun (‘test’).

I wanted two states to represent the higher-level ‘Executable’ state because I know that just writing and running something in a BDD container doesn’t mean that the specification actually runs the system under development, nor does automagically get incorporated into the continuous validation system Gojko was referring to. I wanted this other state so that I could add the transition and indicate the work that still has to happen for that specification to be useful. In keeping with the spirit of Gojko’s post, the format of the specification even after this work is completed, should still be like it came off the whiteboard, that is, a literal representation of the specification.

After having done this, I believe there is value in unifying the conversation around a single noun, and I believe that the best candidate noun – for now - is ‘specification’.

Other random thoughts on this:

• Interesting that this post isn’t about testing, although testing skills certainly help in doing this.  Gojko called this collaborative specification.
• I’ve de-emphasized the design skills required to complete the transition from ‘Not Integrated’ to ‘Integrated’. Unfortunately. But I don’t want to clutter the diagram.
• My dream – presented as a lightning talk at the first Agile Alliance Functional Testing Tools workshop in 2007 – is that the literal executable specification described above (that one that appears as it did on the whiteboard) runs either inside or outside the GUI as requested at runtime. Most for ‘separation of concerns’ reasons. An easy way to validate the back end is a must, so that business rules get automated and validated first. Then it’s a second step to figure out the optimal user experience, and I would still like to use that very same specification to drive the development of that user experience.

Please continue the discussion on Gojko’s blog so that we keep the thoughts and minds directed in a single place.