Personal Kanban for Exploratory Testing–Part 2

Part 1 | Part 2 | Part 3 | Part 4

The sensibility of done is something that testers struggle with. Testing is, after all, like a gas. It will fill whatever volume you give it. As testers we know that we can’t exhaustively test enterprise solutions. Yet our project managers and program managers and product owners and most harshly of all some of our test managers ask us to tell them with no measure of uncertainty that the product works. That we are done testing.

Not so fast.

Uncertainty that a solution ‘works’ and is usable/adoptable by the intended user community is a tester’s friend and foe.Friend, since it’s where we go when we want to understand what to test next. Foe, since we can’t really ever eliminate it even though people would like us to. Like limits in calculus approaching zero, we can test and test but ultimately we can’t get that uncertainty to zero. Not from anyone’s perspective.

So ‘done’ for a tester is a hard place to find.

Here’s a happy discovery about personal kanban: you as the performer only define ‘done’ when you yourself are the beneficiary of the work.

• Washing the car, you do a lap to check your work. Then declare done. You might be the only one that cares about the cleanliness of your car so you don’t have to ask anyone else if you’re really done or not.
• A personal development task like hiking to the top of a mountain for the first time has an explicit criterion, but only you can define the mountain that qualifies; it’s your personal development and someone else’s standards for what constitutes a mountain may or may not matter to you.

In those examples there is no flow of consideration between people in making that leap to done.

In testing, and in many other things, there is a flow of something in behind the scenes. My assertion is that when there is an underlying flow of consideration, you don’t define done on your own. You ask. You stop and talk. You listen. You breathe the same air your customer breathes, put yourself in their shoes, walk a mile. To quote a favourite person of mine, you “slide” into their world perspective and observe.

Good thing good testers come with advanced skills of observation.

In my exploratory testing and personal kanban mashup then, it’s not the testers that define done. We stop and talk about what we did. We complete a miniature retrospective on that exploratory testing session that was just completed with the people that we are testing for.

As I alluded to in Part 1, this was something that we failed to execute consistently in our project. As testers we talked to each other and to the respective area business leads but we didn’t account for the types of things that new users would encounter given the training that they were offered. Our testers were already experts by the time we reached the major testing milestones. As a result, we had issues post go-live that I bet we could have avoided if we had let in more new users into our test room. Lesson learned.

Who you talk to depends on what you are testing. If you’re testing a minimum beneficial feature (MBF) then you need to understand who benefits from that feature and talk to them before you move that MBF into the ‘Done’ column. In an enterprise project, the list of potential beneficiaries is:

• direct and indirect end-users for the relevant MBF
• the solution delivery personnel that configured/implemented that MBF
• the decision makers on the project team

We only had one place to stop and talk to the beneficiaries of a feature, and it was at the end in a buffer before the ‘Done’ column. I changed the name of our ‘Done’ column to ‘Communicated’. Thinking back now, that’s too late and our definition around ‘Ready to be Explored’ wasn’t strong enough; it should have included a criterion around understanding the context of the MBF before testing. I assumed we would be OK because we had business testers that were testing within the confines of their role but because of their ‘expert’ status, this was a shaky assumption.

This is the board that I maintained. I would change the ‘Ready to be Explored’ column description to have clearer entry conditions:

- somebody who would know says the MBF is ready to test
- as a tester, I understand the benefits and context of that feature

The last two columns are all about making sure that we’re communicating our test results – positive and negative – to decision makers. Testing is an information service, after all. If you keep the results to yourself for any length of time that’s – er – inventory. Don’t want that.

Here are some questions and concepts around ‘done’ for personal kanban exploratory testing mashups:

• Avoid letting panicking project managers in the ‘done’ conversation.
• There is always something more you could test.
• The risk of the MBF not working and the risk tolerance of those affected are your guides; be wary of posers. I’ve seen posers inflate and deflate risk tolerance levels to suit their own needs. Stop and talk to those truly affected if you can.
• In an enterprise solution delivery scenario as this one was, the list of beneficiaries is finite so you won’t have to dig up new people/groups to talk to for every MBF; you will get to know them and that’s a good thing. Still, don’t pretend to know them so well that you end up trying to represent them.

I felt like this was progress since it helped me understand and communicate where we were with our user acceptance testing effort and I believe that it ultimately helped the project.

Not satisfied though.

I’m anxious to try out new columns next time: “Live” and “In Active Use” so that usage data can be included in the feedback cycle. When the feature is adopted and you’re learning how that MBF is actually being used and adjusting that MBF based on that feedback in subsequent releases seems like a happy place to be.

In our case, we thought operating user acceptance testing like a “business simulation” would help and maybe it did – but as I mentioned we had problems post go-live because our testers were too good with the product/platform. The usage data we collected wasn’t representative of the true "go-live” usage data.

What I Done Told Ya

• For a tester, ‘done’ is a hard place to find.
• Use a self-declaration of done only when there isn’t a flow of consideration between you (the performer) and another person
• When there is a flow of consideration between you and someone else, find the community of people that are truly affected by ‘done’ and explore the definition of done with them.
• In our case, ‘Explored’ wasn’t done. Communicating the results was an important aspect of ‘Done’.
• I look forward to adding ‘Live’ and ‘In Active Use’ columns to the right-hand side of the board in future uses of kanban for exploratory testing.

What’s Next

In Part 3 I’ll talk about the work-in-process (WIP) limits and the experiments I used to come up with them. Like any system, it takes time to come to an equilibrium and there is typically so much going on in large enterprise solution implementations that thinking about WIP limits can be easily crowded out or clouded.

Thanks for getting to here.