Beyond the Agile Testing Quadrants

You can build the right product and you can build it right, and still not deliver value to the customer/user. For any number of reasons, they don’t adopt it easily, completely, or on time.

You can blame them. Luddites.

You can  blame others. Microsoft.

You can blame yourself. Incompetent. Need a hairshirt.

Or you can evolve your testing to include user adoption and benefits realization. Ask questions aimed at finding user adoption issues and explore risks to benefits realization.

I remember reading a Twitter post from Joshua Kerievsky about usage data and how evolving a product without it now seemed so inadequate once he/they started using it. In my mind, this extends the agile testing quadrants to finding adoption issues in addition to helping discover the right product.

A/B testing is another example. This is using real customer behaviour to guide the evolution of the product. Imagine what a tester could do – with their skills tuned towards asking the important questions at the right time – to this field. I think it’s filled with opportunity.

In the enterprise domain (where I spend all my time) the agile testing quadrants are not enough. You can follow them bravely and assuredly and still not deliver value because of:

• an inadequate end-user communications plan
• an inadequate end-user training program, or having a gap between training and when the end-users need to start using the solution
• using the same testers in every test cycle or continuously, making them more expert and less skilled in finding adoption issues
• not translating pre-release performance testing to post-release performance monitoring
• …

Consequently, because I love the agile testing quadrants for getting the right product built the right way, I thought to extend the idea to include supporting adoption and supporting benefits realization.

Q1: Discover barriers to solution adoption by the support/sustainment/operations group(s). For example, assess the communications and training for support/sustainment personnel.

Q2: Discover barriers to solution adoption by end-users/customers. For example, assess the communications and training for end-users/customers, use A/B testing to determine/optimize features/interactions the customer cares about.

Q3: Discover risks to benefits realization that originate with the end-users/customers using the solution. For example, monitor the benefits to see if there is progress towards their realization.

Q4: Discover risks to benefits realization that originate in support/sustainment/operations. For example, monitor the solution performance on an on-going basis so that performance bottlenecks don’t become barriers to effective usage that would in turn lead to the planned benefits being realized (already a common practice).

It’s a start. I almost put the agile testing quadrants themselves in Q2 and I might yet put them back there. For now, I’m hoping this conveys the way that testers need to start thinking:

• write a test strategy that lasts forever, not merely to the end of the project and/or release; deployment is not done 
• create checklists that the solution delivery team can evolve forever, not to just get the solution shipped/in production but one they can use to continually increase the likelihood of user adoption and benefits realization; deployment is not done 
• explore barriers to solution adoption; expand the list of test targets to include end-user communications and training; deployment is not done 
• explore risks and barriers to benefits being realized; deployment is not done 
• do this continually; deployment still isn’t done

Maybe you detected the theme that deployment is not done in the above points.

You can’t afford to think like this if there are product defects that decrease the likelihood of user adoption. You’ll be too busy writing bug reports. So yeah, this is complementary to agile testing (and the agile testing quadrants concept). These new quadrants are the next step, at least in the enterprise.