Over the course of my career I have had the luxury to position and work with a variety of best-in-class solutions– VMware vSphere, Site Recovery Manager, HCX, Horizon, SRM, vROps Cloud, vRLIC … and lately Docker Desktop, Scout, and Build Cloud – all of which require different strategies and tactics to successfully evaluate a solution.
As part of any pre-sales engagement, running a proof of concept or pilot is a common method companies can use to evaluate software. However, without a framework to approach how to define success, we run the risk of poorly defined criteria, which can lead to scope creep during a POC and reduce the chances of a positive outcome. Evaluating software can be challenging – when I work with customers, often they don’t know how to articulate ‘what good looks like’ – in fact, they frequently ask, “What do other customers do?”
We need to provide a standardized approach to defining how our customers evaluate our solutions. What follows represents a standardized data model and framework for defining and documenting success criteria for presales activities. By providing a standardized approach for defining success during an evaluation effort, we can both constrain the evaluation activities and duration, and at the same time dramatically increase the likelihood of a positive outcome – for both the vendor and the customer.
What are ‘success criteria?’
‘Success Criteria’ – what are they? Essentially, ‘success criteria’ are a set of data points used to measure success or failure of an effort. While this can (obviously) be highly subjective relative to each customer, for any given technical solution usually this can be broken down into ‘use cases,’ ‘test cases,’ ‘expected results,’ and ‘requirements mapping.’
What is a ‘use case?’
A ‘use case‘ is a business-aligned outcome of some kind. In other words, an effort being put forth by some department / team within the company, intended to achieve some kind of measurable result in support of one of the business’ goals. Typically, a use case involves a description of the ways in which a user interacts with a system or product. The following is a sample use case for an app to order food over the internet:
- Use case for meal delivery application: Individuals can use an app to place food orders directly to restaurants. When the user places an order, they are prompted to pay through the app or pay when the food arrives. Once that is confirmed, the restaurant will receive a request through their system. The food will then be prepared, packaged, and delivered to the individual. In this case, the app must be able to receive orders, process payments, and communicate with the restaurant electronically.
- System: Food delivery application
- Primary actor: Customer ordering a meal
- Scenario: The user browses restaurant options. Once the preferred restaurant is selected, they place an order through the application. The user pays online or verifies they will pay in person. The order is sent from the app to the restaurant’s internal system. The restaurant worker receives and processes the electronic order.
This use case illustrates how both the customer and restaurant employee (the actors) interact with the food delivery application (the system) and the expected outcome of each interaction.
What is a ‘test case?’
A ‘test case‘ is a technical task (or series of tasks) that must be accomplished to demonstrate successful achievement of the use case. The simplest set of test cases are simply functionality tests – a set of tasks designed to demonstrate functionality of the application/ solution. Arguably, using such templated success criteria and test cases might better be defined as ‘proof of functionality’ or ‘functional test plans’ for the various aspects of our solution. Ideally, the customer / team we are working with will have their own success criteria they bring to the discussion… Questions you might ask your customer to help uncover custom / unique test cases:
- What are the exact steps that will be used to verify functionality?
- Who is responsible for that solution?
- Do you have a team of folks prepared to perform the testing for that application?
- Do you have a functional test plan for accomplishing this?
- Exactly how will you establish that <solution xyz> will work correctly?
Given the aforementioned use case, examples of test cases might be:
- User can open the mobile app
- User can order a meal using the mobile app
- User is prompted to pay for their meal
- User is able to pay for their meal
- …
… and so on.
While these are a listing of many of the possible tasks or actions associated with the aforementioned use case, note that a customer may not need or desire to prove or accomplish all of the above tasks. It is imperative to discuss, understand, and review the business goals to help establish which test cases might be necessary to execute to demonstrate successful achievement of the use case.
What is a ‘requirement mapping?’
For any given use case and its associated test cases, there are a series of basic tasks that must be executed to be able to achieve the test case in question.
For example, given the first test case above – “User can open the mobile app” – we know the following tasks must be accomplished:
- mobile app must be available / listed in online app store
- user must have privileges to install applications on their mobile device
- mobile device must be connected to a wifi / cellular network for the duration of the installation
Note that requirement mappings are not necessarily a prescriptive, step-by-step, “how-to” guide (although they could be)… rather requirement mappings are specific enough to inform the team what must be accomplished, not necessarily written as a ‘how-to.’ Below is a sample set of test criteria, using basic functionality tests, written for VMware Cloud on AWS – a possible use case here might be to “Migrate an application to VMware Cloud:”
Solution: VMware Cloud on AWS
Use Case: Migrate application from datacenter to VMware Cloud on AWS
| Test | Expected Result | Requirement mapping |
|---|---|---|
| Provision cloud SDDC | Successful instantiation / creation of VMware Cloud on AWS Software Defined Datacenter (SDDC); if using VCDR, provisioned through VCDR interface | Vmware Account activated AWS Account activated AWS VPC created with necessary CIDR and subnets SDDC created and connected to AWS VPC |
| Basic connectivity | Communications between VMC/On-Prem – Direct Connect or IPSec VPN | L3 VPN successfully established – OR – Direct Connect successfully established Open / interact with vCenter in VMC Cold migration of VM is successful |
| Application Load testing – performance benchmark | Compare performance of CUSTOMER application on-prem to in VMC on AWS | Move / migrate CUSTOMER application to VMC on AWS Generate load Compare results |
It is important to note that not only are there specific tests that can be identified, but further that each test has an expected result – this is imperative, as there absolutely must be something to measure for each test. If there is nothing to measure, then all parties are essentially relying on the “I’ll know it when I see it” methodology… which is 100% subjective, cannot be managed, and places one or the other party at a major disadvantage.
If you have thoughts about how you have evaluated technologies in the past, I would love to hear about it! Comment below and join the discussion!
My Occasional IT Blog... 