Test Cards and Learning Cards
Two Strategyzer card formats that give experiments a fixed shape. A test card is written before an experiment; a learning card is written after it. Together they’re the unit of work in experimentation, and the artefacts the COMP1100 submissions are built from.
Test card
Five lines, filled in before running anything:
We believe that ______________________
To verify that, we will ______________________
And measure ______________________
We are right if ______________________
Therefore, we will ask ______________________
The “we are right if” line is what forces falsifiability — you commit to a threshold before you see the data, so you can’t move the goalposts afterwards.
Worked example — users
We believe that our end users are people over 65 who live on their own who would use technology to keep in touch with their family more often because they do not see them much.
To verify that, we will interview 50 people over 65 who live on their own.
And measure how many would like more regular digital interaction with their family.
We are right if 70% of older people say that they would like more regular digital interaction with their family.
Therefore, we will ask “how often would you like to interact with your family members?”
Worked example — paying customer
Same product, different party — note that the user and the customer need separate cards:
We believe that our paying customers are close family members of our end users who would use technology to keep in touch with older family members using a photo-sharing app rather than delegate to a service, because they would like assurance that their family members are doing well in a more personal manner.
To verify that, we will interview 50 people who have family members over 65 that live on their own and have an existing emergency alarm system.
And measure how many would prefer regular (every day) digital interaction with their family member rather than delegating to a service.
We are right if 70% of family members say that they would like more regular digital interaction with their family member rather than delegating to a service.
Therefore, we will ask “how often would you like to interact with your older family member?”
Learning card
Four lines, filled in after:
We believed that ______________________
We observed ______________________
From that we learnt that ______________________
Therefore, we will ______________________
Worked example — hypothesis accepted
We believed that our end users are people over 65 who live on their own who would like to interact digitally with their family more often because they do not see them much.
We observed that 44 out of 50 people we interviewed confirmed that they would like daily digital interaction with their family.
From this we learnt that our end users would consider using a product that supported this.
Therefore, we will accept this hypothesis.
44/50 is 88%, over the 70% threshold the test card committed to — so the decision is already made by the card, not by argument.
Worked example — hypothesis rejected
This is the more instructive one:
We believed that close family members would like to interact digitally with older family members using a photo-sharing app.
We observed that 12 out of 50 people we interviewed confirmed that they would use such an app.
From this we learnt that family members would not use a new product. However, we learnt that some family members would send photos from their smartphone to such an app.
Therefore, we will reject this hypothesis, and test a new hypothesis about how family members would take and send photos for their older family member.
Two things to copy from it. First, the rejection produces a replacement hypothesis, not a dead end. Second, the “however” — the most valuable finding was a side observation that the headline number would have buried. This is what experimentation means by paying attention to outliers.
The templates are in the course’s business model and testing templates.