COMP1100 — Week 3 Notes
Experimentation and Customer Relationships
Week 3 seminar, in two topics. See comp1100 for course logistics.
Outline
- Business model canvas (revision) — hypotheses turning into facts
- Experimentation: why, what makes a good hypothesis, test and learning cards
- Customer relationships: get, keep, grow
- Channels
Experimentation
See [experimentation] — the two types (generative/exploratory and evaluative) and two reasons, the scientific method, Steve Blank’s seven-step lean startup loop, the six data collection methods, and what experiments should look like.
Falsifiability is the whole game. An exercise: design a non-falsifiable hypothesis and say why it isn’t falsifiable. For example —
“This solution will help this segment of customers improve their health over the long term.”
— then convert it into one that can come back false.
Test cards and learning cards
See test-cards-and-learning-cards for both formats and four worked examples from the medical alert system, including the rejected hypothesis that produced a more valuable side-finding than the accepted one.
Customer relationships
See customer-relationships-and-channels — the get/keep/grow lifecycle with its viral loop, the five foundational questions, the awareness→interest→activation funnel and its arithmetic, paid vs earned demand creation, and the churn table.
The churn numbers are the ones worth remembering: at 1% monthly churn you keep 69.8% of customers after three years; at 10% you keep 3.14%.
Key takeaways
- Experiments = gain insights about customers and products.
- Hypotheses must be falsifiable.
- Test cards structure our hypotheses; learning cards structure our learnings.
- Refuted hypothesis ⇒ ask why.
Housekeeping
- All non-personal communication with your legend goes through MS Teams; general questions to ‘General’.
- One-on-one meetings: you must sign up and attend. This is a participation requirement — it’s a meeting with a supervisor.
Homework
As a team: agree on which project challenges to tackle (can be more than one). Derive a set of hypotheses and 5–10 interview questions to understand the problems UQ students have related to your challenges — one set for the whole team.
Individually: interview at least 2 customers each (not family members or close friends), giving 8–10 interviews across the team. It’s fine to go out in pairs, but everyone does their own two. For each: (a) ask for consent, (b) record the interview, (c) transcribe it to text with the software from this week’s activity.
As a team, before next week’s studio: analyse the interview data. What are the common themes? The common pains? Are there particular segments of the UQ student population that have these pains? Then revise your interview questions based on what you learnt — did the interviews suggest new hypotheses?
Next week
Seminar: 2025-08-18-minimum-viable-products-and-prototyping.
Reference material
Customer Relationships and Channels
Two adjacent blocks of the business-model-canvas: how you reach customers, and what relationship you build once you have.
Get, keep, grow
The relationship lifecycle has three phases, and a viral loop running back from grow to get when customers bring in other customers. Effort and cost rise across the three.
Get → Keep → Grow
↖________________↙
viral loop
Foundations for relationships
Five questions to answer before designing any of it:
- Who are our customers?
- What is their motivation for using our product? What matters to them?
- Where do they get their information about products?
- What (or who) influences them?
- How do they buy things?
Get
For a web or mobile product, get decomposes into a funnel:
Awareness → Interest → Activation
And the funnel has brutal arithmetic. The seminar’s worked example:
50,000 clicks at 10c per click → 10% interest → 2% activation
50,000 clicks costs $5,000 and yields 5,000 interested people and 100 activations — $50 per activated customer. Running the numbers early is what stops a channel strategy that can never pay for itself.
Paid vs earned demand creation
| Paid | Earned |
|---|---|
| Public relations agency | Peer-reviewed publications |
| Trade shows | Conference appearances |
| Email campaigns | Blogs, guest articles |
| Social media campaigns | Awards |
Keep
Ways to keep the customers you already have:
- Loyalty programs
- Product updates that keep their interest and continue adding value
- Contests and events to keep engagement
- Horizontal and vertical offerings
- Customer engagement and satisfaction programs — blogs, community support, email
- Social media
Why keeping beats getting: churn
The seminar answers “why focus on keeping existing customers instead of gaining new ones that bring new income?” with arithmetic:
| Monthly churn | Remaining after 3 years | Average customer duration |
|---|---|---|
| 1% | \((1-0.01)^{36} = 69.8\%\) | \(1/0.01 = 100\) months |
| 5% | \((1-0.05)^{36} = 16.6\%\) | \(1/0.05 = 20\) months |
| 10% | \((1-0.10)^{36} = 3.14\%\) | \(1/0.10 = 10\) months |
The gap between 1% and 10% monthly churn is the difference between keeping 70% of your customers after three years and keeping 3%. A small recurring leak dominates almost any acquisition effort — which is why keep is usually the cheapest growth lever available.
Grow
Three moves:
- Up-sell — a bigger version of what they already buy
- Cross-sell — something adjacent
- Referrals — the customer brings the next customer, closing the viral loop
Product-channel fit
Channels are not neutral: a product has to fit the channel that reaches its segment, the same way it has to fit the market. A product whose customers are reached only through a channel you can’t afford, or can’t operate, has a channel problem rather than a product problem.
See value-proposition-canvas for building the customer profile these decisions rest on.
Experimentation
Why experiment?
To gain insights about customers, their problems, and our solutions — before spending the effort to build.
Two types, two reasons
Two types of experiment:
- Generative / exploratory — explore a space.
- Evaluative — find data to support or refute a hypothesis.
Two reasons to experiment:
- Evaluate our understanding of our customers and market fit.
- Evaluate products.
The scientific method
1. State hypothesis → 2. Design experiment → 3. Test with data → 4. Gather insights ↺
The lean startup version of this loop (Steve Blank) expands into seven steps:
- Extract hypotheses
- Prioritise hypotheses
- Design tests
- Prioritise tests
- Run tests
- Capture learnings
- Make progress — and repeat
Both prioritisation steps are easy to skip and shouldn’t be: a team has more hypotheses than it has time, and testing the wrong one first wastes a week.
What makes a good hypothesis?
The full list: testable, reliable (not fake), concise, replicable, specific, ethical. Condensed to four:
| Property | |
|---|---|
| #1 | Measurable/testable ⇒ falsifiable |
| #2 | Specific |
| #3 | Ethical |
| #4 | Simple and clear |
Falsifiability is the load-bearing one. An example of a non-falsifiable hypothesis:
“This solution will help this segment of customers improve their health over the long term.”
Nothing about it can come back false — “improve”, “health” and “long term” are all unmeasurable as stated, so no result could ever refute it. The exercise is to rewrite it until a specific observation could prove it wrong.
Structuring a hypothesis
The test-cards-and-learning-cards format gives hypotheses a fixed shape:
We believe that [target] are [showing behaviour / display interest in] [for reason Y]. To verify that, we will [undertake behaviour] to [identify behaviour]. And measure [particular variable]. We are right if [some measurable statement]. Therefore, we will ask [specific questions].
Types of data collection
Six ways to gather evidence, roughly in increasing order of cost:
| Method | What it is |
|---|---|
| Interviews | Talk with potential customers |
| Observations | Observe people while doing their “jobs” |
| Data collected by others | Find existing data that tests your hypothesis |
| Immersion | Work in your customer’s environment |
| Low fidelity prototyping | Paper prototypes or low-fidelity digital prototypes |
| High-fidelity solutions | Implement new functionality in an existing product |
See prototyping for the last two.
Experiments should
- Be as small and comprehensible as possible.
- Have clear expected outcomes.
- Have a clear definition of success/failure that everyone agrees to — agreed before the results come in, or the team will argue about interpretation afterwards.
- Be as brief as possible when interviewing — preferably under 30 minutes.
More tips
Do: listen; observe people’s behaviour, tone and reactions, not just what they say; ask “why” and “why not”; ask who else you should talk to; ask if you can follow up later.
Don’t: “correct” people (except to clarify a misunderstood question); convince or sell to them.
Learning from experiments
When results come back, interrogate them rather than just recording them:
- If people would not pay for this, what would they pay for?
- If these are not our customers, then who is?
- Did we see any outliers that could open up new ideas?
- Pay attention to why our hypotheses are rejected.
A refuted hypothesis is more informative than a confirmed one — it tells you something you didn’t already believe.
Key takeaways
- Experiments = gain insights about customers and products.
- Hypotheses must be falsifiable.
- Test cards structure our hypotheses; learning cards structure our learnings.
- Refuted hypothesis ⇒ ask why.
Test Cards and Learning Cards
Two Strategyzer card formats that give experiments a fixed shape. A test card is written before an experiment; a learning card is written after it. Together they’re the unit of work in [experimentation], and the artefacts the COMP1100 submissions are built from.
Test card
Five lines, filled in before running anything:
We believe that ______________________
To verify that, we will ______________________
And measure ______________________
We are right if ______________________
Therefore, we will ask ______________________
The “we are right if” line is what forces falsifiability — you commit to a threshold before you see the data, so you can’t move the goalposts afterwards.
Worked example — users
We believe that our end users are people over 65 who live on their own who would use technology to keep in touch with their family more often because they do not see them much.
To verify that, we will interview 50 people over 65 who live on their own.
And measure how many would like more regular digital interaction with their family.
We are right if 70% of older people say that they would like more regular digital interaction with their family.
Therefore, we will ask “how often would you like to interact with your family members?”
Worked example — paying customer
Same product, different party — note that the user and the customer need separate cards:
We believe that our paying customers are close family members of our end users who would use technology to keep in touch with older family members using a photo-sharing app rather than delegate to a service, because they would like assurance that their family members are doing well in a more personal manner.
To verify that, we will interview 50 people who have family members over 65 that live on their own and have an existing emergency alarm system.
And measure how many would prefer regular (every day) digital interaction with their family member rather than delegating to a service.
We are right if 70% of family members say that they would like more regular digital interaction with their family member rather than delegating to a service.
Therefore, we will ask “how often would you like to interact with your older family member?”
Learning card
Four lines, filled in after:
We believed that ______________________
We observed ______________________
From that we learnt that ______________________
Therefore, we will ______________________
Worked example — hypothesis accepted
We believed that our end users are people over 65 who live on their own who would like to interact digitally with their family more often because they do not see them much.
We observed that 44 out of 50 people we interviewed confirmed that they would like daily digital interaction with their family.
From this we learnt that our end users would consider using a product that supported this.
Therefore, we will accept this hypothesis.
44/50 is 88%, over the 70% threshold the test card committed to — so the decision is already made by the card, not by argument.
Worked example — hypothesis rejected
This is the more instructive one:
We believed that close family members would like to interact digitally with older family members using a photo-sharing app.
We observed that 12 out of 50 people we interviewed confirmed that they would use such an app.
From this we learnt that family members would not use a new product. However, we learnt that some family members would send photos from their smartphone to such an app.
Therefore, we will reject this hypothesis, and test a new hypothesis about how family members would take and send photos for their older family member.
Two things to copy from it. First, the rejection produces a replacement hypothesis, not a dead end. Second, the “however” — the most valuable finding was a side observation that the headline number would have buried. This is what [experimentation] means by paying attention to outliers.
The templates are in the course’s business model and testing templates.