COMP1100 — Week 3 Notes

Experimentation and Customer Relationships

Week 3 seminar, in two topics. See comp1100 for course logistics.

Outline

  1. Business model canvas (revision) — hypotheses turning into facts
  2. Experimentation: why, what makes a good hypothesis, test and learning cards
  3. Customer relationships: get, keep, grow
  4. Channels

Experimentation

See [experimentation] — the two types (generative/exploratory and evaluative) and two reasons, the scientific method, Steve Blank’s seven-step lean startup loop, the six data collection methods, and what experiments should look like.

Falsifiability is the whole game. An exercise: design a non-falsifiable hypothesis and say why it isn’t falsifiable. For example —

“This solution will help this segment of customers improve their health over the long term.”

— then convert it into one that can come back false.

Test cards and learning cards

See test-cards-and-learning-cards for both formats and four worked examples from the medical alert system, including the rejected hypothesis that produced a more valuable side-finding than the accepted one.

Customer relationships

See customer-relationships-and-channels — the get/keep/grow lifecycle with its viral loop, the five foundational questions, the awareness→interest→activation funnel and its arithmetic, paid vs earned demand creation, and the churn table.

The churn numbers are the ones worth remembering: at 1% monthly churn you keep 69.8% of customers after three years; at 10% you keep 3.14%.

Key takeaways

  • Experiments = gain insights about customers and products.
  • Hypotheses must be falsifiable.
  • Test cards structure our hypotheses; learning cards structure our learnings.
  • Refuted hypothesis ⇒ ask why.

Housekeeping

  • All non-personal communication with your legend goes through MS Teams; general questions to ‘General’.
  • One-on-one meetings: you must sign up and attend. This is a participation requirement — it’s a meeting with a supervisor.

Homework

As a team: agree on which project challenges to tackle (can be more than one). Derive a set of hypotheses and 5–10 interview questions to understand the problems UQ students have related to your challenges — one set for the whole team.

Individually: interview at least 2 customers each (not family members or close friends), giving 8–10 interviews across the team. It’s fine to go out in pairs, but everyone does their own two. For each: (a) ask for consent, (b) record the interview, (c) transcribe it to text with the software from this week’s activity.

As a team, before next week’s studio: analyse the interview data. What are the common themes? The common pains? Are there particular segments of the UQ student population that have these pains? Then revise your interview questions based on what you learnt — did the interviews suggest new hypotheses?

Next week

Seminar: 2025-08-18-minimum-viable-products-and-prototyping.

Reference material

Customer Relationships and Channels

Two adjacent blocks of the business-model-canvas: how you reach customers, and what relationship you build once you have.

Get, keep, grow

The relationship lifecycle has three phases, and a viral loop running back from grow to get when customers bring in other customers. Effort and cost rise across the three.

  Get  →  Keep  →  Grow
   ↖________________↙
        viral loop

Foundations for relationships

Five questions to answer before designing any of it:

  1. Who are our customers?
  2. What is their motivation for using our product? What matters to them?
  3. Where do they get their information about products?
  4. What (or who) influences them?
  5. How do they buy things?

Get

For a web or mobile product, get decomposes into a funnel:

Awareness → Interest → Activation

And the funnel has brutal arithmetic. The seminar’s worked example:

50,000 clicks at 10c per click → 10% interest → 2% activation

50,000 clicks costs $5,000 and yields 5,000 interested people and 100 activations — $50 per activated customer. Running the numbers early is what stops a channel strategy that can never pay for itself.

Keep

Ways to keep the customers you already have:

  • Loyalty programs
  • Product updates that keep their interest and continue adding value
  • Contests and events to keep engagement
  • Horizontal and vertical offerings
  • Customer engagement and satisfaction programs — blogs, community support, email
  • Social media

Why keeping beats getting: churn

The seminar answers “why focus on keeping existing customers instead of gaining new ones that bring new income?” with arithmetic:

Monthly churn Remaining after 3 years Average customer duration
1% \((1-0.01)^{36} = 69.8\%\) \(1/0.01 = 100\) months
5% \((1-0.05)^{36} = 16.6\%\) \(1/0.05 = 20\) months
10% \((1-0.10)^{36} = 3.14\%\) \(1/0.10 = 10\) months

The gap between 1% and 10% monthly churn is the difference between keeping 70% of your customers after three years and keeping 3%. A small recurring leak dominates almost any acquisition effort — which is why keep is usually the cheapest growth lever available.

Grow

Three moves:

  • Up-sell — a bigger version of what they already buy
  • Cross-sell — something adjacent
  • Referrals — the customer brings the next customer, closing the viral loop

Product-channel fit

Channels are not neutral: a product has to fit the channel that reaches its segment, the same way it has to fit the market. A product whose customers are reached only through a channel you can’t afford, or can’t operate, has a channel problem rather than a product problem.

See value-proposition-canvas for building the customer profile these decisions rest on.

Experimentation

Why experiment?

To gain insights about customers, their problems, and our solutions — before spending the effort to build.

Two types, two reasons

Two types of experiment:

  1. Generative / exploratory — explore a space.
  2. Evaluative — find data to support or refute a hypothesis.

Two reasons to experiment:

  1. Evaluate our understanding of our customers and market fit.
  2. Evaluate products.

The scientific method

1. State hypothesis → 2. Design experiment → 3. Test with data → 4. Gather insights ↺

The lean startup version of this loop (Steve Blank) expands into seven steps:

  1. Extract hypotheses
  2. Prioritise hypotheses
  3. Design tests
  4. Prioritise tests
  5. Run tests
  6. Capture learnings
  7. Make progress — and repeat

Both prioritisation steps are easy to skip and shouldn’t be: a team has more hypotheses than it has time, and testing the wrong one first wastes a week.

What makes a good hypothesis?

The full list: testable, reliable (not fake), concise, replicable, specific, ethical. Condensed to four:

Property
#1 Measurable/testable ⇒ falsifiable
#2 Specific
#3 Ethical
#4 Simple and clear

Falsifiability is the load-bearing one. An example of a non-falsifiable hypothesis:

“This solution will help this segment of customers improve their health over the long term.”

Nothing about it can come back false — “improve”, “health” and “long term” are all unmeasurable as stated, so no result could ever refute it. The exercise is to rewrite it until a specific observation could prove it wrong.

Structuring a hypothesis

The test-cards-and-learning-cards format gives hypotheses a fixed shape:

We believe that [target] are [showing behaviour / display interest in] [for reason Y]. To verify that, we will [undertake behaviour] to [identify behaviour]. And measure [particular variable]. We are right if [some measurable statement]. Therefore, we will ask [specific questions].

Types of data collection

Six ways to gather evidence, roughly in increasing order of cost:

Method What it is
Interviews Talk with potential customers
Observations Observe people while doing their “jobs”
Data collected by others Find existing data that tests your hypothesis
Immersion Work in your customer’s environment
Low fidelity prototyping Paper prototypes or low-fidelity digital prototypes
High-fidelity solutions Implement new functionality in an existing product

See prototyping for the last two.

Experiments should

  • Be as small and comprehensible as possible.
  • Have clear expected outcomes.
  • Have a clear definition of success/failure that everyone agrees to — agreed before the results come in, or the team will argue about interpretation afterwards.
  • Be as brief as possible when interviewing — preferably under 30 minutes.

More tips

Do: listen; observe people’s behaviour, tone and reactions, not just what they say; ask “why” and “why not”; ask who else you should talk to; ask if you can follow up later.

Don’t: “correct” people (except to clarify a misunderstood question); convince or sell to them.

Learning from experiments

When results come back, interrogate them rather than just recording them:

  • If people would not pay for this, what would they pay for?
  • If these are not our customers, then who is?
  • Did we see any outliers that could open up new ideas?
  • Pay attention to why our hypotheses are rejected.

A refuted hypothesis is more informative than a confirmed one — it tells you something you didn’t already believe.

Key takeaways

  • Experiments = gain insights about customers and products.
  • Hypotheses must be falsifiable.
  • Test cards structure our hypotheses; learning cards structure our learnings.
  • Refuted hypothesis ⇒ ask why.

Test Cards and Learning Cards

Two Strategyzer card formats that give experiments a fixed shape. A test card is written before an experiment; a learning card is written after it. Together they’re the unit of work in [experimentation], and the artefacts the COMP1100 submissions are built from.

Test card

Five lines, filled in before running anything:

We believe that          ______________________
To verify that, we will  ______________________
And measure              ______________________
We are right if          ______________________
Therefore, we will ask   ______________________

The “we are right if” line is what forces falsifiability — you commit to a threshold before you see the data, so you can’t move the goalposts afterwards.

Worked example — users

We believe that our end users are people over 65 who live on their own who would use technology to keep in touch with their family more often because they do not see them much.

To verify that, we will interview 50 people over 65 who live on their own.

And measure how many would like more regular digital interaction with their family.

We are right if 70% of older people say that they would like more regular digital interaction with their family.

Therefore, we will ask “how often would you like to interact with your family members?”

Worked example — paying customer

Same product, different party — note that the user and the customer need separate cards:

We believe that our paying customers are close family members of our end users who would use technology to keep in touch with older family members using a photo-sharing app rather than delegate to a service, because they would like assurance that their family members are doing well in a more personal manner.

To verify that, we will interview 50 people who have family members over 65 that live on their own and have an existing emergency alarm system.

And measure how many would prefer regular (every day) digital interaction with their family member rather than delegating to a service.

We are right if 70% of family members say that they would like more regular digital interaction with their family member rather than delegating to a service.

Therefore, we will ask “how often would you like to interact with your older family member?”

Learning card

Four lines, filled in after:

We believed that         ______________________
We observed              ______________________
From that we learnt that ______________________
Therefore, we will       ______________________

Worked example — hypothesis accepted

We believed that our end users are people over 65 who live on their own who would like to interact digitally with their family more often because they do not see them much.

We observed that 44 out of 50 people we interviewed confirmed that they would like daily digital interaction with their family.

From this we learnt that our end users would consider using a product that supported this.

Therefore, we will accept this hypothesis.

44/50 is 88%, over the 70% threshold the test card committed to — so the decision is already made by the card, not by argument.

Worked example — hypothesis rejected

This is the more instructive one:

We believed that close family members would like to interact digitally with older family members using a photo-sharing app.

We observed that 12 out of 50 people we interviewed confirmed that they would use such an app.

From this we learnt that family members would not use a new product. However, we learnt that some family members would send photos from their smartphone to such an app.

Therefore, we will reject this hypothesis, and test a new hypothesis about how family members would take and send photos for their older family member.

Two things to copy from it. First, the rejection produces a replacement hypothesis, not a dead end. Second, the “however” — the most valuable finding was a side observation that the headline number would have buried. This is what [experimentation] means by paying attention to outliers.

The templates are in the course’s business model and testing templates.