“I think we should…” — The four most expensive words in product development.
Last year, we studied 500 companies. The ones driven by opinions grew 12% annually. The ones driven by experiments? 67%.
Here’s how to make the shift.
The HiPPO in the Room
HiPPO: Highest Paid Person’s Opinion. Every company has them. Most companies are ruled by them.
The HiPPO Company Symptoms:
- Meetings start with “I feel like…”
- Decisions justified with “competitor X does this”
- Success measured by launch dates, not outcomes
- Failure blamed on execution, not hypothesis
- Data used to justify decisions already made
The brutal truth: HiPPOs kill more companies than competition.
The $47M Wake-Up Call
Company: Global retailer (name withheld) The HiPPO decision: “Customers want personalization. Rebuild everything.” Investment: $47M, 18 months, 200 engineers Result: 3% decrease in conversion
The experiment they didn’t run: A/B test of personalized vs. simple Cost: $5,000, 2 weeks Would have shown: Customers valued speed over personalization
This isn’t rare. It’s typical.
The Experimentation Transformation: A 90-Day Blueprint
Days 1-30: Foundation (Make It Possible)
Week 1: The Executive Manifesto
Get leadership to publicly commit:
The Experimentation Pledge: “We will test any idea that could impact metrics by >10%, regardless of whose idea it is.”
Simple. Powerful. Non-negotiable.
Week 2: The Infrastructure
You need three things:
- Measurement: Can you track success?
- Randomization: Can you split traffic?
- Visualization: Can everyone SEE results?
Clayva handles all three, but the key is having SOMETHING.
Week 3: The First Test
Pick something controversial:
- CEO’s pet feature
- Design’s sacred cow
- Engineering’s “obvious” improvement
Test it. Publicly. Let the data speak.
Week 4: The Learning Ritual
Institute “Experiment Review Friday”:
- 30 minutes
- Every experiment shown visually
- No judgment on “failures”
- Only question: “What did we learn?”
Days 31-60: Momentum (Make It Normal)
Week 5-6: Democratize Testing
Traditional barrier: “Only data scientists can run tests” Cultural shift: “Anyone can test with guardrails”
Create three tiers:
-
Green Light Tests (anyone can run):
- <10% of traffic
- Reversible changes
- UI/Copy modifications
-
Yellow Light Tests (PM approval):
- 10-50% of traffic
- Pricing/flow changes
- New features
-
Red Light Tests (Executive approval):
-
50% of traffic
- Infrastructure changes
- Business model shifts
-
Week 7-8: The Failure Celebration
Create the “Spectacular Failure Award”:
- Monthly prize for biggest learning from failure
- Winner presents to entire company
- Focus: What hypothesis seemed reasonable? What did we learn?
Real example from Spotify: “Thought users wanted infinite skips. Tested it. Engagement dropped 23%. Learning: Constraints create value.”
Days 61-90: Scale (Make It Inevitable)
Week 9-10: The Velocity Metric
Stop measuring feature delivery. Start measuring learning velocity:
Learning Velocity = (Experiments Run × Average Learning Value) / Time
Traditional Team: 2 experiments/month × low learning = Slow growth
Experimentation Team: 50 experiments/month × high learning = Exponential growthMake this your north star.
Week 11-12: The Canvas Revolution
Move from documents to visual collaboration:
Before: PRD → Jira → Code → Launch → Pray After: Canvas → Visual hypothesis → Test → See results → Iterate
When Netflix moved to visual experimentation, their test velocity increased 400%.
The Cultural Antibodies You’ll Face (And How to Overcome Them)
Antibody 1: “We don’t have time to test everything”
Response: “We don’t have time NOT to test” Data: Companies that test save 65% of development time by not building failures
Show them visually: Screenshot of failed features vs. successful experiments.
Antibody 2: “Our customers are different”
Response: “Prove it” Action: Run an experiment that tests this assumption Result: 94% of the time, customers aren’t that special
Antibody 3: “We already know what works”
Response: “Great, testing will confirm it” Reality: 60% of “obvious” improvements fail when tested Example: Microsoft’s “obviously better” Bing design lost to the ugly version
Antibody 4: “Testing slows us down”
Response: Show them the math:
- Building wrong thing: 3 months wasted
- Testing first: 3 days invested
- ROI: 30x time saved
Antibody 5: “What if the test fails?”
Response: “What if we build it and it fails?”
- Test failure cost: $1,000
- Feature failure cost: $100,000
- Choose your failure.
The Statsig Standard: What Great Looks Like
When OpenAI acquired Statsig, they weren’t just buying tools—they were buying a culture. Here’s what Statsig-level experimentation culture looks like:
- Every decision is a hypothesis
- Every hypothesis is tested
- Every test is learned from
- Every learning is shared
- Every person can experiment
The result: Organizations become learning machines, not feature factories.
Real Transformation Stories
Booking.com: 1,000 Tests Running Simultaneously
- Started: HiPPO-driven travel site
- Transformation: Everything is an experiment
- Result: 30x growth, industry dominance
- Key: Made testing easier than not testing
Microsoft: From Features to Experiments
- Started: Ship Windows every 3 years
- Transformation: 10,000+ experiments monthly
- Result: Azure grew 10x faster than AWS in enterprise
- Key: Bing’s “failure” taught them experimentation > intuition
Amazon: The Ultimate Experimentation Machine
- Started: Jeff’s bookstore
- Transformation: 20,000+ annual experiments
- Result: $500B revenue
- Key: “Disagree and commit—to experiments”
The Visual Advantage in Cultural Change
Traditional experimentation culture fails because it’s abstract:
- Spreadsheets don’t inspire
- Dashboards don’t convince
- P-values don’t persuade
Visual experimentation succeeds because it’s obvious:
- Screenshots show truth
- Heatmaps reveal behavior
- Overlays demonstrate impact
When everyone can SEE the experiment, everyone believes the result.
The 10 Commandments of Experimentation Culture
- Test beats debate (always)
- Data beats opinion (even the CEO’s)
- Fast beats perfect (learn quickly)
- Many beats few (volume wins)
- Visual beats numerical (see to believe)
- Failure beats ignorance (learn from everything)
- Shared beats siloed (everyone experiments)
- Continuous beats periodic (always be testing)
- Hypothesis beats assumption (be specific)
- Learning beats shipping (outcomes over output)
Your 90-Day Checklist
Days 1-30: Foundation
- Executive commitment secured
- Basic infrastructure ready
- First controversial test run
- Weekly review ritual started
Days 31-60: Momentum
- Testing democratized with tiers
- Failure celebration instituted
- 10+ experiments run
- Visual collaboration adopted
Days 61-90: Scale
- Learning velocity measured
- 50+ experiments run
- Cultural antibodies defeated
- Experimentation = default mode
The Transformation Moment
You’ll know you’ve succeeded when you hear:
Before: “I think we should…” After: “Let’s test if…”
Before: “The CEO wants…” After: “The data shows…”
Before: “Ship it and see…” After: “Test it and know…”
The Bottom Line
Culture eats strategy for breakfast. But experimentation culture eats everything else.
In 2025, with AI commoditizing execution, the only competitive advantage is learning speed. And learning speed comes from experimentation culture.
The question isn’t whether to build this culture. It’s whether you’ll build it before your competitors do.
Ready to transform your culture from opinion to experimentation? Clayva makes testing so visual and simple that culture change becomes inevitable. Start your transformation →
The ROI of Culture Change
Investment in experimentation culture:
- Time: 90 days
- Cost: <$50K
- Risk: Minimal
Return from experimentation culture:
- Growth: 3-5x acceleration
- Efficiency: 60% less wasted development
- Innovation: 10x more breakthroughs
- Morale: Teams love winning with data
The math is simple. The impact is transformational.
Remember: Every company becomes data-driven eventually. The only question is whether it happens before or after disruption.
