Free sample lesson ยท When Not to A/B Test: Tess Rosania's Product Decision Frameworks
Slack's Do Not Disturb: Anchoring a Decision in Worldview, Not A/B Tests
Lesson 1 of the Wondering course When Not to A/B Test: Tess Rosania's Product Decision Frameworks
Quiet Hours by Default
Slack made a bold product choice: Do Not Disturb became a default setting.
Out of the box, notifications go quiet from 10pm to 8am. Users do not have to find a setting or remember to switch it on.
Notice that the person never touched a setting: the quiet overnight hours come built in, which is what makes a default so powerful.
That sounds small. It was not.
This lesson looks at why Slack made the call, and what kind of reasoning carried it.
The Real Risk: On-Call Users
Some Slack customers are on call. They need to be woken up when something breaks at 3am.
For them, a quiet default could mean a missed alert. That is a real cost, and Slack knew it.
- Most users
- On-call users
So the decision had a clear downside for a specific group.
Any honest account of this choice has to start there. Slack did not pretend the risk away.
Where A/B Tests Fall Short
Many product teams settle hard calls with an A/B test: show two versions, measure which one performs better.
Slack did not rest this decision on test data.
A test can tell you what people click. It cannot tell you whether software should respect sleep. That is a question about values.
- Hard product call
- Is it about values?: Ask whether the issue is what people click or what software should do.
- Run an A/B test: Measure which version performs better.
- Anchor in worldview: Use your mission and beliefs to decide.
- Make the call
- No
- Yes
When the question is about what you believe, more data will not answer it for you.
The Anchor: Mission and Belief
Instead, Slack grounded the call in two things: its mission and a clear belief.
The belief: software should not be designed for people who do not need sleep.
Everyone needs sleep. A tool that pings people at midnight by default is built for an imaginary user.
The mission sets direction. The belief turns it into a rule. The default follows from both.
- DND default: Notifications go quiet from 10pm to 8am.
- The belief: Software should not be designed for people who do not need sleep.
- Slack's mission: The mission sets direction for everything above it.
Each level rests on the one below
Two Ways to Decide
Both approaches are useful. They answer different questions.
An A/B test compares outcomes. A worldview anchor asks whether a choice fits who you are and what you are trying to do.
- A/B test
- Worldview
A common mistake is treating every decision as a test waiting to happen.
Slack's DND default is a worldview call. The belief decided it.
Owning the Trade-Off
A worldview decision does not erase the cost. On-call customers still face risk from a quiet default.
What changes is how you defend the choice. You point to your mission and belief, and you stand by them.
That takes more conviction than pointing at a chart.
Using This in Your Own Work
You can spot worldview decisions in your own work.
Watch for questions where data shows what happens, yet the real issue is what should happen.
When you find one, name your belief out loud, check it against your mission, and be honest about who pays the cost.
- Spot it: Notice when the real issue is what should happen.
- Name belief: State your belief plainly, as Slack did about sleep.
- Check mission: Confirm the belief fits your mission.
- Own the cost: Be honest about who pays, like on-call users.
- Decide: Make the call and stand by it.
Check yourself
When Slack made Do Not Disturb a default, users had to _____ for notifications to go quiet from 10pm to 8am
- do nothing
- opt in manually
Answer: do nothing
The setting is on out of the box. Users do not have to find a setting or remember to turn it on, which is what made the choice bold.
In Slack's reasoning, the _____ sets direction, the _____ turns that direction into a rule, and the _____ follows from both.
Answer: In Slack's reasoning, the mission sets direction, the belief turns that direction into a rule, and the default follows from both.
The mission gives broad direction. The belief that software should not be designed for people who do not need sleep makes it a concrete rule. The quiet-hours default is the product outcome that follows from both.
Why did Slack not rest its Do Not Disturb default on A/B test results?
- Tests measure behavior but cannot decide whether software should respect sleep
- Tests would have exposed too many on-call users to missed alerts
- Tests are generally unreliable and should be avoided in product work
- Tests had already shown that most users preferred notifications at night
Answer: Tests measure behavior but cannot decide whether software should respect sleep
An A/B test shows what people click or how versions perform. It cannot settle whether software should respect sleep. That is a values question, so more data would not answer it. The cost to on-call users was acknowledged, not denied, and A/B tests are still useful for other questions.
Keep going
This is lesson 1 of 7 in When Not to A/B Test: Tess Rosania's Product Decision Frameworks. Next up:
- Why 'We Must A/B Test Everything' Is a Fallacy
- Quantify the Firing Squad: Turning Vague Opposition Into a Countable Group
- The Hypothesis Tree: Breaking a Belief Into Falsifiable Sub-Claims
- Why Only 1 in 10 Tweets Mattered: The Logic Behind the Ranked Timeline
- While You Were Away: Using Dismissal Rate as a Validation Signal
Wondering turns any topic into short lessons you can read, watch, or listen to, with practice questions and an AI tutor.