Baseline Metrics

Why Baselines Come First

How do you know if your chatbot is truly helping or frustrating users?

Anecdotes will not answer that. One angry comment in a review does not tell you how the flow performs overall.

A performance audit starts with baseline metrics: a snapshot of the current experience before you change anything.

Without that snapshot, every design claim stays a guess.

What a KPI Actually Is

A KPI is a key performance indicator: a quantitative marker of how healthy your conversations are.

For conversational products, two markers carry most of the weight. Task Completion Rate and Fallback Rate.

One tracks success. The other tracks breakdown.

Together they describe the state of the user experience in numbers rather than impressions.

Task Completion Rate

Task Completion Rate (TCR) measures how often a user successfully reaches their goal.

A goal might be booking the appointment, tracking the parcel, or resetting the password.

If the user starts the flow and finishes it, that counts as completion. If they abandon halfway, it does not.

Notice that only the conversation that reaches the user's goal counts toward Task Completion Rate.

TCR is your success signal. It tells you whether the conversation actually delivers what people came for.

Fallback Rate

Fallback Rate tracks how often the system fails to understand an input.

Every fallback is a small moment of friction. The user typed something reasonable and the bot replied with confusion.

A rising fallback rate points straight at the parts of the flow where understanding breaks down.

That makes it a friction map, not just a score.

Documenting Before You Change

Record the figures before you ship anything.

That documented starting point is what turns a redesign into an experiment instead of a hope.

Skip it and you face an impossible question: did the new feature improve the journey, or introduce fresh friction?

You cannot answer that after the fact.

From Opinion to Evidence

Baselines move a team beyond anecdotal feedback and toward a data-driven approach to conversational quality.

They give design managers a structured evaluation process that objectively proves the value of design iterations.

Every decision then carries measurable performance data behind it.

That is how conversational features stay aligned with broader product quality standards.

Your Baseline Checklist

Two numbers, one habit.

Task Completion Rate answers whether users reach their goal. Fallback Rate answers how often the system fails to understand them.

Document both before you touch the flow.

Then every future comparison has something honest to stand on.

Should a design manager record baseline metrics before changing a chatbot flow?

• Yes
• No

Answer: Yes

Yes. Baseline metrics document the current experience, making later comparisons possible and turning a redesign into an experiment.

A chatbot's _____ measures users reaching their goal, while its _____ measures failures to understand input.

Answer: A chatbot's Task Completion Rate measures users reaching their goal, while its Fallback Rate measures failures to understand input.

Task Completion Rate is the success signal. Fallback Rate is the breakdown or friction signal.

A chatbot redesign raises Task Completion Rate but also raises Fallback Rate. What is the best interpretation?

• Success improved, but conversational friction also increased
• Both success and understanding improved
• Success declined, but understanding improved
• The metrics cannot describe conversational quality

Answer: Success improved, but conversational friction also increased

The higher Task Completion Rate suggests more users reach their goals, while the higher Fallback Rate indicates more input-understanding problems. Both metrics should be evaluated together.