Find the Problem Worth Solving

Start With the Bottleneck

In a fictional merchant, staff answer customers' returns requests. Follow the current workflow: staff read a message, find the policy, draft a reply, then send it. Google PAIR recommends starting with the existing workflow and user needs.[[1]](https://pair.withgoogle.com/chapter/user-needs/)

Watching staff at each step shows where the time goes. Some customers phrase returns requests in unusual ways, so staff must stop to work out what is being asked. Then they must adapt the standard reply to fit that request.

Here, unusual wording and reply adaptation cause delay. A fixed template is one approved returns reply that staff insert unchanged. Interpretation means working out what an unusually worded message asks and tailoring the reply to it. An AI draft would take on that interpretation for staff.

So would a better template solve the problem before an AI draft does? A useful draft must follow the store’s approved returns policy and reduce the work staff do. Compare it with the template on those two points. This is how to judge AI suitability for this task.

Set a Credible Baseline

What is the smallest fair comparison?

A conventional baseline is a credible non-AI alternative. Option A inserts a fixed, approved returns template. Option B drafts a message-specific reply for staff review.

Compare them before committing. A polished demo or customer interest does not show that tailored drafting beats the template.

Match Tool to Task

Would you use AI for a Cancel button?

A fixed Cancel button needs predictable behavior. A deterministic control gives the same intended action, and rules can be easier to maintain.[[1]](https://pair.withgoogle.com/chapter/user-needs/)

Variable language changes the choice. If unusual returns wording causes delay, compare a tailored draft against the approved template. If behavior must remain fixed, choose the deterministic control.

Measure the Local Benefit

What evidence would justify the tailored draft?

Compare both options on the same kinds of returns requests. Count replies that staff reviewed and found policy-correct. Measure staff time spent producing each reviewed reply.

These outcomes test the observed delay directly. If staff spend most of their time locating the right policy, faster drafting may not address the delay. In that case, test whether a better policy-search tool helps staff find the needed passage sooner, compared with how they search today.

Keep Evidence Within Bounds

Can Copilot research estimate savings for returns staff? No.

GitHub randomly split 95 professional developers building a JavaScript HTTP server. Mean completion time, the average minutes to finish, was 161 without Copilot and 71 with it.[[2]](https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/)

{{interactive:bar-chart-visualization instanceId="course-media-a19d69f0-4a3c-4448-a148-13643c06bbf0" altText='Two bars show mean completion time for the JavaScript HTTP-server task: without Copilot, 161 minutes; with Copilot, 71 minutes; the experiment randomly split 95 professional developers.' data='{"title":"One controlled task: JavaScript HTTP server (2022)","unit":"minutes","yLabel":"Mean task completion time (minutes)","bars":[{"id":"without-copilot","label":"Without Copilot","value":161},{"id":"with-copilot","label":"With Copilot","value":71}]}'}}

GitHub's 2022 HTTP-server experiment reported mean completion times of 161 minutes without Copilot and 71 with it, a result limited to that task and study population. · Data: Eirini Kalliamvakou, GitHub, September 7, 2022; chart by Wondering

Four parts of that study narrow what it shows. The population was professional developers. The task was building one JavaScript HTTP server. The comparator was working without Copilot. The outcome shown in this chart is task completion time. The study also measured other outcomes; this chart does not represent all of them.

This is bounded benefit evidence: population, task, comparator, and outcome limit the claim. It suggests a hypothesis, not a current, universal estimate.

Returns staff are a different population doing a different task. So merchant savings still require a local returns-workflow comparison: tailored drafts versus the approved template, on the merchant's own requests.

Write a Conditional Decision

Can your team state the decision clearly?

Write the problem statement: returns staff are the user; unusual wording is the bottleneck; tailored drafting is the feature; the approved template is the alternative. You will carry this problem statement into the feature specification in the next lesson.

Compare policy-correct reviewed replies and staff time. Useful local improvement means the tailored draft does better than the template on those measures for the merchant's own requests. That improvement is needed before launch, but it does not approve launch by itself. The next lesson adds the draft's boundaries, acceptance conditions and a reviewer.

If the draft shows useful local improvement, set those boundaries, acceptance conditions and reviewer in the next lesson. If it does not help, keep the template or test policy search when policy lookup is the observed bottleneck. Record any evidence still missing.

For a fixed Cancel button, _____ is the better initial choice.

• a deterministic control
• a tailored draft

Answer: a deterministic control

A deterministic control gives the same intended action every time, so it fits behavior that must remain fixed. A tailored draft is suited to variable language that causes interpretation-related delay.

Returns staff are delayed mainly by unusual customer wording and adapting replies. Which comparison best tests a defensible AI feature specification?

• Tailored draft versus the approved returns template, measuring correctness and staff time
• Tailored draft versus a fixed Cancel button, measuring customer interest
• Approved template versus improved policy search, measuring the number of AI labels used
• Tailored draft versus a polished demo, measuring whether managers prefer its wording

Answer: Tailored draft versus the approved returns template, measuring correctness and staff time

Compare a message-specific tailored draft with the approved returns template, then measure policy-correct reviewed replies and staff time. This tests whether AI improves the observed bottleneck rather than assuming an AI feature is needed.

A local comparison of a tailored returns draft with the approved template finds no useful improvement in policy-correct reviewed replies or staff time. What is the most defensible decision?

• Keep the template or test policy search, while recording missing evidence
• Launch because an external productivity percentage predicts savings
• Launch because customers showed interest in an AI feature
• Replace the template with a tailored draft without further measurement

Answer: Keep the template or test policy search, while recording missing evidence

Without useful local improvement, do not launch the tailored drafting feature. Keep the template or test policy search if finding the policy may be the actual bottleneck, and record the missing evidence.