Automate customer service work that is frequent, low risk, emotionally neutral and easy to correct, such as hours, order status, booking and common how-to questions. Keep people on complaints, exceptions and anything costly or hard to undo. Then make it simple to reach a person at any point, because that is what customers worry about most.
What do customers think about AI in customer service?
It pays to start with the customer’s view. In a Gartner survey of 5,728 customers conducted in December 2023 and published in July 2024, 64% said they would prefer that companies did not use AI in their customer service. In the same survey, 53% said they would consider switching to a competitor if they found out a company was going to use AI for customer service.
The reasons are the useful part. Gartner reported that the top concern was that it would get more difficult to reach a person, followed by AI displacing jobs and AI giving wrong answers.
Those concerns are design problems you can solve. Customers are asking for two things: a quick route to a person, and correct answers. The framework below is built around both.
The four tests: volume, risk, emotion and reversibility
List the 15 to 20 most common reasons customers contact you. Then score each one on four tests. This framework is our own working method, and it is simple enough to run in an afternoon.
| Test | Question to ask | Points toward automation | Points toward a person |
|---|---|---|---|
| Volume | How often does this come up? | Daily, with a consistent answer | Rare or different every time |
| Risk | What happens if the answer is wrong? | A minor inconvenience | Legal, safety, health or large financial harm |
| Emotion | How is the customer likely to feel? | Neutral, just needs information | Upset, anxious or grieving |
| Reversibility | Can a mistake be undone easily? | Yes, with a quick correction | No, or only at real cost |
A request that passes all four tests is a strong candidate for full automation. A request that fails one test can often be handled by AI with a person approving the final step. A request that fails two or more should stay with your team.
What usually belongs with AI
- Hours, location, service area and pricing that is already public
- Order, delivery or appointment status
- Booking, rescheduling and reminders
- Common how-to questions answered from your own help content
- Collecting details before a person takes over
Answers should come from your own approved content. Our guide to what RAG is for business chatbots explains how that works.
What usually belongs with people
- Complaints and service failures
- Billing disputes, refunds outside policy and exceptions
- Cancellations where you want to understand the reason
- Anything involving health, safety or legal questions
- Conversations with a distressed customer
How should you design the handoff to a human?
The Gartner finding makes handoff the most important part of the design. These seven rules keep it dependable.
- Offer a person at any time. Let customers type or say “agent” at any point and honor it without argument.
- Set clear triggers. Hand off when the request is out of scope, when the agent is unsure, after two failed attempts, or when the customer’s language shows frustration.
- Pass the context. Send the full conversation and the details already collected so the customer does not repeat themselves.
- Be honest about timing. If no one is available, say when a person will reply and by what channel, then keep that promise.
- Require approval for actions that are hard to undo. Refunds above a set amount, cancellations and account changes get a human check.
- Say it is AI. Tell customers at the start that they are talking with an AI assistant. Several laws address this, as covered in AI disclosure laws for chatbots.
- Close the loop. After a handoff, record what the AI could not handle and use it to improve the content.
How do you monitor quality after launch?
AI tools can state wrong information with confidence. NIST’s Generative AI Profile calls this confabulation, defined as “the production of confidently stated but erroneous or false content.” The same document lists human-AI configuration as a risk, which includes people over-relying on an AI system.
The NIST AI Risk Management Framework states that “AI systems should be tested before their deployment and regularly while in operation.” For a customer service agent, that translates into a routine:
- Before launch: test with real past questions, edge cases and attempts to pull the agent off topic.
- Weekly: read a sample of conversations. Check accuracy, tone and whether handoffs happened when they should have.
- Monthly: review resolution rate, handoff rate, repeat contacts and customer ratings. Look at the trend, since a single number can mislead.
- Whenever something changes: retest after you update prices, policies or the underlying content.
Assign one person to own this review. For a fuller approach, see our AI risk management starter for small business.
Should AI replace your support team or assist it?
Automation is one option. Assistance is another, and the best evidence so far is about assistance.
An NBER working paper, “Generative AI at Work,” by Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied 5,179 customer support agents who were given a generative AI conversational assistant. Access to the tool increased productivity, measured as issues resolved per hour, by 14% on average. Novice and low-skilled workers improved by 34%, while experienced and highly skilled workers saw minimal impact. The authors also report that AI assistance improved customer sentiment and increased employee retention. The paper was first issued in April 2023 and later published in The Quarterly Journal of Economics in 2025.
That study covered one large company, so treat it as strong evidence from one setting. The practical point for a small business is that an AI assistant drafting replies for your team can help newer staff most, and it keeps a person in every conversation.
Many businesses use both: AI answers the simple requests directly and assists people on the complex ones.
The short version
Score each request type on volume, risk, emotion and reversibility, and automate the ones that pass all four. Make reaching a person easy, pass the full context on every handoff, and review real conversations on a schedule. Use AI to assist your team on the harder work that stays human.



