Chat Performance

​

Why Chat Monitoring Is Different

​

Accessing Chat Performance Metrics

​

Filtering for Chat Channels

​

Key Chat Performance Metrics

​

Response Time Analysis

• Time from user message to AI’s first response
• Average response latency across conversations
• 95th percentile response time (catching slowest responses)
• Response time during peak hours vs. off-peak

User expectations by response time:
- 0-3 seconds: Excellent, feels instant
- 3-7 seconds: Good, acceptable wait
- 7-15 seconds: Fair, user notices delay
- 15-30 seconds: Poor, user getting impatient
- 30+ seconds: Unacceptable, high abandonment risk

Impact:
Every 5 seconds of additional delay increases abandonment by ~10-15%

1. Navigate to Analyze → Conversations
2. Filter to your chat channels (Web, Slack, WhatsApp)
3. Open 20-30 conversations
4. Manually calculate time between user message and AI response
5. Average the results to establish your baseline

• 0-5 seconds average: Excellent chat performance
• 5-10 seconds average: Good, room for optimization
• 10-20 seconds average: Fair, likely impacting satisfaction
• 20+ seconds average: Poor, investigate latency issues urgently

​

Conversation Resolution Rate

Chat Resolution Rate = Resolved Conversations / Total Chat Conversations × 100%

80%+ resolution = Excellent
- AI handles most common questions autonomously
- Users trust and engage with the AI
- Knowledge base covers expected topics well

60-80% resolution = Good
- Reasonable automation with clear improvement opportunities
- Some topic gaps causing escalations
- Review unresolved conversations for patterns

40-60% resolution = Fair
- Significant knowledge gaps or guidance issues
- Users frequently need human help
- May indicate wrong deployment strategy for chat

Below 40% = Poor
- Major problems preventing effective chat automation
- Fundamental knowledge gaps or technical issues
- Urgent investigation required

​

User Satisfaction in Chat (CSAT)

Chat CSAT = (Good + Amazing ratings) / Total Rated Chat Conversations × 100%

Chat users rate based on:
- Speed of response (immediate gratification)
- Conversational flow (natural dialogue)
- Conciseness (no long-winded explanations)
- Tone (friendly, helpful attitude)

Ticket users rate based on:
- Thoroughness (complete information)
- Accuracy (correct answer)
- Follow-through (problem actually solved)
- Professionalism (formal, competent)

• 85%+ = Excellent (users love the chat experience)
• 75-85% = Good (solid chat satisfaction)
• 65-75% = Fair (chat experience needs improvement)
• Below 65% = Poor (fundamental chat quality issues)

• AI is answering but not conversationally
• Responses too formal or lengthy for chat
• Tone feels robotic or unhelpful
• Missing pleasantries (greetings, acknowledgment)

• Knowledge gaps preventing answers
• AI escalating too frequently
• Users frustrated by lack of help

• AI is friendly but not helpful
• Good tone but missing information
• Users appreciate effort despite not resolving issue

​

Handoff Timing and Patterns

Causes:
- User explicitly requests human ("I need to speak to a person")
- AI detects high-stakes topic (billing, cancellation)
- Sentiment-based escalation triggers

Interpretation:
- Normal for certain topics
- Review if happening frequently for routine questions
- May indicate overly conservative escalation rules

Causes:
- AI couldn't answer follow-up questions
- User expressed frustration with AI responses
- Complexity escalated beyond AI capability
- Knowledge gap revealed during conversation

Interpretation:
- Indicates partial AI value (handled initial exchange)
- Review these conversations for knowledge improvements
- Good handoff timing if complexity genuinely requires human

Causes:
- AI struggled through long conversation before giving up
- User patience exhausted, explicitly requested human
- Circular conversation pattern (AI repeating itself)

Interpretation:
- Problematic - wasted user time before escalating
- Should have escalated earlier if unable to help
- Review for AI confusion or inadequate knowledge

• Business hours (9 AM - 5 PM): Expected for complex questions
• After hours/weekends: May indicate AI confidence issues when team isn’t available
• Specific days: Monday mornings, Friday afternoons may have patterns

1. Filter conversations by escalation trigger
2. Identify most common escalation reasons
3. For each reason, determine if escalation was necessary or avoidable
4. Add knowledge or refine guidance to reduce avoidable escalations
5. Adjust escalation rules if escalating too early or too late

​

Chat Abandonment Rate

• Channels: Web, Slack, WhatsApp (your chat channels)
• Status: Unresolved
• Date range: Last 7-30 days

Abandonment Rate = Unresolved Chat Conversations / Total Chat Conversations × 100%

• 0-10%: Excellent (minimal abandonment)
• 10-20%: Good (reasonable abandonment for difficult questions)
• 20-35%: Fair (significant abandonment, investigate causes)
• 35%+: Poor (losing too many users mid-conversation)

Pattern: User asks question, never responds to AI's answer

Causes:
- AI response took too long to arrive (user left)
- AI answer didn't address the question
- User found answer elsewhere (help docs, Google)
- User got distracted or lost interest

Action:
- Check response time for these conversations
- Review answer relevance
- Improve initial response quality

Pattern: Back-and-forth exchange, then user stops responding

Causes:
- AI couldn't answer follow-up questions
- Conversation becoming circular (repeating information)
- User frustrated with lack of progress
- AI provided partial solution, "good enough" for user

Action:
- Read 10-20 of these conversations
- Identify where conversations break down
- Add missing knowledge for common follow-ups
- Improve guidance for handling multi-turn conversations

If many 1-message conversations are unresolved:
- Response time or initial answer quality issues

If many 3-5 message conversations are unresolved:
- Knowledge gaps for follow-up questions
- AI not handling multi-turn dialogue well

If many 6+ message conversations are unresolved:
- AI getting lost in long conversations
- Should escalate earlier if unable to resolve

​

Peak Hour Coverage

• Traffic volume is 3-5x higher than off-peak
• Users expect instant responses
• Multiple concurrent conversations strain resources
• Human agents are busy and can’t easily take handoffs

• Shows conversation volume by day of week and hour
• Darker colors indicate higher volume
• Identify your busiest 3-4 hour blocks

• Select specific date ranges during peak hours
• Note CSAT, Resolution Rate, and involvement levels
• Export conversations for detailed analysis

Example comparison:

Peak hours (10 AM - 2 PM weekdays):
- 250 conversations per day
- 72% CSAT
- 68% resolution rate
- 15 second avg response time

Off-peak hours (8 PM - midnight):
- 40 conversations per day
- 82% CSAT
- 85% resolution rate
- 5 second avg response time

Insight: Performance degrades during peak load
Action: Investigate whether issue is response time, quality, or both

• AI latency increases during high traffic
• May indicate infrastructure scaling issues
• Consider optimizing data provider queries
• Review caching strategy for common questions

• Resolution rate drops during peaks
• More escalations during busy periods
• May indicate human agents too busy to take handoffs gracefully
• Could suggest AI hesitates to handle questions during peaks

• Peak times see more “Not Involved” conversations
• Agents handling conversations manually without AI
• May indicate team doesn’t trust AI during critical periods
• Review guidance for peak hour scenarios

​

Using the General View for Chat Analysis

​

Conversation Status Chart

• Track daily resolution trends for chat channels
• Identify days when resolution rate dips (investigate those days)
• Correlate status changes with deployments or knowledge updates
• Monitor whether unresolved conversations (abandonment) is growing

1. Apply chat channel filters (Web, Slack, WhatsApp)
2. Set date range to last 30 days
3. Look for trends: Is green (resolved) growing? Is yellow (unresolved) shrinking?
4. Click specific dates to view conversations from that day

​

AI Involvement Rate Chart

Mature chat deployment:
- 70-80% Fully Autonomous (green)
- 10-15% Public Involvement (blue)
- 5-10% Private Involvement (purple)
- 5-10% Not Involved (gray)

Chat differs from ticketing:
- Higher autonomous rate expected (chat handles simpler questions)
- Lower private involvement (less time for copilot review in real-time)
- Lower public involvement (either AI handles or escalates quickly)

• Chat is deployed for questions too complex for full automation
• Knowledge gaps preventing autonomous handling
• Guidance too conservative, escalating unnecessarily
• Review public involvement conversations for improvement opportunities

• Team handling many chats manually without AI assistance
• Integration may not be triggering AI for all conversations
• Agents may be disabling AI during busy periods
• Review deployment settings and team training

​

Message Volume Chart

Messages per Conversation = Total Messages / Total Conversations

Ideal chat conversation length:
- 2-4 messages: Quick questions, efficient resolution
- 5-6 messages: Moderate complexity, good dialogue
- 7-10 messages: Complex but manageable
- 10+ messages: Potentially problematic, investigate

Long chat conversations indicate:
- AI not answering concisely
- Knowledge gaps requiring many clarifying questions
- Circular conversations (AI repeating itself)
- Complex issues that should escalate earlier

​

User Sentiment Chart

• Users express frustration faster in chat than email
• Positive sentiment validates good chat experience
• Sentiment shift mid-conversation indicates AI performance

1. Filter to chat channels
2. Note baseline sentiment distribution
3. Filter to negative sentiment conversations
4. Review whether negative sentiment correlates with:
   • Long response times
   • Unresolved status
   • Many-message conversations
   • Specific topics or times

60-70% Neutral: Normal for factual questions
20-30% Positive: Good experiences, satisfied users
5-15% Negative: Acceptable rate, often reflects problem frustration, not AI frustration

​

Real-Time Performance Monitoring

​

Daily Quick Check (5 minutes)

1. Open Analyze → Metrics, General view
2. Filter to chat channels (Web, Slack, WhatsApp)
3. Review yesterday’s performance:
   • CSAT score: Did it drop from baseline?
   • Resolution rate: Any significant change?
   • Total conversations: Traffic volume normal?
4. Check Conversation Status Chart: Any unusual spikes in escalations or unresolved?

• Sudden CSAT drops (5+ percentage points): Investigate immediately
• Resolution rate degradation: May indicate knowledge issue or technical problem
• Volume spikes: Ensure AI is handling load without quality loss

​

Weekly Deep Dive (30-45 minutes)

1. Review previous week’s chat metrics:
   • Filter date range to past 7 days
   • Note CSAT, resolution rate, involvement rate
   • Compare to previous week’s performance
2. Analyze low-rated conversations:
   • Navigate to Analyze → Conversations
   • Filter: Chat channels, Rating 1-2 stars, Last 7 days
   • Read 10-15 conversations
   • Document common issues (response quality, speed, knowledge gaps)
3. Check abandonment patterns:
   • Filter: Chat channels, Status: Unresolved, Last 7 days
   • Review 10-15 abandoned conversations
   • Note where users are leaving (after how many messages?)
   • Identify topics with high abandonment
4. Review escalation triggers:
   • Filter: Chat channels, Status: Escalated, Last 7 days
   • Check Handoff Chart for escalation reasons
   • Determine if escalations were necessary or avoidable
   • Plan knowledge improvements to reduce avoidable escalations
5. Track improvements:
   • Review conversations for topics you recently improved
   • Verify that changes are having desired effect
   • Document what’s working to replicate success

​

Real-Time Chat Monitoring (During Critical Periods)

1. Keep Analyze → Conversations open and filtered to chat channels
2. Refresh every 15-30 minutes to see new conversations
3. Spot-check recent conversations for quality
4. Watch for unusual patterns (sudden escalation spike, low ratings)

• Define thresholds for manual intervention (e.g., 3 consecutive 1-star ratings)
• Assign team member to monitor metrics during event
• Prepare to adjust AI deployment if quality issues arise
• Have backup plan to route conversations to humans if AI struggles

​

Conversation Flow Analysis

​

Multi-Turn Dialogue Quality

1. Navigate to Analyze → Conversations
2. Filter to resolved chat conversations
3. Use Conversation Length Chart to filter to 5-10 message conversations
4. Review 20-30 examples

Turn 1: User asks initial question
Turn 2: AI provides direct answer with key information
Turn 3: User asks clarifying follow-up
Turn 4: AI addresses follow-up without repeating initial answer
Turn 5: User confirms understanding or thanks AI
Turn 6: AI acknowledges and offers additional help

Quality indicators:
- Each AI response adds new information
- No circular repetition
- Natural conversation flow
- Concise responses (2-4 sentences typically)
- Appropriate tone throughout

Turn 1: User asks initial question
Turn 2: AI provides lengthy, comprehensive answer (8+ sentences)
Turn 3: User asks related follow-up (didn't read full answer)
Turn 4: AI repeats information from Turn 2
Turn 5: User expresses frustration
Turn 6: AI repeats again or escalates

Problems:
- Initial answer too long for chat format
- AI not tracking what it already said
- Missing follow-up knowledge
- Escalating too late after frustrating user

​

Context Retention Analysis

1. Filter to chat conversations with 6+ messages
2. Read user follow-up questions
3. Check if AI responses acknowledge previous context
4. Look for phrases indicating context loss:
   • “As I mentioned earlier…” (good, maintaining context)
   • “I don’t have information about that” (bad, after already discussing topic)
   • User repeating question multiple times (bad, AI not understanding)

• AI treats each user message as independent query
• Follow-up questions reference information from earlier in conversation
• AI doesn’t connect related questions across messages
• Pronouns and references not resolved (user says “What about the other option?” AI doesn’t know what option)

• Ensure guidance emphasizes conversation continuity
• Add examples of multi-turn conversations to training
• Review knowledge structure for related topics that should link together
• Test conversation memory by asking follow-ups in Preview Panel

​

Greeting and Closing Quality

1. Filter to chat conversations
2. Review first AI message in 30-50 conversations
3. Evaluate:
   • Does AI greet user warmly?
   • Is tone conversational, not formal?
   • Does AI acknowledge the question clearly?
   • Is first response fast (ideally under 5 seconds)?

"Hi! I'd be happy to help you with that..."
"Hey there! Let me explain how that works..."
"Hi! Great question about [topic]..."

"Based on your inquiry regarding..." (too formal)
"I will now retrieve information about..." (robotic)
*No greeting, jumps straight to answer* (cold)
"Please wait while I search..." (slow, lacks confidence)

"Is there anything else I can help with?"
"Glad I could help! Let me know if you have other questions."
"Hope that clarifies it! Feel free to ask if you need anything else."

*No closing* - AI just stops responding
"This conversation is now closed." (abrupt)
"Please rate your experience." (pushy without helpfulness check)

​

User Engagement Metrics

​

Repeat User Rate

1. Navigate to Analyze → Metrics, General view
2. Note “Unique Users” count for selected time period
3. Note “Conversations” count for same period
4. Calculate: Conversations per User = Conversations / Unique Users

Conversations per User:
- 1.0-1.2: Low engagement, mostly one-time users
- 1.3-1.8: Moderate engagement, some repeat users
- 1.9-2.5: Good engagement, healthy repeat rate
- 2.5+: High engagement, users trust and return frequently

• Indicates users find chat valuable (come back)
• Lower customer acquisition cost (retention vs. new users)
• Trust in AI capability (wouldn’t return if first experience was poor)

• First-time experience may not be compelling enough
• Users solving problem once and never need help again (could be good!)
• Users avoiding chat after poor initial experience (needs investigation)

​

Follow-Up Question Rate

1. Use Conversation Length Chart to see message distribution
2. Calculate percentage of conversations with 3+ user messages

User message counts:
- 1 message: User asked, got answer, left (or abandoned)
- 2 messages: User asked, got answer, confirmed/thanked
- 3+ messages: User had follow-up questions or clarifications

Follow-up rate = Conversations with 3+ user messages / Total × 100%

Benchmarks:
- 30-45%: Normal, healthy engagement
- 45-60%: High engagement, complex questions or incomplete initial answers
- 15-30%: Low engagement, either very clear answers or users leaving quickly
- Below 15%: Potentially concerning, investigate abandonment

• Users engaged in productive dialogue
• AI handling complex questions through conversation
• Good multi-turn performance

• Users not getting clear answers initially
• Having to ask same question multiple ways
• AI not understanding or answering directly
• Review these conversations for knowledge or guidance issues

• AI providing clear, complete answers immediately
• Users getting what they need quickly
• Efficient resolution (ideal for chat)

• Users abandoning after poor first answer
• Not bothering to ask follow-ups
• Lost confidence in AI after initial interaction

​

Survey Completion Rate

1. Review Conversation Rating Chart in General view
2. Note “Abandoned” bar (surveys offered but not completed)
3. Calculate: Survey Completion = Rated / (Rated + Abandoned) × 100%

60%+ completion: Excellent (users engaged enough to rate)
40-60% completion: Good (reasonable survey engagement)
25-40% completion: Fair (many users skip survey)
Below 25%: Poor (survey fatigue or poor placement)

• CSAT score based on small sample may not represent true satisfaction
• Missing feedback from users who abandoned or had neutral experience
• Hard to identify problems without representative ratings

• Time survey appropriately (immediately after resolution, not after idle time)
• Keep survey simple (1 question: star rating, optional comment)
• Frame survey helpfully: “How did I do?” not “Rate your experience”
• Don’t offer survey if conversation was very short (1-2 messages)

​

Chat Quality Indicators

​

Response Relevance

1. Read 20-30 random chat conversations
2. For each, ask: “Did the first AI response directly address the user’s question?”
3. Calculate: Relevance Rate = Directly Relevant Responses / Total

• AI provides related but not directly relevant information
• AI answers a similar question, not the one asked
• AI gives generic response when user wants specific information
• AI misunderstands due to typos or informal phrasing

• Add examples of actual user questions to guidance
• Include common phrasings and variations in knowledge
• Adjust guidance to prioritize direct answers over comprehensive explanations
• Test with real user questions in Preview Panel

​

Response Conciseness

1. Read 30-50 chat conversations
2. Count sentences in typical AI responses
3. Note which responses feel too long

Simple questions: 1-2 sentences ideal, 3 maximum
"What are your hours?"
Good: "We're open Monday-Friday 9am-5pm EST."
Too long: "Thank you for asking about our hours. Our business hours are Monday through Friday from 9:00am to 5:00pm Eastern Standard Time. We're closed on weekends and major holidays. If you need assistance outside these hours, you can email us and we'll respond the next business day."

Moderate complexity: 3-4 sentences, 5 maximum
"How do I reset my password?"
Good: "Click 'Forgot Password' on the login page. Enter your email address, and we'll send you a reset link within a few minutes. The link is valid for 24 hours."

Complex topics: 5-6 sentences, with offer to explain more
"How does your pricing work?"
Good: "We have three plans: Basic ($29/mo), Pro ($79/mo), and Enterprise (custom). Basic includes up to 100 users and core features. Pro adds advanced analytics and integrations. Want me to explain the differences in more detail?"

• Users asking follow-ups that were already answered (didn’t read full response)
• Users saying “too much info” or “just tell me X”
• High abandonment after AI’s first long response
• Many follow-up questions because initial answer buried key info

• Add guidance: “Keep responses to 2-3 sentences for simple questions”
• Show examples of concise vs. verbose answers
• Instruct AI to offer more detail rather than providing it upfront
• Remove unnecessary preambles (“Thank you for contacting us today…”)

​

Tone Appropriateness

1. Read 20-30 chat conversations
2. Note formal or robotic phrasing
3. Check if tone matches your brand voice

- Conversational: "Let me help you with that" not "I shall assist you"
- Friendly: "Happy to explain!" not "I will now provide information"
- Natural: "That's a great question" not "Your inquiry has been noted"
- Brief: "Sure!" not "Certainly, I would be happy to assist"
- Human: "I understand that's frustrating" not "Your frustration is acknowledged"

"Per your request..." (too formal)
"I will now retrieve..." (robotic)
"Thank you for contacting..." (call center script)
"Your inquiry regarding..." (ticket language)
"Please be advised..." (legal tone)
"As previously stated..." (condescending)

• Add tone guidance: “Respond conversationally, like a helpful colleague”
• Provide good/bad example pairs in guidance
• Test tone by reading responses out loud—if you wouldn’t say it to a friend, it’s too formal
• Review low-rated conversations for tone complaints

​

Best Practices for Chat Monitoring

​

Establish Chat-Specific Baselines

Your baselines (example):
- Chat CSAT: 78% (vs 85% for tickets - acceptable difference)
- Chat resolution: 82% (vs 70% for tickets - expected difference)
- Chat autonomous rate: 75% (vs 65% for tickets)
- Avg chat length: 4.2 messages (vs 8.5 for tickets)

Track changes in these baselines over time:
Week 1: 78% CSAT baseline
Week 4: 81% CSAT (+3pp improvement)
Week 8: 80% CSAT (maintaining improvement)

• Week of [date]
• Chat CSAT: X%
• Chat resolution: Y%
• Avg messages per conversation: Z
• Notes: What changed this week?

​

Weekly Chat Review Ritual

1. Check weekend chat performance:
   • Filter metrics to Saturday-Sunday
   • Note CSAT and resolution rate
   • Compare to weekday performance
   • Check if AI maintained quality without team available
2. Review last week’s improvements:
   • Did knowledge additions improve specific topics?
   • Did guidance changes affect tone or conciseness?
   • Track one improvement at a time to measure impact
3. Identify this week’s focus:
   • Pick 1-2 specific issues to address
   • Set measurable goal (e.g., “Reduce ‘Billing’ topic abandonment from 25% to 18%”)
   • Plan specific actions (add 3 billing snippets, update guidance)

1. Quick metrics check for the week
2. Note any significant changes or incidents
3. Plan weekend monitoring if needed
4. Document week’s performance for next Monday review

​

Conversation Sampling Strategy

• 3 most recent conversations (spot-check quality)
• 2 lowest-rated from today (identify acute issues)
• 2-3 random conversations (avoid bias)

• 10 lowest-rated (understand dissatisfaction)
• 10 escalated (identify avoidable escalations)
• 10 unresolved/abandoned (understand where users leave)
• 10 highly-rated (learn what’s working well)
• 10 random (representative sample)

• Deep dive across all rating levels
• Segment by topic (30 conversations per top topic)
• Segment by channel if using multiple chat platforms
• Track improvement in previously problematic areas

​

Correlate Chat Metrics with Business Outcomes

Track: What percentage of new users start chat conversations?
Insight: If 60% of new users chat, chat is critical to onboarding
Action: Optimize AI for common new user questions

Track: Do users who chat convert at higher rates?
Calculate: Conversion rate for users with chat vs. without
Insight: If chat users convert 2x more, chat directly drives revenue
Action: Proactively offer chat to users showing buying signals

Track: How many chats would have been email or tickets?
Calculate: Chat resolutions × avg ticket handling time × agent cost
Insight: AI chat deflects 300 tickets/week = 75 hours saved = $X savings
Action: Demonstrate ROI to leadership for continued investment

Track: Do users who chat successfully have higher LTV?
Calculate: Average LTV of users with successful chat experience
Insight: Good chat experience correlates with retention
Action: Prioritize chat quality as retention investment

​

Optimization Strategies for Chat

​

Speed Optimization

1. Check data provider sync status (slow providers delay responses)
2. Review knowledge retrieval performance (too many sources slow search)
3. Test during peak vs. off-peak (infrastructure scaling issue?)
4. Check external API timeouts (integration dependencies)

Strategy 1: Prioritize fast data providers
- Snippets respond instantly (direct knowledge)
- Tables respond quickly (structured data)
- Large PDFs respond slowly (full-text search)
- External APIs vary (depends on their performance)

Action: Move frequently-accessed info to snippets

Strategy 2: Optimize knowledge scope
- Reduce total knowledge volume if extremely large (100k+ documents)
- Remove outdated or rarely-used data providers
- Use audience targeting to limit knowledge per conversation

Strategy 3: Implement caching
- Common questions should hit cached responses
- Review top 20 questions and ensure answers are readily available

​

Abandonment Reduction

• Do users leave after 1st AI response? (Answer quality issue)
• Do users leave after 3-4 messages? (Follow-up knowledge gaps)
• Do users leave after long delays? (Response time issue)

Example:
- "Refund Policy": 40% unresolved (high abandonment)
- "Product Features": 15% unresolved (normal)
- "Billing": 35% unresolved (high abandonment)

Focus: Improve Refund Policy and Billing knowledge first

1. Read 20 conversations about that topic
2. Note what information users needed but didn’t get
3. Add that information to knowledge base
4. Update guidance for better handling of that topic
5. Monitor next week’s abandonment rate for that topic

​

CSAT Improvement

1. Filter to low ratings:
   • Conversations with 1-2 star ratings
   • Last 30 days
   • Chat channels only
2. Categorize complaints:
   • Speed issues: “Too slow” “Took forever” “Waited too long”
   • Accuracy issues: “Wrong answer” “Didn’t help” “Incorrect information”
   • Tone issues: “Rude” “Unhelpful” “Robotic” “Formal”
   • Completeness issues: “Not enough detail” “Vague” “Didn’t answer my question”
3. Quantify each category:
   Copy

   From 50 low-rated conversations:
   - 22 (44%) cited accuracy/completeness issues
   - 12 (24%) cited tone/helpfulness issues
   - 10 (20%) cited speed issues
   - 6 (12%) other/unclear
   
   Focus: Fix accuracy/completeness first (highest impact)
   

4. Address root causes:
   • Accuracy issues → Add knowledge, improve data provider quality
   • Tone issues → Refine guidance, add tone examples
   • Speed issues → Optimize response time (see Speed Optimization above)
   • Completeness issues → Ensure answers directly address questions
5. Measure improvement:
   • Track CSAT weekly after changes
   • Read follow-up low-rated conversations to verify issues resolving
   • Target 3-5 percentage point improvement per month

​

Escalation Optimization

- Account-specific information (billing history, private data)
- Complex troubleshooting (requires technical expertise)
- Emotional situations (angry customer needs human empathy)
- Policy exceptions (requires human judgment)
- Sales negotiations (pricing, contracts)

- Simple questions AI should know ("What are your hours?")
- Follow-up questions after partial answer (missing knowledge)
- User requested human because AI was unhelpful (quality issue)
- AI gave up too easily ("I don't have that information" without trying)
- Escalation triggered by keyword but not actually complex

1. Add missing knowledge:
   • Create snippets for questions AI incorrectly escalates
   • Ensure common questions have direct answers
2. Adjust escalation rules:
   • Review trigger keywords (too sensitive?)
   • Increase AI confidence threshold for autonomous handling
   • Add more specific escalation criteria (not just keywords)
3. Improve guidance:
   • Instruct AI to attempt answering before escalating
   • Provide examples of when to escalate vs. when to try
   • Add guidance for handling follow-up questions

​

Troubleshooting Common Chat Issues

​

Response Time Suddenly Increased

1. Check deployment status (recent changes to profile or knowledge?)
2. Review data provider sync health (any providers failing or timing out?)
3. Test in Preview Panel (is delay affecting all questions or specific topics?)
4. Check for infrastructure issues (external APIs down?)
5. Review traffic volume (sudden spike causing congestion?)

• Roll back recent profile changes if timing correlates
• Disable slow or failing data providers temporarily
• Contact support if infrastructure issue
• Scale up resources if traffic spike is sustained

​

CSAT Dropped Suddenly

1. Compare date ranges: When exactly did drop occur?
2. Check for deployments: Did profile change around that time?
3. Read recent low-rated conversations: What specific complaints?
4. Check if drop is topic-specific: Filter by topic, see if isolated
5. Review team changes: New agents affecting handoffs?

• If after deployment: Review changes, roll back if necessary
• If topic-specific: Add knowledge for that topic urgently
• If tone-related: Adjust guidance for better tone
• If unclear: Keep monitoring, may be temporary anomaly

​

High Abandonment Rate

1. Calculate abandonment by message count (where are users leaving?)
2. Check response times (slow responses causing abandonment?)
3. Read abandoned conversations (what questions not being answered?)
4. Filter by topic (specific topics causing abandonment?)
5. Compare to previous period (sudden change or gradual trend?)

• Immediate triage: Add knowledge for top 3 abandonment topics
• Speed fixes: Optimize slow data providers
• Quality fixes: Improve initial response relevance
• Follow-up fixes: Add knowledge for common follow-up questions

​

Users Repeatedly Asking Same Question

• AI not directly answering the question
• AI providing related info instead of specific answer
• AI’s answer buried in long response (user didn’t see it)
• AI misunderstanding question due to phrasing

1. Improve answer directness:
   • Update guidance: “Answer the specific question asked in first sentence”
   • Add examples of direct vs. indirect answers
2. Add knowledge variations:
   • Create snippets with multiple phrasings of same question
   • Include common follow-up formulations
3. Improve conciseness:
   • Shorten responses so key info isn’t buried
   • Lead with direct answer, offer more detail after

​

Chat Works Well in Testing, Poorly in Production

1. Test questions don’t match real user questions:
   • Testing with well-formed questions
   • Real users ask with typos, informal language, context
   • Solution: Test with exact user questions from conversation history
2. Testing during off-peak, production during peak:
   • Response time fine in testing, slow during traffic
   • Solution: Test during actual peak hours
3. Testing individual messages, not full conversations:
   • Single-message test looks good
   • Multi-turn dialogue breaks down
   • Solution: Test full conversation flows, not isolated Q&A
4. Different audiences or channels:
   • Testing Web channel, production includes Slack with different expectations
   • Solution: Test across all active channels

​

Next Steps

• Review Conversations - Read actual chat conversations to understand metrics in context
• General Metrics Dashboard - Access the complete General view for chat analytics
• Analyze Topics - Identify which chat topics need improvement
• Improve Answers - Use chat insights to refine knowledge and guidance
• Test Your AI - Create test suites specifically for chat scenarios
• Ticketing Performance - Compare chat performance to ticketing metrics