- Product Questions (General Users)
- Billing and Subscription Support
- Technical Troubleshooting
- Account Management
- Onboarding Flow

- Enterprise Customer Interactions
- Free Tier Support
- Trial User Engagement
- Premium Support Cases

- Website Chat Behavior
- Zendesk Agent Assist
- Slack Internal Support
- Salesforce Case Deflection

User: "What's the difference between your Pro and Enterprise plans?"

User: "I'm interested in your product"
Assistant: "Great! What's your main use case?"
User: "I need to support 1000 customers per month"
Assistant: "For that volume, I'd recommend our Business or Enterprise plan."
User: "What's the pricing difference?"

User: "Do you offer student discounts for non-profit organizations in Europe?"

User: "It's not working"

Expected: "Enterprise plan includes priority support, dedicated account manager,
and custom SLA. Pricing starts at $500/month."
Threshold: 75%

✓ Pass: "Our Enterprise tier gives you priority support with a dedicated account
manager and customized SLA. Plans begin at $500 monthly."

✗ Fail: "Enterprise is our premium option with lots of benefits. Contact sales
for pricing."

Pattern: \$\d+/month
Use: Verify pricing is mentioned with dollar amount and /month

Pattern: (?i)(enterprise|business|pro) plan
Use: Ensure specific plan tiers are referenced

Pattern: ^(I understand|I see|Got it)
Use: Validate response starts with acknowledgment

Pattern: (?i)(I don't know|I'm not sure|I can't help)
Use: Ensure AI doesn't express uncertainty inappropriately

Pattern: \[TODO\]|PLACEHOLDER
Use: Verify no template text appears in responses

Pattern: (?i)confidential|internal only
Use: Prevent leakage of internal information

Options: helpful, neutral, unhelpful
Expected: helpful
Use: Validate response helpfulness

Options: sales, support, product info, account management
Expected: product info
Use: Check topic categorization

Options: empathetic, professional, casual, robotic
Expected: empathetic, professional
Use: Assess tone appropriateness

✓ Good: "User asks about Enterprise features - AI lists key differentiators"
✗ Avoid: "Test if GPT-4 embedding similarity > 0.85"

✓ Good: Separate tests for "pricing accuracy" and "tone appropriateness"
✗ Avoid: One test checking pricing, tone, format, and tool usage

- Misspellings: "enterprize plan"
- Vague requests: "I need help"
- Out of scope: "What's the weather?"
- Multi-part: "Tell me about pricing and also can I get a demo?"

Expected: "Enterprise plan costs $500/month"
Actual: "Enterprise pricing is $500 monthly"
Threshold: 95%
Result: Failed at 88% similarity

Fix: Lower threshold to 85% or make expected value less specific

Pattern: \$500/month
Actual: "The cost is $500 per month"
Result: Failed (per month ≠ /month)

Fix: Update pattern to \$500.?(per month|/month|monthly)

Expected: "Pro plan includes 50 seats"
Actual: "Pro plan is great for growing teams"
Result: Failed - missing specific information

Fix: Add seat information to knowledge base or guidance

Expected: Professional, concise answer
Actual: Casual, verbose response
Result: Failed tone check

Fix: Check audience filters - may be using wrong guidance rule

Example: "User asks about mobile app availability"
Expected: "Our mobile app is available on iOS and Android.
Download from the App Store or Google Play."

Result: Failed - AI says "I don't have information about mobile apps"
Reason: Knowledge gap

Action: Add snippet about mobile app availability
Content: Platform supports mobile apps on iOS and Android,
available in respective app stores

Build new version with updated knowledge
Deploy to testing environment

Result: Passed - AI now provides correct information

Keep the test in suite to prevent future breaks

1. User asks product question → AI provides accurate answer
2. User requests pricing → AI explains tiers clearly
3. User needs help → AI searches knowledge, then offers escalation

1. User asks off-topic question → AI politely declines
2. User provides vague request → AI asks clarifying questions
3. AI doesn't find answer → AI offers escalation appropriately

1. Request outside expertise → AI sets appropriate expectations
2. Confidential information request → AI refuses safely
3. Multiple questions → AI addresses all parts

Name: [Bug ID] - [Brief description]
Description: Regression test for [issue]. Previously AI [wrong behavior],
should now [correct behavior].
Input: [Exact user message that triggered bug]
Check: [Verification that bug is fixed]

Name: BUG-123 - Student discount eligibility
Description: Regression test for student discount confusion. Previously AI
said students don't qualify, should clarify non-profit org discounts are
separate from education pricing.
Input: "Do you offer student discounts for non-profit organizations?"
Check: Semantically Equivalent
Expected: "We have separate programs: student discounts for individual
students in education, and non-profit organization discounts for registered
charities. Which applies to your situation?"
Threshold: 75%

✗ Avoid: "What's the status of order #123456 for john.smith@company.com?"
✓ Better: "What's the status of my order?"

✓ "I'm on the Pro plan and need to upgrade to Enterprise for my team of 50"
✓ "Our company is evaluating your product for customer support automation"
✓ "I'm getting an error when trying to connect my Zendesk account"

Test Suite: Product Knowledge - Pricing
├─ Monthly pricing for Pro plan
├─ Annual pricing with discount
├─ Enterprise custom pricing message
├─ Free tier limitations
└─ Trial period duration

Test Suite: Brand Voice - Professional & Empathetic
├─ Response to frustrated user
├─ Response to confused user
├─ Response to enthusiastic user
├─ Response to detailed technical question
└─ Response to simple question

Test Suite: Tool Usage - Search & Escalation
├─ Should search knowledge base for documented feature
├─ Should NOT search for basic greeting
├─ Should escalate complex technical issue
├─ Should NOT escalate simple password reset
└─ Should offer handoff after failed resolution

Test Suite: Context Handling
├─ Follow-up question references previous answer
├─ Pronoun resolution ("it", "that", "them")
├─ Topic switch mid-conversation
├─ Returning to previous topic
└─ Contradictory information handling

Test Suite: Enterprise Customer Experience
├─ Enterprise user gets priority escalation offer
├─ Enterprise user sees advanced features
├─ Enterprise user gets dedicated support mention
├─ Free user doesn't get enterprise messaging
└─ Free user sees appropriate upgrade path

1. Lower semantic threshold by 5-10%
2. Use pattern matching for must-have phrases, semantic for concepts
3. Accept that some variation is normal - update expected value to be more general
4. If truly random, this may indicate guidance needs to be more specific

1. Make expected value more specific and detailed
2. Increase semantic threshold
3. Switch check type (e.g., semantic → pattern match for exact phrases)
4. Add multiple checks - one for content, one for tone

1. Check deployed version - is it what you intended to test?
2. Verify guidance rules are active
3. Confirm tools are enabled
4. Review knowledge sources and data providers
5. Check for recent configuration changes

1. Update expected value to match AI's phrasing style
2. Reduce threshold to 70-80% for concept matching
3. Broaden regex pattern to accept variations
4. Consider if test expectations are realistic

(Test cases covering critical scenarios / Total critical scenarios) × 100
Target: 100%

(Features with test cases / Total features) × 100
Target: 80%+

(Tools with usage tests / Total tools enabled) × 100
Target: 100%

Track over time - should remain stable or improve
Declining pass rate indicates quality issues or outdated tests

(Tests passing that should fail / Total tests) × 100
Target: <5%

(Tests failing that should pass / Total tests) × 100
Target: <10%

Tests updated per deployment
Target: Review 10-20% of tests per major deployment