Salesforce Decode
Salesforcedecode
Back to questions
AgentforceAdvancedab-testingexperimentationoptimizationprompts

Run A/B experiments on agent prompts and topics with statistical rigor

Real World Scenario

Product wants to test two refund policy phrasings to maximize containment without hurting CSAT. Ad-hoc toggling caused inconsistent customer experiences last quarter.

Expected Answer

• Define experiment units—session or customer—with consistent assignment for duration of interaction • Power analysis determining sample size before declaring winner on containment or CSAT • Isolate one variable per experiment: prompt wording OR topic threshold, not both • Instrument variant ID in session metadata for downstream analytics joins • Set automatic stop rules for guardrail breaches—CSAT drop, error spike, compliance flag • Run experiments per channel; chat and email behavior differs materially • Document winner promotion through standard CI/CD with eval regression gate

Follow-Up Questions & Answers

Click to expand — each follow-up includes a direct, interview-ready answer

Direct answer: Define experiment units—session or customer—with consistent assignment for duration of interaction Also consider: Power analysis determining sample size before declaring winner on containment or CSAT In practice: Isolate one variable per experiment: prompt wording OR topic threshold, not both Optimize for scale and operational observability.

Architect Perspective

Agent optimization without experimentation discipline is guesswork. Treat prompts like product features—hypothesis, metric, sample size, ship or revert.