Improved AR Calculation with Additional Grounding Factors
The accuracy component of Automated Resolutions (AR) is now measured by incorporating Processes, Actions, Coaching, and Company Descriptions as additional grounding factors.
What's Changed
Previously, AR evaluation only considered knowledge content to ground the evaluation of accuracy. Now, additional grounding factors are considered.
These enhancements provide a more precise accuracy assessment, reducing cases where conversations were previously judged as inaccurate—particularly when the AI agent leveraged processes, actions, or coaching during a conversation.
How This Impacts AR
AR should generally stay the same or improve
- More grounding factors now contribute to verify accuracy.
- Conversations that previously lacked knowledge references may now be correctly classified as accurate based on new grounding factors.
- Contradictions between knowledge and other elements (processes, actions, coaching, or company descriptions) may result in a conversation being considered inaccurate and therefore unresolved.
Why We're Confident AR Accuracy Has Improved
- False positives, where the LLM predicts a conversation was resolved despite the user explicitly saying in the survey that it wasn't, and false negatives, where the LLM predicts a conversation was not resolved despite the user explicitly saying in the survey that it was, have both decreased as a result of this calculation change.