Duke Researchers Propose New Framework for Evaluating AI Scribes Amid Surge in Venture Capital Funding

In a significant development for the healthcare technology sector, researchers at Duke University have proposed a novel framework for evaluating artificial intelligence (AI) scribing tools. This comes at a time when venture capital is pouring into AI scribe companies, with several firms announcing substantial funding rounds in recent months.

The SCRIBE Framework: A New Standard for AI Evaluation

The Duke researchers have introduced SCRIBE, a comprehensive evaluation and governance method designed to assess the performance of AI scribes in clinical settings. This framework combines human review with technological evaluation, addressing the lack of standardized assessment methods for these increasingly popular tools.

SCRIBE aims to provide healthcare delivery organizations with a more efficient way to compare commercial AI scribe tools and evaluate their performance over time. The framework incorporates multiple evaluation techniques, including:

• Human review of AI-generated transcripts and SOAP notes
• Automated evaluations such as ROUGE, Word Error Rate, and F1 scores
• Large language model assessments to reduce manual labor
• Simulation reviews to test edge case scenarios

Dr. Michael Pencina, one of the researchers involved in the study, emphasized the importance of maintaining human oversight in the evaluation process. "The design is guided by the principle that no single method can comprehensively capture all performance dimensions," he stated.

Venture Capital Fuels AI Scribe Market Growth

The introduction of SCRIBE comes amid a flurry of investment activity in the AI scribe sector. Several companies have recently announced significant funding rounds:

• Nabla: $70 million Series C
• Abridge: $300 million Series E
• Commure: $200 million raise
• Ambience and Suki: $70 million each in 2024

This influx of capital underscores the growing interest in AI technologies aimed at reducing healthcare staff burnout and improving clinical workflow efficiency.

Implications for Healthcare Delivery and Future Research

The Duke study, published in NPJ Digital Medicine, tested the SCRIBE framework using an in-house developed ambient dictation scribe (ADS) tool across 40 clinical visits. The researchers found that their ADS tool performed well across multiple metrics, particularly in clarity, completeness, and relevance.

Looking ahead, the Duke team plans to conduct further research, including:

• Testing the framework on commercially available ADS tools
• Performing a multisite study with several ADS products to systematically compare them
• Assessing the impact of AI scribes on patient care

The researchers suggest that health systems could potentially use SCRIBE to conduct head-to-head comparisons of the more than 50 AI scribe vendors currently in the market.

As AI scribes continue to gain traction in healthcare settings, the SCRIBE framework represents a significant step towards standardizing evaluation methods and ensuring the responsible deployment of these technologies in clinical practice.