Future Doctor unveils clinical safety‑effectiveness benchmark; MedGPT leads comparative evaluation

Future Doctor has introduced a purpose-built evaluation framework intended to shift medical AI assessment from narrow accuracy checks to clinically meaningful safety and effectiveness tests. The CSEDB framework bundles 30 indicators—split into 17 safety and 13 effectiveness metrics—and exercises models using 2,069 open‑ended clinical questions spanning 26 specialties, with development input from 32 clinicians across 23 core specialties at China’s major hospitals. The paper, published in npj Digital Medicine, argues that standard testing regimes miss safety‑critical failure modes such as missed acute symptoms, contraindicated advice, and poor prioritization in multi‑morbidity cases. In a head‑to‑head assessment under the CSEDB protocol, Future Doctor’s MedGPT scored highest on combined, safety, and effectiveness dimensions compared with a set of prominent generalist large language models. That finding highlights a recurring tradeoff: systems optimized for broad capability do not automatically deliver the safety profile required for frontline clinical work. If the CSEDB gains traction, procurement and regulatory conversations will likely shift toward demonstrable, repeatable safety metrics rather than benchmarked knowledge recall alone. Hospitals and regulators would gain a structured way to demand performance under clinical constraints, which could lengthen vendor evaluation cycles but reduce downstream patient risk. For AI developers, the benchmark creates a quantifiable target for safety engineering and a potential market advantage for teams that prioritize constrained clinical behavior. The involvement of multiple specialty experts and a large corpus of open‑ended items strengthens the benchmark’s ecological validity compared with typical multiple‑choice style evaluations. However, adoption will depend on independent validation, transparency of test corpora, and whether results generalize across diverse health systems and languages. In short, CSEDB reframes evaluation from “can an AI answer?” to “can it act safely and usefully in clinical workflows,” and MedGPT’s top ranking positions Future Doctor favorably—if the framework is accepted by purchasers and regulators.