BRIDGE: benchmarking large language models for understanding real-world clinical practice texts
Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Med. 29 , 1930–1940 (2023). Article CAS PubMed Google Scholar Kaplan, J. et al. Scaling laws for neural language models. Preprint at https://arxiv.org/abs/2001.08361 (2020). Zhang, S. et al. Instruction tuning for large language models: a survey. ACM Comput. Surv. 58 , 169 (2026). Google Scholar Min, B. et al. Recent advances in natural language processing via large pre-trained language models: a survey. ACM Comput. Surv. 56 , 30 (2024). Article Google Scholar Qin, C. et al. Is ChatGPT a general-purpose natural language processing task solver? In Proc. 2023 Conference on Empirical Methods in Natural Language Processing (eds Bouamor, H et al.) 1339–1384 (Association for Computational Linguistics, 2023); https://doi.org/10.18653/v1/2023.emnlp-main.85 Huang, J. & Chang, K. C.-C. Towards reasoning in large language models: a survey. In Findings of the Association for Computational Linguistics (eds Rogers, A. et al.) 1049–1065 (Association for Computational Linguistics, 2023); https://doi.org/10.18653/v1/2023.findings-acl.67 Tang, L. et al. Evaluating large language models on medical evidence summarization. npj Digit. Med. 6 , 158 (2023). Article PubMed PubMed Central Google Scholar Van Veen, D. et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med. 30 , 1134–1142 (2024). Article PubMed PubMed Central Google Scholar Jiang, L. Y. et al. Health system-scale language models are all-purpose prediction engines. Nature 619 , 357–362 (2023). Article CAS PubMed PubMed Central Google Scholar Goh, E. et al. GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nat. Med. 31 , 1233–1238 (2025). Article CAS PubMed PubMed Central Google Scholar Duan, Z. et al. Multi-center benchmarking of large language models for clinical decision support in lung cancer screening. Cell Rep. Med. 6 , 102465 (2025). Article CAS PubMed PubMed Central Google Scholar Jeblick, K. et al. ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports. Eur. Radiol. 34 , 2817–2825 (2024). Article PubMed Google Scholar Goodman, R. S. et al. Accuracy and reliability of chatbot responses to physician questions. JAMA Netw. Open 6 , e2336483 (2023). Article PubMed PubMed Central Google Scholar Sheng, B. et al. Large language models for diabetes care: potentials and prospects. Sci. Bull. 69 , 583–588 (2024). Article Google Scholar Zaretsky, J. et al. Generative artificial intelligence to transform inpatient discharge summaries to patient-friendly language and format. JAMA Netw. Open 7 , e240357 (2024). Article PubMed PubMed Central Google Scholar Wang, X. et al. ChatGPT: promise and challenges for deployment in low- and middle-income countries. Lancet Reg. Health West. Pac. 41 , 100905 (2023). PubMed PubMed Central Google Scholar Ferryman, K., Mackintosh, M. & Ghassemi, M. Considering biased data as informative artifacts in AI-assisted health care. N. Engl. J. Med. 389 , 833–838 (2023). Article PubMed Google Scholar McCoy, L. G., Manrai, A. K. & Rodman, A. Large language models and the degradation of the medical record. N. Engl. J. Med. 391 , 1561–1564 (2024). Article PubMed Google Scholar Takita, H. et al. A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians. npj Digit. Med. 8 , 175 (2025). Article PubMed PubMed Central Google Scholar Raji, I. D., Daneshjou, R. & Alsentzer, E. It’s time to bench the medical exam benchmark. NEJM AI 2 , AIe2401235 (2025). Article Google Scholar Jin, D. et al. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl. Sci. 11 , 6421 (2021). Article CAS Google Scholar Liévin, V., Hother, C. E., Motzfeldt, A. G. & Winther, O. Can large language models reason about medical questions?. Patterns 5 , 1–12 (2024). Article Google Scholar Jin, Q., Dhingra, B., Liu, Z., Cohen, W. & Lu, X. PubMedQA: a dataset for biomedical research question answering. In Proc. 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (eds Inui, K. et al.) 2567–2577 (Association for Computational Linguistics, 2019); https://doi.org/10.18653/v1/D19-1259 Chen, Q. et al. Benchmarking large language models for biomedical natural language processing applications and recommendations. Nat. Commun. 16 , 3280 (2025). Article CAS PubMed PubMed Central Google Scholar Kreimeyer, K. et al. Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review. J. Biomed. Inform. 73 , 14–29 (2017). Article PubMed PubMed Central Google Scholar OpenAI. GPT-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 (2023). Singhal, K. et al. Toward expert-level medical question answering with large language models. Nat. Med. 31 , 943–950 (2025). Article CAS PubMed PubMed Central Google Scholar Soroush, A. et al. Large language models are poor medical coders—benchmarking of medical code querying. NEJM AI 1 , AIdbp2300040 (2024). Article Google Scholar Barile, J. et al. Diagnostic accuracy of a large language model in pediatric case studies. JAMA Pediatr. 178 , 313–315 (2024). Article PubMed PubMed Central Google Scholar Arora, R. K. et al. HealthBench: evaluating large language models towards improved human health. Preprint at https://arxiv.org/abs/2505.08775 (2025). Goh, E. et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Netw. Open 7 , e2440969 (2024). Article PubMed PubMed Central Google Scholar Nayak, A. et al. Comparison of history of present illness summaries generated by a chatbot and senior internal medicine residents. JAMA Int. Med. 183 , 1026–1027 (2023). Article Google Scholar Sandmann, S. et al. Benchmark evaluation of DeepSeek large language models in clinical decision-making. Nat. Med. 31 , 2546–2549 (2025). Article CAS PubMed PubMed Central Google Scholar Tordjman, M. et al. Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoning. Nat. Med. 31 , 2550–2555 (2025). Article CAS PubMed Google Scholar Bedi, S. et al. Holistic evaluation of large language models for medical tasks with MedHELM. Nat. Med. 32 , 943–951 (2026). Article CAS PubMed Google Scholar Wu, C. et al. Towards evaluating and building versatile large language models for medicine. npj Digit. Med. 8 , 58 (2025). Article CAS PubMed PubMed Central Google Scholar Qiu, P. et al. Towards building multilingual language model for medicine. Nat. Commun. 15 , 8384 (2024). Article CAS PubMed PubMed Central Google Scholar Zack, T. et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit. Health 6 , e12–e22 (2024). Article CAS PubMed Google Scholar Liu, X. et al. Uncovering language disparity of ChatGPT on retinal vascular disease classification: cross-sectional study. J. Med. Internet Res. 26 , e51926 (2024). Article PubMed PubMed Central Google Scholar Kim, J., Cai, Z. R., Chen, M. L., Simard, J. F. & Linos, E. Assessing biases in medical decisions via clinician and AI chatbot responses to patient vignettes. JAMA Netw. Open 6 , e2338050 (2023). Article PubMed PubMed Central Google Scholar Zeng, Q. et al. GreenPLM: cross-lingual transfer of monolingual pre-trained language models at almost no cost. In Proc. 32nd International Joint Conference on Artificial Intelligence (ed. Elkind, E.) 6290–6298 (International Joint Conferences on Artificial Intelligence Organization, 2023); https://doi.org/10.24963/ijcai.2023/698 Hofmann, V., Kalluri, P. R., Jurafsky, D. & King, S. AI generates covertly racist decisions about people based on their dialect. Nature 633 , 147–154 (2024). Article CAS PubMed PubMed Central Google Scholar Ong, J. C. L. et al. Ethical and regulatory challenges of large language models in medicine. Lancet Digit. Health 6 , e428–e432 (2024). Article CAS PubMed Google Scholar Ji, Y. et al. Mitigating the risk of health inequity exacerbated by large language models. npj Digit. Med. 8 , 246 (2025). Article PubMed PubMed Central Google Scholar Chang, Y. et al. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol. 15 , 39 (2024). Article Google Scholar Singhal, K. et al. Large language models encode clinical knowledge. Nature 620 , 172–180 (2023). Article CAS PubMed PubMed Central Google Scholar Xie, Q. et al. Medical foundation large language models for comprehensive text analysis and beyond. npj Digit. Med. 8 , 141–150 (2025). Article PubMed PubMed Central Google Scholar Liu, X. et al. A generalist medical language model for disease diagnosis assistance. Nat. Med. 31 , 932–942 (2025). Article CAS PubMed Google Scholar Chiang, W.-L. et al. Chatbot arena: an open platform for evaluating LLMs by human preference. In 41st International Conference on Machine Learning (eds Salakhutdinov, R. et al.) 8359–8388 (JMLR, 2024). Minssen, T., Vayena, E. & Cohen, I. G. The challenges for regulating medical use of ChatGPT and other large language models. J. Am. Med. Assoc. 330 , 315–316 (2023). Article Google Scholar Bedi, S. et al. Testing and evaluation of health care applications of large language models: a systematic review. J. Am. Med. Assoc. 333 , 319–328 (2025). Article Google Scholar Tang, Y.-D., Dong, E.-D. & Gao, W. LLMs in medicine: the need for advanced evaluation systems for disruptive technologies. Innovation 5 , 100622 (2024). PubMed PubMed Central Google Scholar Wu, J. et al. Clinical text datasets for medical artificial intelligence and large language models—a systematic review. NEJM AI 1 , AIra2400012 (2024). DeepSeek-AI et al. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. Preprint at https://arxiv.org/abs/2501.12948 (2025). OpenAI et al. GPT-4o system card. Preprint at https://arxiv.org/abs/2410.21276 (2024). Gemini Team et al. Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. Preprint at https://arxiv.org/abs/2403.05530 (2024). Comanici, G. et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Preprint at https://arxiv.org/abs/2507.062611 (2025). The Llama 4 herd: the beginning of a new era of natively multimodal AI innovation. Meta AI https://ai.meta.com/blog/llama-4-multimodal-intelligence/ (2025). Yang, A. et al. Qwen3 technical report. Preprint at https://arxiv.org/abs/2505.09388 (2025). Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33 , 1877–1901 (2020). Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. In Proc. 36th International Conference on Neural Information Processing Systems (eds Koyejo, S. et al.) 24824–24837 (Curran Associates Inc., 2022). Chen J. et al. Towards medical complex reasoning with LLMs through medical verifiable problems. In Findings of the Association for Computational Linguistics 14552–14573 (Association for Computational Linguistics, 2025). Sellergren, A. et al. MedGemma technical report. Preprint at https://arxiv.org/abs/2507.05201 (2025). Baichuan-M2 Team et al. Baichuan-M2: scaling medical capability with large verifier system. Preprint at https://arxiv.org/abs/2509.02208 (2025). Jiang, A. Q. et al. Mistral 7B. Preprint at https://arxiv.org/abs/2310.06825 (2023). Qwen Team et al. Qwen2.5 technical report. Preprint at https://arxiv.org/abs/2412.15115 (2025). Gemma Team et al. Gemma 3 technical report. Preprint at https://arxiv.org/abs/2503.19786 (2025). Grattafiori, A. et al. The Llama 3 herd of models. Preprint at https://arxiv.org/abs/2407.21783 (2024). Nexusflow/Athene-V2-Chat. Hugging Face https://huggingface.co/Nexusflow/Athene-V2-Chat (2024). OpenAI et al. gpt-oss-120b & gpt-oss-20b model card. Preprint at https://arxiv.org/abs/2508.10925 (2025). Romanov, A. & Shivade, C. Lessons from natural language inference in the clinical domain. In Proc. 2018 Conference on Empirical Methods in Natural Language Processing (eds Riloff, E., Chiang, D., Hockenmaier, J. & Tsujii, J.) 1586–1596 (Association for Computational Linguistics, 2018); https://doi.org/10.18653/v1/D18-1187 Blinov, P. et al. RuMedNLI: a Russian natural language inference dataset for the clinical domain. PhysioNet https://doi.org/10.13026/GXZD-CF80 (2022). Percha, B. Modern clinical text mining: a guide and review. Annu. Rev. Biomed. Data Sci. 4 , 165–187 (2021). Article PubMed Google Scholar Huang, K. et al. A foundation model for clinician-centered drug repurposing. Nat. Med. 30 , 3601–3613 (2024). Article CAS PubMed PubMed Central Google Scholar Savage, T., Nayak, A., Gallo, R., Rangan, E. & Chen, J. H. Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine. npj Digit. Med. 7 , 20 (2024). Article PubMed PubMed Central Google Scholar Kasai, J., Kasai, Y., Sakaguchi, K., Yamada, Y. & Radev, D. Evaluating GPT-4 and ChatGPT on Japanese medical licensing examinations. Preprint at https://doi.org/10.48550/arXiv.2303.18027 (2023). Chen, H. et al. Large language models and global health equity: a roadmap for equitable adoption in LMICs. Lancet Reg. Health West. Pac. 63 , 1–8 (2025). Google Scholar Hegselmann, S. et al. Large language models are powerful electronic health record encoders. Preprint at https://doi.org/10.48550/arXiv.2502.17403 (2025). Klang, E. et al. Assessing retrieval-augmented large language models for medical coding. NEJM AI 2 , AIcs2401161 (2025). Article Google Scholar Zakka, C. et al. Almanac—retrieval-augmented language models for clinical medicine. NEJM AI 1 , AIoa2300068 (2024). Article Google Scholar Wu, J., Wu, X., Zheng, Y. & Yang, J. Clinical pathway-aware large language models for reliable and transparent medical dialogue. J. Biomed. Inform. 172 , 104942 (2025). Article PubMed Google Scholar Pfohl, S. R. et al. A toolbox for surfacing health equity harms and biases in large language models. Nat. Med. 30 , 3590–3600 (2024). Article CAS PubMed PubMed Central Google Scholar DeepSeek-AI et al. DeepSeek LLM: scaling open-source language models with longtermism. Preprint at https://doi.org/10.48550/arXiv.2401.02954 (2024). Buckley, T. A., Crowe, B., Abdulnour, R.-E. E., Rodman, A. & Manrai, A. K. Comparison of frontier open-source and proprietary large language models for complex diagnoses. JAMA Health Forum 6 , e250040 (2025). Article PubMed PubMed Central Google Scholar Yang, X. et al. A large language model for electronic health records. npj Digit. Med. 5 , 194 (2022). Article PubMed PubMed Central Google Scholar Hager, P. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. Med. 30 , 2613–2622 (2024). Article CAS PubMed PubMed Central Google Scholar Wornow, M. et al. The shaky foundations of large language models and foundation models for electronic health records. npj Digit. Med. 6 , 135 (2023). Article PubMed PubMed Central Google Scholar Meshkin, H. et al. Harnessing large language models’ zero-shot and few-shot learning capabilities for regulatory research. Brief. Bioinform. 25 , bbae354 (2024). Article CAS PubMed PubMed Central Google Scholar Agrawal, M., Hegselmann, S., Lang, H., Kim, Y. & Sontag, D. Large language models are few-shot clinical information extractors. In Proc. 2022 Conference on Empirical Methods in Natural Language Processing (eds Goldberg, Y. et al.) 1998–2022 (Association for Computational Linguistics, 2022). Qin, L. et al. A survey of multilingual large language models. Patterns 6 , 101118 (2025). Article PubMed PubMed Central Google Scholar Liu, F. et al. A multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis. npj Digit. Med. 8 , 86 (2025). Article PubMed PubMed Central Google Scholar Das, B. C., Amini, M. H. & Wu, Y. Security and privacy challenges of large language models: a survey. ACM Comput. Surv. 57 , 1–39 (2025). Google Scholar Lehman, E. et al. Do we still need clinical language models? In Proc. Conference on Health, Inference, and Learning (eds Mortazavi, B. J. et al.) 578–597 (PMLR, 2023). Wang, X. et al. Self-consistency improves chain of thought reasoning in language models. Preprint at https://doi.org/10.48550/arXiv.2203.11171 (2023). Wu, J., Wu, X. & Yang, J. Guiding clinical reasoning with large language models via knowledge seeds. In Proc. Thirty-Third International Joint Conference on Artificial Intelligence (ed. Larson, K.) 7491–7499 (International Joint Conferences on Artificial Intelligence Organization, 2024). Wu, J. et al. Large language models leverage external knowledge to extend clinical insight beyond language boundaries. J. Am. Med. Inform. Assoc. 31 , 2054–2064 (2024). Article PubMed PubMed Central Google Scholar Qwen Team. QwQ-32B: embracing the power of reinforcement learning. Qwen https://qwenlm.github.io/blog/qwq-32b/ (2025). Shah, N. H., Entwistle, D. & Pfeffer, M. A. Creation and adoption of large language models in medicine. J. Am. Med. Assoc. 330 , 866–869 (2023). Article Google Scholar Karargyris, A. et al. Federated benchmarking of medical artificial intelligence with MedPerf. Nat. Mach. Intell. 5 , 799–810 (2023). Article PubMed PubMed Central Google Scholar Rodman, A., Zwaan, L., Olson, A. & Manrai, A. K. When it comes to benchmarks, humans are the only way. NEJM AI 2 , AIe2500143 (2025). Article Google Scholar Cui, H. et al. TIMER: temporal instruction modeling and evaluation for longitudinal clinical records. npj Digit. Med. 8 , 577–585 (2025). Article PubMed PubMed Central Google Scholar Gallifant, J. et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat. Med. 31 , 60–69 (2025). Article CAS PubMed PubMed Central Google Scholar Gu, B., Desai, R. J., Lin, K. J. & Yang, J. Probabilistic medical predictions of large language models. npj Digit. Med. 7 , 367 (2024). Article PubMed PubMed Central Google Scholar National NLP clinical challenges (n2c2). Harvard Medical School https://n2c2.dbmi.hms.harvard.edu/home (2026). CLEF eHealth lab series. CLEF eHealth https://clefehealth.imag.fr/clefehealth.imag.fr/index.html (2024). Goldberger, A. L. et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 101 , e215–e220 (2000). Article CAS PubMed Google Scholar Hugging face–the AI community building the future. Hugging Face https://huggingface.co/ (2026). Kwon, W. et al. Efficient memory management for large language model serving with PagedAttention. In Proc. 29th Symposium on Operating Systems Principles 611–626 (Association for Computing Machinery, 2023); https://doi.org/10.1145/3600006.3613165 Lin, C.-Y. ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out 74–81 (Association for Computational Linguistics, 2004). Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. BLEU: a method for automatic evaluation of machine translation. In Proc. 40th Annual Meeting on Association for Computational Linguistics (eds Isabelle, P. et al.) 311–318 (Association for Computational Linguistics, 2001); https://doi.org/10.3115/1073083.1073135 Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: evaluating text generation with BERT. In International Conference on Learning Representations (2020). Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12 , 2825–2830 (2011). Google Scholar Bird, S. NLTK: the natural language toolkit. In Proc. COLING/ACL 2006 Interactive Presentation Sessions 69–72 (Association for Computational Linguistics, 2006); https://doi.org/10.3115/1225403.1225421 Google Research. Python ROUGE implementation. GitHub https://github.com/google-research/google-research/tree/master/rouge (2026). Xu, R., Wang, Z., Fan, R.-Z. & Liu, P. Benchmarking benchmark leakage in large language models. Preprint at https://arxiv.org/abs/2404.18824 (2024).
Source: Nature