Letter to the editor to comment on “Diagnosis, treatment, and prevention of ankle sprains: Comparing free chatbot recommendations with clinical guidelines”

To the Editor

We read with interest the recent paper by Roch et al. comparing chatbot-generated advice for lateral ankle sprains to a clinical standard . This work provides a valuable assessment of AI in a common medical context, employing a notably thorough scoring methodology. A key methodological consideration warrants attention. The study uses the 2018 Dutch guideline as its reference standard . While important, musculoskeletal medicine advances rapidly. A six-year-old benchmark may evaluate chatbots against outdated principles. This chronological alignment is crucial for assessing AI trained on continually updated data. Chatbot responses might incorporate valid post-2018 evidence, potentially scored incorrectly against older guidelines. For example, newer studies have refined the value of prompt functional support and specific rehabilitation techniques, possibly explaining divergences between AI suggestions and the 2018 guideline ^, . Dependence on a potentially outdated benchmark complicates interpretation. A response deemed “incomplete” might instead reflect recent advancements. That Perplexity didn’t reference the 2018 guide may indicate its use of newer literature rather than a limitation. This highlights a fundamental AI assessment challenge: perceived accuracy depends on the reference standard. An outdated reference may misrepresent an AI’s capabilities. We suggest contextualizing findings with evidence published since the reference benchmark. Future studies might utilize contemporary consensus, incorporating recent systematic reviews and international guidelines to evaluate AI performance ^, . This would assess both historical fidelity and alignment with current evidence-based standards, enhancing the clinical relevance of this important work.

The authors (Dr. Friederike Roch et al.) of the aforementioned published paper (Diagnosis, Treatment, and Prevention of Ankle Sprains: Comparing Free Chatbot Recommendations with Clinical Guidelines), Foot and Ankle Surgery, Volume 31, Issue 4, June 2025, Pages 329–351) that this Letter to the editor pertains to were contacted by the journal but they declined the invitation to submit a reply letter.