Large Language Models Put to the Test on Spondylodiscitis

Spondylodiscitis is a complex spinal infection that's often misunderstood. Patients are turning to large language models (LLMs) for answers, but are these AI-generated responses reliable? A recent study set out to evaluate the quality of information provided by LLMs on this topic.

The researchers created a list of frequently asked questions about spondylodiscitis, using a variety of sources to ensure the questions were relevant and accurate. They then submitted these questions to three popular LLMs - GPT-4, GPT-4o, and Google Gemini - and asked seven experienced spine surgeons to rate the responses.

The results were mixed. While 38.6% of the responses were rated as excellent, a significant number needed clarification due to insufficient information, language issues, or overly detailed answers. The surgeons generally had a positive view of AI-generated patient information, but expressed concerns about reliability and direct communication with patients.

The study found that the quality of responses varied depending on the question. For example, questions about complications received high ratings, while those related to treatment and prognosis were rated lower. The researchers concluded that LLMs may be useful for patient education, but only with close supervision from clinicians.

The study also highlighted the need for more advanced, specialized models that can provide accurate and reliable information on specific medical topics. By improving the quality of communication between clinicians and patients, LLMs could become a valuable tool in healthcare. However, more research is needed to achieve this goal.

The surgeons' ratings revealed some interesting insights. They tended to agree on the quality of the responses, with an interrater reliability of 0.7. The study's findings suggest that LLMs have the potential to support patient education, but their limitations must be acknowledged and addressed.