Health Informatics Journal · Published 2026-07-01 · DOI 10.1177/14604582261470624
Sare Safi, Amir Golmakani, Saeed Rahmani, Nikoo Bayan, Seyed Mohamadmehdi Moshtaghion, Hadis Najafimehr, Haleh Kangari
Purpose To assess the accuracy, comprehensiveness, and reliability of Large Language Model chatbots in answering frequently asked questions about refractive errors. Methods Forty-four questions about refractive errors were posed to four chatbots, including Copilot, Perplexity, Gemini, and ChatGPT. Responses to each question were independently evaluated by three experts using a three-point accuracy scale. The readability of the chatbots’ responses was evaluated using several indices. Similarity was assessed using Sentence-Bidirectional Encoder Representations from Transformers (SBERT). Inter-rater agreement among the graders was evaluated using the Gwet Agreement Coefficient 1 (AC1) statistic. Results The overall agreement among the graders for all chatbot responses was almost perfect (Gwet AC1: 0.87). All chatbots received scores above 85% in every category, and there was no significant difference in chatbot accuracy scores ( P= 0.168 ). The highest mean comprehensiveness score was observed for Perplexity (8.13 ± 0.95, P = 0.028). The similarity scores of the chatbots were very close to each other. All readability scores showed significant differences ( P <0.001) between the chatbots. Conclusion All four chatbots showed comparable accuracy in answering questions about refractive errors. Readability levels of all chatbots exceeded public health recommended thresholds; however, ChatGPT produced relatively more accessible output.
Abstract from DOAJ. Public domain (CC0 1.0).
Read the article at the publisher →
Safi, S., Golmakani, A., Rahmani, S., et al. (2026). Evaluation of the large language models Chatbot’s responses to frequently asked queries on refractive errors. Health Informatics Journal. https://doi.org/10.1177/14604582261470624