Inter-Model Agreement of ChatGPT, Claude, and Grok for IBO-Based Lateral Cephalometric Measurements: An Exploratory Comparative Study
Keywords:
Artificial intelligence, Cephalometry, Large language models, Multimodal artificial intelligence, Orthodontics, ReliabilityAbstract
Background: General-purpose multimodal artificial intelligence (AI) systems can analyze radiographic images, but their numerical consistency for quantitative orthodontic measurements is unknown. Objective: To compare the inter-model agreement of ChatGPT, Claude, and Grok when estimating standardized Italian Board of Orthodontics (IBO) cephalometric measurements on the same lateral cephalograms. Methods: Twenty pre-existing lateral cephalograms from 20 patients aged 10-60 years were analyzed by ChatGPT, Claude, and Grok with the same standardized prompt. Twelve IBO parameters were requested. Three linear measurements were excluded because metric calibration was not validated, leaving nine angular parameters. Model-specific descriptive statistics, pairwise mean absolute differences (MAD), root mean square error, and two-way absolute-agreement single-measure intraclass correlation coefficients (ICC) were calculated. Friedman tests with Holm correction assessed systematic differences between models. Results: Complete-case sample size ranged from 16 to 20 across angular parameters. ICCs were poor for all nine measurements (0.03-0.26). Pooled pairwise MADs were 2.14 degrees for ChatGPT-Claude, 2.71 degrees for ChatGPT-Grok, and 2.59 degrees for Claude-Grok. No angular parameter showed a statistically significant systematic difference among models after Holm correction. Conclusions: ChatGPT, Claude, and Grok produced broadly similar group-level cephalometric estimates but poor case-level numerical agreement. General-purpose multimodal AI systems should not be considered interchangeable cephalometric measurement tools. Validation against an expert human reference standard is required before clinical use.
Downloads
References
1. Ongkosuwito EM, Katsaros C, van 't Hof MA, Bodegom JC, Kuijpers-Jagtman AM. The reproducibility of cephalometric measurements: a comparison of analogue and digital methods. Eur J Orthod. 2002;24(6):655-665. doi: 10.1093/ejo/24.6.655.
2. Narkhede S, Rao P, Sawant V, Sachdev SS, Arora S, Pawar AM, et al. Digital versus manual tracing in cephalometric analysis: A systematic review and meta-analysis. J Pers Med. 2024;14(6):566. doi: 10.3390/jpm14060566.
3. Schwendicke F, Samek W, Krois J. Artificial intelligence in dentistry: Chances and challenges. J Dent Res. 2020;99(7):769-774. doi: 10.1177/0022034520915714.
4. Khanagar SB, Al-Ehaideb A, Maganur PC, Vishwanathaiah S, Patil S, Baeshen HA, et al. Developments, application, and performance of artificial intelligence in dentistry - A systematic review. J Dent Sci. 2021;16(1):508-522. doi: 10.1016/j.jds.2020.06.019.
5. Subramanian AK, Chen Y, Almalki A, Sivamurthy G, Kafle D. Cephalometric analysis in orthodontics using artificial intelligence-A comprehensive review. Biomed Res Int. 2022;2022:1880113. doi: 10.1155/2022/1880113.
6. Kiełczykowski M, Kamiński K, Perkowski K, Zadurska M, Czochrowska E. Application of artificial intelligence (AI) in a cephalometric analysis: A narrative review. Diagnostics (Basel). 2023;13(16):2640. doi: 10.3390/diagnostics13162640.
7. Schwendicke F, Chaurasia A, Arsiwala L, Lee JH, Elhennawy K, Jost-Brinkmann PG, et al. Deep learning for cephalometric landmark detection: systematic review and meta-analysis. Clin Oral Investig. 2021;25(7):4299-4309. doi: 10.1007/s00784-021-03990-w.
8. Mesquita GQTB, Vieira WA, Vidigal MTC, Travencolo BAN, Beaini TL, Spin-Neto R, et al. Artificial intelligence for detecting cephalometric landmarks: A systematic review and meta-analysis. J Digit Imaging. 2023;36(3):1158-1179. doi: 10.1007/s10278-022-00766-w.
9. Polizzi A, Leonardi R. Automatic cephalometric landmark identification with artificial intelligence: An umbrella review of systematic reviews. J Dent. 2024;146:105056. doi: 10.1016/j.jdent.2024.105056.
10. Hendrickx J, Gracea RS, Vanheers M, Winderickx N, Preda F, Shujaat S, et al. Can artificial intelligence-driven cephalometric analysis replace manual tracing? A systematic review and meta-analysis. Eur J Orthod. 2024;46(4):cjae029. doi:10.1093/ejo/cjae029.
11. Hwang HW, Park JH, Moon JH, Yu Y, Kim H, Her SB, et al. Automated identification of cephalometric landmarks: Part 2-Might it be better than human? Angle Orthod. 2020;90(1):69-76. doi: 10.2319/022019-129.1.
12. Hwang HW, Moon JH, Kim MG, Donatelli RE, Lee SJ. Evaluation of automated cephalometric analysis based on the latest deep learning method. Angle Orthod. 2021;91(3):329-335. doi: 10.2319/021220-100.1.
13. Kim YH, Lee C, Ha EG, Choi YJ, Han SS. A fully deep learning model for the automatic identification of cephalometric landmarks. Imaging Sci Dent. 2021;51(3):299-306. doi: 10.5624/isd.20210077.
14. Jeon S, Lee KC. Comparison of cephalometric measurements between conventional and automatic cephalometric analysis using convolutional neural network. Prog Orthod. 2021;22(1):14. doi: 10.1186/s40510-021-00358-4.
15. Meriç P, Naoumova J. Web-based fully automated cephalometric analysis: Comparisons between App-aided, computerized, and manual tracings. Turk J Orthod. 2020;33(3):142-149. doi: 10.5152/TurkJOrthod.2020.20062.
16. Mahto RK, Kafle D, Giri A, Luintel S, Karki A. Evaluation of fully automated cephalometric measurements obtained from web-based artificial intelligence driven platform. BMC Oral Health. 2022;22(1):132. doi: 10.1186/s12903-022-02170-w.
17. Çoban G, Öztürk T, Hashimli N, Yağci A. Comparison between cephalometric measurements using digital manual and web-based artificial intelligence cephalometric tracing software. Dental Press J Orthod. 2022;27(4):e222112. doi: 10.1590/2177-6709.27.4.e222112.oar.
18. Duran GS, Gökmen Ş, Topsakal KG, Görgülü S. Evaluation of the accuracy of fully automatic cephalometric analysis software with artificial intelligence algorithm. Orthod Craniofac Res. 2023;26(3):481-490. doi: 10.1111/ocr.12633.
19. Bao H, Zhang K, Yu C, Li H, Cao D, Shu H, et al. Evaluating the accuracy of automated cephalometric analysis based on artificial intelligence. BMC Oral Health. 2023;23(1):191. doi: 10.1186/s12903-023-02881-8.
20. Indermun S, Shaik S, Nyirenda C, Johannes K, Mulder R. Human examination and artificial intelligence in cephalometric landmark detection-is AI ready to take over? Dentomaxillofac Radiol. 2023;52(6):20220362. doi: 10.1259/dmfr.20220362.
21. Silva TP, Pinheiro MCR, Freitas DQ, Gaêta-Araujo H, Oliveira-Santos C. Assessment of accuracy and reproducibility of cephalometric identification performed by 2 artificial intelligence-driven tracing applications and human examiners. Oral Surg Oral Med Oral Pathol Oral Radiol. 2024;137(4):431-440. doi: 10.1016/j.oooo.2024.01.011.
22. Bor S, Çınarsoy Ciğerim S, Kotan S. Comparison of AI-assisted cephalometric analysis and orthodontist-performed digital tracing analysis. Prog Orthod. 2024;25(1):41. doi: 10.1186/s40510-024-00539-x.
23. Kunz F, Stellzig-Eisenhauer A, Widmaier LM, Zeman F, Boldt J. Assessment of the quality of different commercial providers using artificial intelligence for automated cephalometric analysis compared to human orthodontic experts. J Orofac Orthop. 2025;86(3):145-160. doi: 10.1007/s00056-023-00491-1.
24. Alhamdani NA, Khoshnaw BM, Omer ZQ. Assessing the accuracy and reliability of fully AI-driven cephalometric analysis in comparison to digital manual methods. Al-Rafidain J Med Sci. 2025;8(2):71-75. doi: 10.54133/ajms.v8i2.1839.
25. Azari-Mehr A, Bisbal-Puchades A, Marqués-Martínez L, Carmona-Santamaria M, García-Miralles E, Aura-Tormos JI, et al. Clinical reliability of AI-based cephalometric analysis using WebCeph: A comparative agreement study. J Clin Med. 2026;15(11):4155. doi: 10.3390/jcm15114155.
26. Ribas-Sabartés J, Sánchez-Molins M, d'Oliveira NG. The accuracy of algorithms used by artificial intelligence in cephalometric points detection: A systematic review. Bioengineering (Basel). 2024;11(12):1286. doi: 10.3390/bioengineering11121286.
27. Lin YM, Auta A, Brown J, Cure R, Ireland R. The accuracy of artificial intelligence in identifying cephalometric landmarks: A scoping review. Int Orthod. 2026;24(4):101205. doi: 10.1016/j.ortho.2026.101205.
28. Nam Y, Kim DY, Kyung S, Seo J, Song JM, Kwon J, et al. Multimodal large language models in medical imaging: Current state and future directions. Korean J Radiol. 2025;26(10):900-923. doi: 10.3348/kjr.2025.0599.
29. Zhang A, Zhao E, Wang R, Zhang X, Wang J, Chen E. Multimodal large language models for medical image diagnosis: Challenges and opportunities. J Biomed Inform. 2025;169:104895. doi: 10.1016/j.jbi.2025.104895.
30. von der Stück MS, Vuskov R, Westfechtel S, Siepmann R, Kuhl C, Truhn D, et al. Visual large language models in radiology: A systematic multimodel evaluation of diagnostic accuracy and hallucinations. Life (Basel). 2026;16(1):66. doi: 10.3390/life16010066.
31. Sozer A, Sahin MC, Sozer B, Erol G, Tufek OY, Nernekli K, et al. Do LLMs have 'the eye' for MRI? Evaluating GPT-4o, Grok, and Gemini on brain MRI performance: First evaluation of Grok in medical imaging and a comparative analysis. Diagnostics (Basel). 2025;15(11):1320. doi: 10.3390/diagnostics15111320.
32. Hatipoğlu Ö, Baytar M, Hatipoğlu FP. Diagnostic and localization performance of multimodal large language models in the interpretation of dental periapical radiographs. BMC Oral Health. 2026;26(1):1428. doi: 10.1186/s12903-026-08870-x.
33. Italian Board of Orthodontics. Linee guida della sessione d'esami I.B.O.: Certificato d'Eccellenza in Ortodonzia. 2024-2025.
34. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155-163. doi: 10.1016/j.jcm.2016.02.012.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Al-Rafidain Journal of Medical Sciences ( ISSN 2789-3219 )
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Published by Al-Rafidain University College. This is an open access journal issued under the CC BY-NC-SA 4.0 license (https://creativecommons.org/licenses/by-nc-sa/4.0/).
