Diagnostic accuracy of generative large language artificial intelligence models for the assessment of dental crowding

Abstract

Background: Generative artificial intelligence (AI) models have shown potential for addressing text-based dental enquiries and answering exam questions. However, their role in diagnosis and treatment planning has not been thoroughly investigated. This study aimed to investigate the reliability of different generative AI models in classifying the severity of dental crowding.

Methods: Two experienced orthodontists categorized the severity of dental crowding in 120 intraoral occlusal images as mild, moderate, or severe (40 images per category). These images were then uploaded to three generative AI models (ChatGPT-4o mini, Microsoft Copilot, and Claude 3.5 Sonnet) and prompted to identify the dental arch and to assess the severity of dental crowding. Response times were recorded, and outputs were compared to orthodontists' assessments. A random image subset was re-analyzed after one week to evaluate model consistency.

Results: Claude 3.5 Sonnet successfully classified the severity of dental crowding in 50% of the images, followed by ChatGPT-4o mini (44%), and Copilot (34%). Visual recognition of the dental arches was higher with Claude and ChatGPT-4o mini (99%) compared to Copilot (72%). Response generation was significantly longer for ChatGPT-4o mini than for Claude and Copilot (p < .0001). However, the response times were comparable for both Claude and Copilot (p = .98). Repeated analyses showed improvement in image classification for both ChatGPT-4o mini and Copilot, while Claude 3.5 Sonnet misclassified a significant portion of the images.

Conclusions: The performance of ChatGPT-4o mini-, Microsoft Copilot, and Claude 3.5 Sonnet in analyzing the severity of dental crowding often did not match the evaluations made by orthodontists. Further developments in image processing algorithms of commercially available generative AI models are required prior to reliable use for dental crowding classification.