Hybrid CNN-MLLM system achieves 94.91% accuracy for image-based nutrition estimation in Type 1 diabetes

Accurate carbohydrate counting (CC) is crucial for individuals with Type 1 diabetes (T1D) to manage insulin boluses effectively, yet it remains a significant cognitive burden and source of error. Traditional methods often rely on manual estimation, which can be inconsistent and time-consuming, leading to suboptimal glycemic control and increased risk of hypo- or hyperglycemia. Existing food recognition systems typically focus on general dietary logging and lack direct integration with personalized insulin therapy parameters, leaving a critical gap for real-time, patient-specific decision support.

This study developed an image-based nutrition estimation and insulin decision-support module within the AI-assisted Diabetes Care (AIDCARE) platform. The system employs a convolutional neural network (CNN) for food item classification from single meal images. Three CNN architectures—ResNet50, Inception V3, and EfficientNet-B0—were evaluated on a curated food image dataset containing 40 food categories. A separate multimodal large language model (MLLM)-based component was then used to estimate portion size, enabling carbohydrate and nutrient values to be scaled. The system integrates these estimated values with user-specific clinical parameters, such as the insulin-to-carbohydrate ratio and insulin sensitivity factor, to generate advisory bolus guidance. A crucial safety feature requires user confirmation or correction of recognized food categories and estimated portions before displaying insulin recommendations.

Evaluation on a curated food image dataset demonstrated strong performance from the hybrid architecture. Among the tested CNN models, EfficientNet-B0 achieved the highest classification accuracy, reaching 94.91% validation accuracy. This model also showed robust performance across other metrics, with 95.55% precision, 94.87% recall, and a 94.90% F1-score for food category classification. The MLLM-based portion estimation component delivered precise results, exhibiting a mean absolute error (MAE) of 12.27 g and a root mean square error (RMSE) of 15.11 g. These estimated carbohydrate values are then dynamically combined with individual user-specific clinical parameters, including their insulin-to-carbohydrate ratio and insulin sensitivity factor, to formulate personalized advisory bolus guidance. The system incorporates a critical safety step, requiring explicit user confirmation or correction of both the identified food category and the estimated portion size before any insulin recommendations are presented, ensuring patient oversight and reducing potential errors. This integration of accurate image analysis with personalized physiological data represents a significant step towards more precise and user-friendly Type 1 diabetes management.

The EfficientNet-B0 CNN achieved 94.91% validation accuracy for food classification, while the MLLM portion estimator had an MAE of 12.27 g.