Dermato · Published 2026-06-11 · DOI 10.3390/dermato6020022
Serra Aksoy, Pinar Demircioglu, Ismail Bogrekci
<b>Background/Objectives:</b> Vision Transformers (ViTs) have demonstrated impressive performance in dealing with large-scale natural image datasets. They have started to be used in medical image classification problems as well. However, how they behave under real-world conditions, such as data scarcity and extreme class imbalance, has not been well investigated. In this study, we examine the feasibility of using a standard Vision Transformer Base model that learned from scratch how to classify skin lesion images into multiple classes using the ISIC 2019 dataset. <b>Methods:</b> The Vision Transformer architecture was trained from scratch using stratified splitting of the data, class-balanced cross-entropy loss, multi-seed initialization, and control of hyperparameters such as patch size and dropout rate. The evaluation of the Vision Transformer architecture was performed using a hold-out test set with metrics such as accuracy, macro-F1, weighted-F1, and analysis of the confusion matrix. <b>Results:</b> Across all configurations, the training exhibited substantial instability and consistent overfitting behavior, with an average accuracy gap between validation and test sets of 22.7%. Test accuracy ranged from 8.0% to 37.8%, showing high sensitivity to initialization. For minority classes, the F1-score remained very low (F1 < 0.05) even though the classes were balanced in the loss function. <b>Conclusions:</b> The results indicate that a standard ViT-Base model trained from scratch can exhibit pronounced instability and a tendency toward majority-class bias when applied to multi-class skin lesion classification under conditions of extreme class imbalance and data scarcity. The findings point to the limitations of using simple transformer models without pre-training or other forms of inductive bias in scarce data settings.
Abstract from DOAJ. Public domain (CC0 1.0).
Read the article at the publisher →
Aksoy, S., Demircioglu, P., Bogrekci, I. (2026). Systematic Failure of Vision Transformers in Imbalanced Skin Lesion Classification. Dermato. https://doi.org/10.3390/dermato6020022