Cost-sensitive DenseNet-121 with progressive curriculum augmentation and dynamic threshold optimization for imbalanced skin cancer classification

Abstract

Skin cancer is one of the leading causes of cancer-related mortality worldwide, where early melanoma detection substantially improves patient survival. However, automated dermoscopic image classification remains challenging because malignant lesions are severely underrepresented, causing deep learning models to become biased toward benign cases. This study proposes a clinically oriented deep learning framework for imbalanced skin cancer classification using DenseNet-121 on a rigorously curated subset of the ISIC 2024 dataset. Prior to model development, patient-level integrity validation and duplicate inspection were performed, followed by patient-level data partitioning to eliminate data leakage. A total of 12,000 dermoscopic images were curated and divided into 8,296 training, 1,904 validation, and 1,800 testing images. The proposed framework integrates curriculum-based progressive augmentation, cost-sensitive optimization using weighted binary cross-entropy, MixUp and CutMix regularization, and dynamic threshold optimization to prioritize malignant lesion detection. Experimental results demonstrate that the proposed model achieved a test area under the receiver operating characteristic curve (AUC) of 0.8959 and a malignant recall (sensitivity) of 92.68%, correctly identifying 38 of 41 malignant lesions while missing only three cases. Although the clinically optimized decision threshold reduced precision because of increased false positives, it substantially decreased false negatives, making the framework well suited for preliminary melanoma screening. These findings demonstrate that combining rigorous patient-level curation, progressive augmentation, cost-sensitive learning, and clinically driven threshold optimization provides an effective strategy for improving malignant lesion detection in highly imbalanced dermoscopic datasets.