Mask Region-based Convolutional Neural Network in Object Detection: A Review

Abstract

The integration of Mask R-CNN in object detection has emerged as a prominent advancement in computer vision, enabling accurate instance segmentation and object recognition in complex visual environments. This review systematically analyzes the progress of Mask R-CNN applications from 2020 to 2025, focusing on three key domains where this framework has demonstrated notable performance: data augmentation, occlusion handling, and small object detection. Extensive analysis reveals that data augmentation significantly enhances detection accuracy, particularly in scenarios with limited training data, by increasing mean Average Precision (mAP), F1-score, and recall. Techniques such as flipping, rotation, synthetic data generation, and noise perturbation consistently improved performance across various application contexts. In occluded environments, architectural enhancements including attention modules and feature pyramid networks enabled more robust detection of partially hidden objects, with improvements in precision, IoU, and segmentation quality. Furthermore, for small object detection, adaptations such as depth-based filtering, lightweight backbones, and enhanced feature extraction modules proved effective in capturing fine-grained details. Despite these advancements, challenges persist, including high computational cost, limited real-time capability, and sensitivity to data scarcity and environmental complexity. By critically evaluating these trends and limitations, this review offers valuable insights into the current state and future potential of Mask R-CNN-based object detection systems.