RESUMEN
This study aims to apply Mask Regional Convolutional Neural Network (Mask R-CNN) to cervical cancer screening using pap smear histological slides. Based on our current literature review, this is the first attempt of using Mask R-CNN to detect and analyze the nucleus of the cervical cell, screening for normal and abnormal nuclear features. The data set were liquid-based histological slides obtained from Thammasat University (TU) Hospital. The slides contained both cervical cells and various artifacts such as white blood cells, mimicking the slides obtained in actual clinical settings. The proposed algorithm achieved mean average precision (mAP) of 57.8%, accuracy of 91.7%, sensitivity of 91.7%, and specificity of 91.7% per image. As we needed to evaluate the efficiency of our algorithm in comparison to single cell classification algorithm (Zhang et al., IEEE JBHI, vol. 21, no. 6, pp. 1633, 2017), we modified our method to also classify single cells on TU dataset test using Mask R-CNN segmentation. The results obtained had an accuracy of 89.8%, sensitivity of 72.5%, and specificity of 94.3%.