RESUMEN
BACKGROUND: For virtual skincare using a touch feedback interface, reconstructing a 3D skin tactile surface from a mobile skin image is imperative for a dermatologist to palpate the skin surface that presents tactile characteristics of the subcutaneous tissues. However, the precise tactile reconstruction from a single view image is a challenging research problem due to varying illumination conditions. METHODS: In this study, a deep learning-based tactile reconstruction scheme is proposed to restore tactile properties from light distortion and reconstruct the 3D tactile surface from a mobile skin image. Our method consists of light distortion removal using deep learning, cGAN, and 3D tactile surface generation based on image gradients. RESULTS: The proposed method was tested by conducting two evaluation experiments in terms of removing light distortion and reconstructing 3D skin tactile surface in comparison with other well-known methods. The results demonstrated that our method outperforms existing other methods in both illumination-free image restoration and 3D surface reconstruction. CONCLUSION: The proposed method is a promising approach in that tactile property distorted by illuminations can be completely restored using deep learning with a smaller training set and the precise reconstruction of 3D skin tactile surface can be achieved to be ready for a remotely touchable interface for virtual skincare applications.
Asunto(s)
Aprendizaje Profundo , Percepción del Tacto , Humanos , Palpación , Piel/diagnóstico por imagen , TactoRESUMEN
BACKGROUND: Haptic skin palpation with three-dimensional skin surface reconstruction from in vivo skin images in order to acquire both tactile and visual information has been receiving much attention. However, the depth estimation of skin surface, using a light field camera that creates multiple images with a micro-lens array, is a difficult problem due to low-resolution images resulting in erroneous disparity matching. METHODS: Multiple low-resolution images decoded from a light field camera have limitations to accurate 3D surface reconstruction needed for haptic palpation. To overcome this, a deep learning method, Generative Adversarial Networks, was employed to generate super-resolved skin images that preserve surface detail without blurring, and then, accurate skin depth was estimated by taking multiple subsequent steps including lens distortion correction, sub-pixel shifted image generation using phase shift theorem, cost-volume building, multi-label optimization, and hole filling and refinement, which is a new approach for 3D skin surface reconstruction. RESULTS: Experimental results of the deep-learning-based super-resolution method demonstrated that the textural detail (wrinkles) of super-resolved skin images is well preserved, unlike other super-resolution methods. In addition, the depth maps computed with our proposed algorithm verify that our method can produce more accurate and robust results compared to other state-of-the-art depth map computation methods. CONCLUSION: Herein, we first proposed depth map estimation of skin surfaces using a light field camera and subsequently tested it with several skin images. The experimental results established the superiority of the proposed scheme.