Two-stage multi-view deep network for 3D human pose reconstruction using images and its 2D joint heatmaps through enhanced stack-hourglass approach

Abstract

Human beings easily reconstruct the 3D pose of a human from a 2D image, but 3D human pose reconstruction (HPR) continues to exist as a challenging task for machines. Traditional methods can reconstruct the 3D pose from the image directly or from the 2D joint locations that have been used for 3D HPR. Such traditional strategies have their merits and demerits. In this paper, we have tried to combine the merits of such traditional techniques with that of a deep architecture model. By this strategy, the model delivers both of the merits concurrently in a multi-view scenario and also fuses this knowledge on the upcoming step with early and late fusion strategies. We also introduce an enhanced stack-hourglass network for the prediction of 2D keypoint heatmaps. The predicted 2D keypoint heatmaps and the image have been utilized with simple CNN neural architecture along with both of the fusion strategies for 3D pose reconstruction. Experimental results show that the proposed method achieves comparable performance to the state-of-the-art methods on MPII, and Human3.6M datasets. © 2021, The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.