Fusion of Forward and Backward LSTMs for Effective Occlusion Reconstruction in Gait Sequences

Abstract

Gait-based person identification in the presence of occlusion is a challenging problem and research work in this domain is still in its infancy. Some methods that have been developed in the past are based on certain assumptions such as gait features over a cycle following a Gaussian, or a complete cycle can be reconstructed from multiple occluded cycles. Recent deep neural network-based approaches to occlusion handling also mostly focus on making feature-level reconstruction instead of frame-level reconstruction, and thus their effectiveness is likely to suffer if several frames in a cycle are corrupted due to occlusion. There exists a single work on LSTM-based occlusion reconstruction which predicts frames based on the previous unoccluded/reconstructed frames only and does not utilize the complete sequence information effectively. In this paper, we improve the existing work on occlusion reconstruction by performing a two-way prediction using LSTMs and finally combining the two predictions. Our reconstruction model is based on (i) a forward LSTM that performs reconstruction in the forward direction by predicting each occluded frame from a few previous unoccluded or already reconstructed frames and (ii) a backward LSTM that carries out reconstruction in the backward direction by predicting each occluded frame from a few succeeding unoccluded or already reconstructed frames. To train the LSTMs, an extensive gallery set is constructed from the CASIA-B and the OU-ISIR LP data, and the mean-square loss between the corresponding generated and target frames is minimized. Next, a fusion network combines the predictions from the two LSTMs to generate the final reconstructed frame corresponding to each occluded frame. Evaluation of our approach has been performed using synthetically occluded sequences generated from the OU-ISIR LP, OU-ISIR MVLP and CASIA-B data and real occluded sequences present in the TUM-IITKGP and GREW data. The effectiveness of the proposed reconstruction model has been verified by the Dice score and the accuracy of gait-based recognition using some popular gait recognition methods. A detailed comparative study with recent gait recognition and video frame prediction methods in the literature also showcases the effectiveness of occlusion reconstruction using the proposed model. © The Author(s), under exclusive licence to Springer Nature Singapore Pte Ltd. 2025.