group 2024 banner

CVPR 2023 Paper On Unbiased Scene Graph Generation in Videos

Our paper 'unbiased scene graph generation in videos' is accepted in CVPR'23. The work addresses the critical problem of biased visual relationship detection in videos.
  • The task of dynamic scene graph generation (SGG) from videos is complicated and challenging due to the inherent dynamics of a scene, temporal fluctuation of model predictions, and the long-tailed distribution of the visual relationships in addition to the already existing challenges in image-based SGG. Existing methods for dynamic SGG have primarily focused on capturing spatiotemporal context using complex architectures without addressing the challenges mentioned above, especially the long-tailed distribution of relationships. This often leads to the generation of biased scene graphs. To address these challenges, we introduce a new framework called TEMPURA: TEmporal consistency and Memory Prototype guided UnceRtainty Attenuation for unbiased dynamic SGG. TEMPURA employs object-level temporal consistencies via transformer-based sequence modeling, learns to synthesize unbiased relationship representations using memory-guided training, and attenuates the predictive uncertainty of visual relations using a Gaussian Mixture Model (GMM). Extensive experiments demonstrate that our method achieves significant (up to 10% in some cases) performance gain over existing methods highlighting its superiority in generating more unbiased scene graphs.

Unbiased Scene Graph Generation in Videos , S. Nag, K. Min, S. Tripathi, and A. Roy-Chowdhury, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2023.