Understanding how humans interact with objects in complex visual scenes is fundamental to human-centric artificial intelligence. Yet, current Human-Object Interaction (HOI) detection frameworks often falter when faced with visual occlusions, vague interaction semantics, and scenarios that extend beyond predefined categories. We present Multi-View Context-Aware Disentangled Interaction Network (MCD-Inet), a unified framework designed to robustly detect HOIs under these challenging, open-world conditions.
Our approach integrates a novel Multi-View Context Fusion mechanism that synthesizes fine grained spatial cues and broad semantic information across instance, interaction, and global levels. To decouple the learning of object localization from interaction semantics, we introduce a Disentangled Interaction Decoder, enabling clearer task specific representations. Complementing this, our Bilateral Knowledge Adaptation module bridges high-resolution spatial features from the visual backbone with abstract semantics from a pre-trained vision language model.
To overcome the challenge of occluded interactions, we design an Occlusion-Robust Enhancement module, which extrapolates missing regions and mines contextual cues to recover interaction evidence. Finally, MCD-INet leverages multi-modal prompts incorporating both textual templates and visual exemplars to extend its detection capabilities to unseen HOI triplets. Extensive evaluations on HICO-DET, V-COCO, and the HICO-Ambiguous datasets confirm the superiority of our approach, achieving a new state-of-the-art mAP of 40.7% and demonstrating significant resilience in occluded and zero-shot scenarios.
MCD-INet offers a powerful step toward generalizable, context-aware HOI detection in real-world visual environments. HOI detection is a cornerstone task in human-centric scene understanding, aiming to predict triplets in the form of . This task requires models to not only localize humans and objects in an image but also understand and identify the interactions between them1.
Applications of HOI detection span a wide array of domains, including robotics, autonomous systems, activity recognition, and intelligent surveillance. A further potential application lies in forensic and investigative video analysis, where HOI detection may assist in identifying segments containing interactions such as carrying, holding, concealing, or using particular objects. Such predictions may help prioritize relevant footage for human review; however, an automatically detected human object relationship should be understood as a technical observation rather than a determination of criminal conduct or responsibility2.
However, despite remarkable advancements in recent years, HOI detection continues to face significant challenges in real-world settings. One of the most persistent challenges arises from visual occlusion3. In real-world scenarios, humans and objects are often partially or fully obscured, particularly in crowded environments.
Extract — continue reading at the source.
