FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds

FactorJEPA restructures V-JEPA’s monolithic future-latent predictor into explicitly supervised layout, agent, and interaction subspaces, adding visibility-aware entity aggregation and cross-channel separation to substantially improve future prediction, intervention sensitivity, and robustness under occlusion in dense urban scenes.

➡️ 𝐊𝐞𝐲 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬 𝐨𝐟 𝐅𝐚𝐜𝐭𝐨𝐫𝐉𝐄𝐏𝐀:

🌏 𝑫𝑬𝑵𝑺𝑬𝑾𝑶𝑹𝑳𝑫: 𝑨 𝑵𝒆𝒘 𝑾𝒐𝒓𝒍𝒅-𝑴𝒐𝒅𝒆𝒍𝒊𝒏𝒈 𝑹𝒆𝒈𝒊𝒎𝒆: Introduces DENSEWORLD-115k, built from ~1,000 hours of drive-through, walk-through, and aerial footage across 22 Indian cities, specifically targeting high agent density, heterogeneous mixed traffic, persistent occlusion, soft spatial boundaries, and rapid multi-agent negotiation. The authors show that frozen JEPA-family representations struggle in this regime—V-JEPA 2.1 reaches only 44.4% action top-1—motivating world models whose latent structure explicitly represents agents and their interactions rather than relying on correlated scene-level shortcuts.

🧩 𝑭𝒂𝒄𝒕𝒐𝒓𝑱𝑬𝑷𝑨: 𝑳𝒂𝒚𝒐𝒖𝒕–𝑨𝒈𝒆𝒏𝒕–𝑰𝒏𝒕𝒆𝒓𝒂𝒄𝒕𝒊𝒐𝒏 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒐𝒓 𝑺𝒖𝒓𝒈𝒆𝒓𝒚: Replaces V-JEPA’s monolithic predictor with three structured predictive coordinates, $C=[C_L,C_A,C_I]$, synthesized back into the future embedding as $$\hat{Y}=C_LA_L^\top+C_AA_A^\top+C_IA_I^\top$$. Agents are aggregated through a soft visibility gate so occluded entities are downweighted rather than discarded, while interactions use sparse, weighted pairwise relational embeddings. DINOv2-derived structural targets semantically anchor each channel, and covariance + nonlinear RBF dependence penalties suppress cross-factor leakage. Training preserves the pretrained JEPA machinery and performs targeted “predictor surgery,” progressively emphasizing layout → agents → interactions while only unfreezing the top encoder blocks.

📈 𝑺𝒕𝒓𝒖𝒄𝒕𝒖𝒓𝒆𝒅 𝑭𝒖𝒕𝒖𝒓𝒆 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒊𝒐𝒏 𝑩𝒆𝒂𝒕𝒔 𝑮𝒆𝒏𝒆𝒓𝒊𝒄 𝑭𝒊𝒏𝒆-𝑻𝒖𝒏𝒊𝒏𝒈: Against LoRA, DoRA, Auto-RGN, full fine-tuning, continual SSL and other adaptations, FactorJEPA improves the metrics most directly tied to world modeling: Future-frame L1, Causal L1, and Mask-ratio robustness. With the full 115k corpus on the 1B V-JEPA 2.1 backbone, its advantage over the strongest competitor reaches 33.2×, 13.9×, and 43.3× paired-CI widths, respectively, while Motion cosine also becomes a 20.0×-CI win. Method rankings replicate strongly between 1B and 2B backbones $$(\rho=0.895–0.979)$$ across the four primary diagnostics), suggesting that explicitly structuring how a JEPA represents the future—not merely adapting more parameters—is the key contribution.