Mitigating Modality Dropout in Multimodal Learning
Instruction: Explain techniques to handle the scenario where one or more modalities are missing during inference in a multimodal AI system. Fictional inference contract: camera frames older than 200 ms are unavailable. At t=1,000 ms the most recent frame timestamp is 750 ms, while audio is current. An audio-only fallback is validated for speech commands but not for object identification. Decide behavior for each task and after a fresh frame returns.
Context: This question probes the candidate's ability to design robust multimodal AI systems that can still function effectively even when some modalities are unavailable, emphasizing redundancy and fault tolerance.
Updated
Official answer available
Read the opening below, then unlock the full answer and practical guidance.
I mitigate modality dropout by training the system to survive partial input availability instead of assuming every modality is always present...