Design a multimodal affect-analysis research system with explicit inference limits
Instruction: For a consented research study, propose a system using facial video and speech audio to predict a clearly defined annotation or self-report target. Explain synchronization, encoders, fusion, evaluation and why predictions cannot establish a person's private emotional state.
Updated
Example Answer
I would first define the target: for example, a participant's voluntary self-report for a short interaction, with uncertain or mixed responses retained. That target is not an objective reading of the mind. Facial movements are not a reliable one-to-one diagnostic of emotion across people and situations; adding audio does not remove the need to validate the construct.
For a proposed research baseline, align consented video frames and audio segments to the same event timestamps. Use a video encoder for visible motion and an audio encoder over log-mel features for speech patterns, then fuse their representations with a small classifier and explicit missing-modality masks. Start with each modality alone and a simple fusion baseline before testing cross-attention. Do not substitute one person's audio for another's face or use voice identity as an unnoticed shortcut.
Split by participant and recording session, evaluate later or held-out contexts, and report class performance, calibration, abstentions and uncertainty across relevant populations. Test occlusion, noise and disagreement. Protect recordings and restrict the study's use: predicted labels do not establish honesty, intent, diagnosis or suitability for consequential decisions.
Common Poor Answer
A weak answer assumes facial and vocal patterns reveal true emotions, or supplies a model name without defining labels, alignment and validation.