Multimodal systems · Glossary term
What is Multimodal Fusion?
Combining evidence or learned representations from more than one modality to produce a joint representation, prediction, or generated output.
Why does Multimodal Fusion matter?
Modalities can supply complementary evidence, but naïve combination can amplify noise, timing errors, or one dominant stream.
Multimodal Fusion in practice
Establish single-modality baselines, specify the fusion point and masks, test missing and contradictory inputs, and report which modalities drive each evaluated slice.
What is the common confusion about Multimodal Fusion?
Fusion is the combination operation. Alignment establishes correspondence, and merely placing two modalities in one request does not prove either occurred successfully.
Learn Multimodal Fusion in the course
Start with
- Cross-Attention Fusion
The projection layer aligns one image vector with one caption vector. A real vision-language decoder needs every text token to attend to every patch token, so the model can ground each word in a…
Taught in Phase 19: Capstone Projects.
Related terms
- Early FusionCombining raw or low-level representations from several modalities before most task-specific modeling occurs.
- Late FusionProcessing modalities through separate encoders or predictors and combining their high-level representations, scores, or decisions near…
- Cross-AttentionAttention in which the query representation comes from one sequence or representation while keys and values come from another.
- Modality AlignmentLearning or establishing correspondences between representations from different modalities so semantically or temporally related items can…
- Multimodal ModelA model that learns from, relates, or generates more than one modality through representation, alignment, fusion, translation, or…
Sources
More terms in Multimodal systems
This entry comes from glossary/terms.md on GitHub. Browse all 250 glossary terms.