Things I learned from - What is Multimodal AI? How LLMs Process Text, Images, and More

Overall message
The video explains what “multimodal AI” means today, how the underlying architectures have evolved from simple “bolt-on” systems to truly unified models, and what new capabilities this enables.

  1. What counts as a modality
    • A modality is just a kind of data (text, image, audio, LIDAR, thermal, etc.).
    • A single-modality model treats only one type of data; e.g., a large language model (LLM) takes in tokens derived from text strings and produces only text.
  2. Two generations of multimodal architecture

A. Modular feature-level fusion (early method, still used)

  • You keep a separate model per modality: an LLM for text plus an additional vision encoder (e.g., CLIP) for images.
  • The image is fed into the vision encoder, which extracts a numeric “feature vector” (a summary) and hands that over to the language model.
  • Pros: Cheaper, easier to swap components; good for narrow, enterprise tasks.
  • Cons: Information can be lost because the vision encoder doesn’t know what the LLM will ask about; the LLM never sees the raw image, only the compressed description.

B. Native multimodality (current “gold standard”)

  • One shared vector space is learned for all modalities.
    – Text is tokenized and embedded as usual.
    – Images are broken into small 2-D patches, each getting its own vector in the same space.
    – Audio is chunked and embedded analogously.
  • Because everything inhabits the same high-dimensional space, the model can reason about them jointly.
    – Benefits: finer fidelity; question-aware attention (it can zoom in on the exact part of a screenshot you asked about).
    – No cross-modal translation layer—no lossy intermediate representation.
  1. Special case: video
    • Video has a temporal dimension.
      Early approach (feature-level): sample a handful of static frames; loses motion information.
      Native approach (current): small 3-D “spatio-temporal tokens” (tiny 3-D cubes of pixels × time). Each token already contains motion, so no need to infer change by comparing frames.
  2. Any-to-any generation
    Because every input and output form lives in the same space, the model can accept any mix of modalities and produce any mix of modalities. Example prompt: “Explain how to tie a tie.” The system can reply with both text instructions and a short, internally coherent video clip of the process.

Key takeaway
State-of-the-art multimodal AI isn’t just gluing separate models together; it embeds multiple data types in one shared internal language, yielding more nuanced understanding, better detail preservation, and the flexibility to both ingest and generate across modalities in any combination.

This summary was generated by AI from the YouTube video captions.