Image, audio and multimodal systems

By the end, you can

• Describe what a multimodal AI system is and name one everyday capability for an image system and one for an audio system (AF-1).
• Name at least one characteristic failure mode each for image-generating and audio-generating systems (AF-1).
• Explain why synthetic image, audio or video content needs the same verification habit as any other unverified claim (AF-1).

Before you start

This is Module 1, Lesson 3 of the AI Foundations course, and the last lesson before Module 2's task-specification work. It builds on Lessons 1 and 2's vocabulary — pattern, training data, model, generation — and extends it beyond text to images, audio and systems that combine more than one of these at once. It also gives a first, honest look at synthetic-media risk; a fuller treatment of AI risk sits later in this course.

Beyond text: image, audio and multimodal systems

Everything in Lesson 2 about pattern continuation over tokens has close relatives that work on images and audio instead of, or alongside, text. A system that only handles one of these — only text, or only images — works within a single **modality**. A system that can take in or produce more than one type of input or output in the same task is called **multimodal**. Google's own developer documentation describes one of its models this way: it is "our first multimodal embedding model, mapping text, images, video, audio and PDFs into a unified embedding space" — an **embedding space** being, roughly, a shared numerical representation that lets a system compare content across different formats. In plainer terms: a single system built to work across several kinds of content at once, rather than one system per content type.

Within image and audio systems, it helps to separate two different jobs, because they fail in different ways:

Some multimodal systems combine both: they can take an image you supply, generate a description of it, and then generate answers to follow-up questions about it in the same conversation.

• **Recognition or understanding systems** take existing content and describe or transcribe it — for example, converting spoken audio into written text, or describing what is visible in a photograph.
• **Generation systems** create new content from an instruction — for example, producing a new image from a written description, or synthesising a spoken voice reading text aloud.

Characteristic failure modes

Each of these system types has its own well-known way of failing fluently rather than failing obviously — the same underlying shape as Lesson 2's text hallucination, applied to a different kind of output.

None of these failures require anything to have gone "wrong" with the system in a broken sense. They are the ordinary output of a pattern-based generation or recognition process encountering a case that does not closely resemble its training examples — the image or audio equivalent of Lesson 2's point that fluent is not the same as correct.

• **Image generation** can produce a picture that looks convincing at a glance but contains details that would not occur in a real photograph — an inconsistency in a hand, an object, a shadow, or garbled text rendered inside the image itself. The picture is not flagged as uncertain; it simply looks as confident as a correct one.
• **Audio and speech recognition** can mishear an uncommon name, a technical term or a heavy accent, and produce a transcript that reads as a clean, plausible sentence rather than an obvious error — the same "fluent but wrong" pattern, in transcribed-text form.
• **Multimodal description** can misread one specific detail in an image — a number on a sign, a date on a document, a face — while describing everything else about the image accurately, so the one wrong detail is easy to miss inside an otherwise correct-sounding description.

Deepfake awareness — a first honest look

The same generation capability that can produce a useful illustration or a helpful read-aloud voice can also be used to produce **synthetic media that impersonates a real person** — a fabricated photo, a cloned voice, or an altered video. The US National Institute of Standards and Technology (NIST) has published a report specifically on this problem, examining "authenticating content and tracking its provenance," "labeling synthetic content, such as using watermarking," and "detecting synthetic content," alongside preventing the most seriously harmful misuses. **Provenance** here means a piece of content's origin and history — who or what created it, and what has been done to it since. One concrete response effort is the Coalition for Content Provenance and Authenticity (C2PA), a standard — backed by a broad coalition of publishers and technology companies — for attaching a verifiable provenance record to the content file itself.

These provenance tools are useful but not yet universal — plenty of real and synthetic content in circulation carries no such record either way. That gap is exactly why the practical habit matters more than any single tool: **treat a surprising image, audio clip or video the same way you would treat a surprising text claim — verify it before you act on it or share it further.** This course returns to verification and risk in more depth later; for this module, the goal is simply to recognise that synthetic media exists, is improving, and is not self-labelling.

A worked example: an urgent voice message

Imagine you receive a short voice message that sounds exactly like a colleague, asking you to urgently transfer money or share a password because they are "stuck in a meeting." Under real time pressure, the fluency of the voice is itself persuasive — it sounds like them, so it feels verified. But sounding right is not the same as being verified, for exactly the same reason a fluent LLM answer is not automatically a correct one.

A safer habit is to verify through a second, independent channel before acting: call the person back on a known number, message them on a separate app, or ask a question only the real person could answer — rather than treating the audio itself as proof. This is the same "verify before you act" instinct this lesson has been building toward, applied to the highest-pressure case: a request for money or access, arriving in a convincing voice.

Accessibility notes

This lesson is text-first, with no images, audio, video or downloadable artifacts of its own — its descriptions of image and audio AI systems are conveyed entirely in prose. The practice exercise's model answer sits behind a native disclosure control that is reachable and operable by keyboard and correctly announced by screen readers. The knowledge check uses native radio-button inputs with a visible question and options, and posts its result to a live status region so assistive technology announces the outcome without a page reload.

Sources and limits

This lesson synthesises the sources below into a practical learning model. It is not a security standard, legal advice or a guarantee that any particular agent design is safe.

1. Gemini API models — Google AI for Developers. Describes a multimodal embedding model that maps text, images, video, audio and PDFs into a unified embedding space.
2. Reducing Risks Posed by Synthetic Content: An Overview of Technical Approaches to Digital Content Transparency — NIST. Surveys authentication, provenance-tracking, labelling and detection approaches for AI-generated or altered images, video, audio and text, including deepfake-related harms.
3. C2PA — C2PA (Coalition for Content Provenance and Authenticity). Describes an open technical standard, backed by a broad coalition of publishers and technology providers, for establishing the origin and edit history of digital content.
4. Artificial Intelligence Risk Management Framework 1.0 — NIST AI Resource Center. Frames an AI system as an engineered system that generates outputs such as predictions or generated content, not an authoritative or self-verifying source.