Multi-modal AI

Multi-modal = model yang memahami lebih dari satu jenis input: teks + gambar, teks + audio, teks + video. Membuka use case baru di web apps.

Vision (Image Input)

// Analyze image
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{
    role: "user",
    content: [
      { type: "text", text: "Deskripsi gambar ini dan extract text yang ada" },
      { type: "image_url", image_url: {
        url: "data:image/jpeg;base64,..." // atau URL
      }},
    ],
  }],
});

// Use cases:
// - OCR (extract text from receipt, ID card)
// - Product description from photo
// - Accessibility (describe images for screen readers)
// - Visual QA (tanya tentang diagram, chart, screenshot)

Audio (Speech-to-Text)

// Whisper API
const transcription = await openai.audio.transcriptions.create({
  file: fs.createReadStream("recording.mp3"),
  model: "whisper-1",
  language: "id", // Indonesian
});

console.log(transcription.text);
// "Selamat pagi, hari ini kita akan membahas..."

// Use cases:
// - Meeting transcription
// - Voice notes → text
// - Podcast search

Text-to-Speech

const response = await openai.audio.speech.create({
  model: "tts-1",
  voice: "alloy",
  input: "Selamat datang di BelajarWeb.dev",
});

// Stream audio to client
const buffer = Buffer.from(await response.arrayBuffer());
// Play in browser using Audio API

Combining Modalities

• Voice assistant — Speech-to-text → LLM → Text-to-speech
• Image search — Upload foto → Vision API deskripsi → Semantic search
• Document processing — Scan dokumen → OCR → Extract data → Store