Evaluasi Output AI

Bagaimana kamu tahu AI memberikan jawaban yang benar? Evaluasi adalah kunci untuk iterasi prompt dan memilih model yang tepat.

Evaluation Methods

1. Manual Review

Baca output, beri rating 1-5. Lambat tapi paling akurat.

2. LLM-as-Judge

// Gunakan LLM lain untuk menilai output
const evaluation = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{
    role: "system",
    content: `Evaluasi jawaban berikut berdasarkan:
    1. Akurasi (1-5)
    2. Kelengkapan (1-5)
    3. Kejelasan (1-5)

    Question: ${question}
    Answer: ${answer}

    Return JSON: { accuracy, completeness, clarity, explanation }`
  }],
  response_format: { type: "json_object" },
});

3. Automated Metrics

• Exact match — Output = expected? (untuk structured output)
• Contains check — Output mengandung keyword penting?
• Cosine similarity — Seberapa mirip output vs reference?

Evaluation Dataset

// Buat test cases
const evalSet = [
  {
    input: "Apa itu closure di JavaScript?",
    expectedTopics: ["fungsi", "scope", "variabel luar"],
    badOutputIndicators: ["Python", "Java class"],
  },
  // ... 50+ test cases
];

// Run evaluations
for (const test of evalSet) {
  const output = await getAIResponse(test.input);
  const hasTopics = test.expectedTopics.every(t => output.includes(t));
  const hasBad = test.badOutputIndicators.some(b => output.includes(b));
  console.log({ pass: hasTopics && !hasBad, output });
}

A/B Testing Prompts

Ubah prompt → jalankan eval set → bandingkan skor. Treat prompt changes seperti code changes — test before deploy.