Implementasi Semantic Search

Semantic search memahami makna query, bukan hanya keyword. Membangun semantic search dari awal untuk web app.

Architecture

// 1. Indexing pipeline
Documents → Chunk → Embed → Store in Vector DB

// 2. Search pipeline
Query → Embed → Vector similarity search → Rank → Return results

Full Implementation

import { Pinecone } from "@pinecone-database/pinecone";
import OpenAI from "openai";

const openai = new OpenAI();
const pinecone = new Pinecone();
const index = pinecone.index("docs");

// Index documents
async function indexDocuments(documents) {
  const vectors = [];

  for (const doc of documents) {
    const chunks = splitIntoChunks(doc.content, 500);

    for (let i = 0; i < chunks.length; i++) {
      const embedding = await openai.embeddings.create({
        model: "text-embedding-3-small",
        input: chunks[i],
      });

      vectors.push({
        id: `${doc.id}-${i}`,
        values: embedding.data[0].embedding,
        metadata: {
          text: chunks[i],
          title: doc.title,
          url: doc.url,
        },
      });
    }
  }

  await index.upsert(vectors);
}

// Search
async function search(query, topK = 5) {
  const embedding = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: query,
  });

  const results = await index.query({
    vector: embedding.data[0].embedding,
    topK,
    includeMetadata: true,
  });

  return results.matches.map(m => ({
    text: m.metadata.text,
    title: m.metadata.title,
    score: m.score,
  }));
}

Hybrid Search

Gabungkan keyword + semantic untuk hasil terbaik:

• Semantic — Tangkap intent dan sinonim
• Keyword (BM25) — Exact match untuk nama, kode, ID
• Reranking — Gunakan cross-encoder untuk re-rank top results