How to Generate Embeddings

📌 The Embedding Pipeline

In the previous lesson, we learned that an embedding is a mathematical representation of a word or sentence's meaning (a vector). But how do you actually turn your company's PDFs or user queries into these vectors?

You do this by passing your text through an Embedding Model. Just like there are LLMs designed to chat (like GPT-4), there are models specifically trained only to convert text to numbers. They are incredibly fast and very cheap.

📌 Generating Embeddings via API (OpenAI)

The most common way to generate embeddings is by calling a proprietary API, such as OpenAI's. Their current flagship embedding model is called text-embedding-3-small (which generates vectors with 1,536 dimensions) or text-embedding-3-large (3,072 dimensions).

Here is what the actual process looks like in code:

1. The Input: You send a request to the API containing your text: "The quick brown fox jumps over the lazy dog."
2. The Model: You specify which model to use (e.g., model: "text-embedding-3-small").
3. The Output: The API responds almost instantly with a JSON object. Instead of text, it contains an array: [-0.012, 0.045, 0.001, -0.092...] (and 1,532 more numbers).

Once you have this array, you save it into a specialized database called a Vector Database (like Pinecone, Weaviate, or Qdrant) alongside the original text. Now, it is ready to be searched by meaning!

📌 Open-Source Alternatives

While OpenAI's API is incredibly cheap (you can embed thousands of pages for a few cents), many enterprises prefer not to send their internal documents to a third-party server.

Because embedding models are much smaller than conversational LLMs, it is very common to run open-source embedding models locally on your own servers. Models like BAAI/bge-large-en-v1.5 or all-MiniLM-L6-v2 (available on Hugging Face) can run entirely offline, guaranteeing total data privacy while still providing excellent semantic search capabilities.