Build Your Own Vector Database from a Repository of Files

1. What you are building

You will build a local database that answers questions by meaning, not merely by matching exact words. A search for “Who approves the annual spending plan?” can retrieve a chunk that says “The board adopts the yearly budget,” even if the exact words differ.

2. Core concepts from the video

Embeddings

An embedding model turns text into a fixed-length array of floating-point numbers. The video uses Ollama's nomic-embed-text, producing 768-dimensional vectors in its demonstration (about 02:46–04:13). Semantically related text points in similar directions even when it shares few words.

Cosine similarity

cosine(A, B) = dot(A, B) / (norm(A) * norm(B))

Cosine similarity compares vector direction. After normalizing every vector to length 1, the dot product equals cosine similarity. That makes a FAISS inner-product index a straightforward cosine-search engine.

Index versus database

The source video builds four versions: a brute-force SQLite/NumPy database, FAISS by itself, FAISS plus SQLite, and ChromaDB using HNSW (about 06:41–33:41).

3. Choose the right architecture

This guide uses SQLite + FAISS IndexIDMap2(IndexFlatIP). It improves on the video's instructional FAISS index position + 1 = SQLite ID shortcut by storing explicit stable IDs in FAISS. That matters as soon as files change or rows are deleted.

4. Prerequisites

• Python 3.10 or newer.
• Ollama installed and running locally.
• Enough disk space for SQLite plus the FAISS index. A 768-dimensional float32 vector uses about 3,072 raw bytes before index/database overhead.
• A repository or folder you are authorized to process.
• Do not index passwords, API keys, student records, medical records, personnel files, or other sensitive data unless the storage and access controls are appropriate.

5. Install and prepare

mkdir my-vector-db
cd my-vector-db
python3 -m venv .venv
source .venv/bin/activate        # macOS/Linux
# Windows PowerShell: .venv\Scripts\Activate.ps1

curl -o repo_vector_db.py https://awheatsandbox.com/public/guides/repo_vector_db.py.txt
curl -O https://awheatsandbox.com/public/guides/repo_vector_db.requirements.txt
pip install -r repo_vector_db.requirements.txt

uv venv
source .venv/bin/activate
uv pip install -r repo_vector_db.requirements.txt

ollama pull nomic-embed-text
ollama serve

curl http://localhost:11434/api/embed \
  -d '{"model":"nomic-embed-text","input":"The board approves the annual budget."}'

6. Prepare the repository of files

Choose one root folder. Every stored source path is relative to this root, so results stay portable.

knowledge-repository/
├── policies/
│   ├── budget-policy.md
│   └── purchasing.pdf
├── procedures/
│   └── onboarding.docx
├── code/
│   ├── app.py
│   └── schema.sql
└── meeting-notes/
    └── 2026-07-15.md

The companion program supports common text/code formats plus PDF, DOCX, and PPTX. It skips symlinks and common build/cache folders including .git, node_modules, .venv, dist, and build.

7. Extract and chunk the files

Embedding an entire 100-page document as one vector makes retrieval vague. The program instead divides each file into overlapping chunks. Defaults:

• 1,800 characters per chunk—roughly a few paragraphs.
• 250 characters of overlap—preserves context across boundaries.
• Prefer paragraph or newline boundaries where possible.
• Store the repository-relative path, chunk number, and approximate line range for every chunk.

How to tune chunk size

Character chunking is intentionally understandable. A production improvement is structure-aware chunking: headings for Markdown, functions/classes for code, pages/sections for documents, and speaker/topic turns for transcripts.

8. Create the SQLite source-of-truth schema

The companion program creates these tables automatically:

files(
  path PRIMARY KEY, sha256, size, mtime_ns, indexed_at
)

chunks(
  id PRIMARY KEY, path, chunk_index, start_line, end_line,
  text, vector BLOB, vector_dim,
  UNIQUE(path, chunk_index)
)

settings(key PRIMARY KEY, value)

The file SHA-256 detects changes. The recoverable vector BLOB allows the FAISS index to be rebuilt without contacting Ollama. The stable numeric chunks.id is also stored inside FAISS.

9. Generate embeddings correctly

The program calls Ollama's /api/embed endpoint in batches. Conceptually:

payload = {"model": "nomic-embed-text", "input": [chunk1, chunk2, ...]}
response = POST http://localhost:11434/api/embed
vectors = response["embeddings"]

Every vector is converted to float32. The first successful batch determines the stored vector dimension. The program records the embedding model and refuses to query with a different model.

Important embedding rules

1. Use the same model for documents and queries.
2. Never silently mix dimensions or models in one index.
3. Batch calls for throughput, but reduce --batch-size if memory or request limits cause errors.
4. Measure retrieval quality on your real documents; a larger model is not automatically better for your domain.

10. Build the FAISS cosine index

The program normalizes every vector, creates an inner-product index, wraps it with stable ID support, and saves it atomically:

base = faiss.IndexFlatIP(dimension)
index = faiss.IndexIDMap2(base)
faiss.normalize_L2(vectors)
index.add_with_ids(vectors, sqlite_chunk_ids)
faiss.write_index(index, temporary_file)
os.replace(temporary_file, final_index_file)

For unit-length vectors:

inner_product(query, document) = cosine_similarity(query, document)

IndexFlatIP is exact: it checks every vector, but its optimized C++ implementation is usually far faster than repeatedly loading and deserializing SQLite BLOBs in Python. At much larger scale, consider HNSW, IVF, PQ, or a complete vector-database service.

11. Run the first repository sync

From your vector DB project directory:

python repo_vector_db.py \
  --db vector_store/knowledge.sqlite3 \
  --index vector_store/vectors.faiss \
  sync /absolute/path/to/knowledge-repository

Expected phases:

1. Discover supported files and calculate SHA-256 hashes.
2. Compare hashes with the files table.
3. Extract and chunk only new or changed files.
4. Embed those chunks through Ollama.
5. Update SQLite in a transaction.
6. Rebuild and atomically replace the FAISS index from all SQLite vectors.

Check health:

python repo_vector_db.py \
  --db vector_store/knowledge.sqlite3 \
  --index vector_store/vectors.faiss \
  status

The important invariant is:

chunks == index_vectors

Useful sync controls

# Smaller chunks for code-heavy repositories
python repo_vector_db.py sync /repo --chunk-chars 1200 --overlap-chars 180

# Index only selected types
python repo_vector_db.py sync /repo --extensions md,txt,pdf,docx

# Lower batch size when Ollama runs out of memory
python repo_vector_db.py sync /repo --batch-size 8

# Re-embed everything after intentionally changing the model
python repo_vector_db.py --model new-model sync /repo --rebuild-all

12. Search and receive source-cited results

python repo_vector_db.py search \
  "Who must approve purchases above the bidding threshold?" \
  --top-k 5

Each result includes:

• cosine similarity score;
• repository-relative file path;
• approximate source line range;
• the retrieved text excerpt.

Request the complete chunk:

python repo_vector_db.py search "special education staffing process" --top-k 8 --full-text

Restrict results to a repository area:

python repo_vector_db.py search "board approval" --path-prefix policies/

13. Handle changed and deleted files safely

Run the same sync command whenever the repository changes. The program:

• leaves unchanged files alone;
• deletes old chunks for changed files and inserts newly embedded chunks;
• removes database rows for files no longer present;
• rebuilds FAISS using stable SQLite chunk IDs.

This gives incremental embedding while keeping index recovery simple. Rebuilding an exact flat FAISS index is fast enough for many personal repositories; if index rebuild time becomes a problem, graduate to an update-friendly ANN index or a vector database.

Repair an index mismatch

python repo_vector_db.py \
  --db vector_store/knowledge.sqlite3 \
  --index vector_store/vectors.faiss \
  rebuild-index

python repo_vector_db.py status

Back up both knowledge.sqlite3 and vectors.faiss. SQLite is the more important file because it contains the recoverable vectors and source records.

14. Connect retrieval to an LLM (RAG)

Retrieval-Augmented Generation adds one more layer: put the top chunks into an LLM prompt and require citations.

results = search_index(question, ..., top_k=5)

context = "\n\n".join(
    f"SOURCE {r['path']}:{r['start_line']}-{r['end_line']}\n{r['text']}"
    for r in results
)

prompt = f"""
Answer only from the supplied sources.
If the answer is not supported, say you could not find it.
Cite file paths and line ranges after each material claim.

QUESTION:
{question}

SOURCES:
{context}
"""

Production-quality additions

• Apply access-control filters before content reaches the LLM.
• Use hybrid retrieval: keyword/BM25 plus semantic vectors.
• Rerank the top 20–50 candidates with a cross-encoder, then pass the best 5–10 chunks.
• Deduplicate overlapping chunks from the same file.
• Include a minimum-score or “insufficient evidence” rule validated on real questions.
• Log retrieved source IDs and model/version for audits.

15. Evaluate quality and speed

The video stresses that testing only 50 items can produce misleading architecture choices. Its benchmarks are demonstration results from one setup—not universal guarantees. Repeat measurements on your hardware and content.

Create a retrieval test set

1. Write 25–100 realistic questions.
2. For each question, label one or more correct source files/chunks.
3. Run search at k=1, 3, 5, 10.
4. Calculate Recall@K: the fraction of questions where a correct source appears in the top K.
5. Measure p50 and p95 search latency separately from embedding latency.
6. Inspect false positives and modify chunking, model, filters, or reranking.

Comparison lesson from the video

The video's measured pattern is more important than any single number: SQLite/NumPy brute force slows linearly; FAISS flat exact search is much faster but still O(N); ChromaDB's HNSW has overhead at tiny scale but its approximate graph search slows far less as N grows (about 29:17–33:41).

16. Security and privacy

• Secrets: exclude .env, credential files, private keys, token exports, browser profiles, and database dumps.
• Permissions: a vector database can reveal semantic matches to content that ordinary folder permissions would hide. Preserve authorization boundaries.
• PII and regulated records: do not index protected education, health, personnel, or legal records without an approved environment and retention policy.
• Backups: encrypt backups and protect both SQLite and FAISS files.
• Prompt injection: repository files are untrusted data. A retrieved document can contain instructions intended to manipulate an LLM. Treat retrieved text as quoted evidence, not system instructions.
• Remote embedding services: this guide uses local Ollama so text stays on the machine. If you replace it with a cloud API, review data handling and contracts first.
• Deletion: after removing sensitive content, sync the repository and rotate/delete old backups containing that content.

17. Troubleshooting

18. Quick command reference

# First build / later incremental update
python repo_vector_db.py sync /absolute/path/to/repository

# Health check
python repo_vector_db.py status

# Search
python repo_vector_db.py search "your question" --top-k 5

# Search a subsection
python repo_vector_db.py search "your question" --path-prefix policies/

# Print complete chunks
python repo_vector_db.py search "your question" --full-text

# Recover FAISS from SQLite
python repo_vector_db.py rebuild-index

# Re-embed after intentionally changing models
python repo_vector_db.py --model another-embedding-model sync /repo --rebuild-all

Source and extension notes

The conceptual progression, formulas, instructional classes, and benchmark narrative come from Rahul Pandey's “Build Vector Database From Scratch” and its companion repository. Repository crawling, document extraction, overlapping chunks, SHA-256 change detection, stable FAISS IDs, incremental synchronization, recovery, CLI commands, security controls, and RAG integration are practical extensions created for this guide's specific goal: building your own searchable vector database from a repository of files.

View the source map and research notes.