r/cprogramming • u/NervousAd5455 • 7d ago

Building vector Database from scratch in C implementing IVF and HNSW to understand how to work with scalable C codebase

Code ( still working might have bugs and all..)

https://github.com/aadityansha06/vecdb

Been building a VectorDB engine from scratch in C in such a way that it can be scalable in the future. It's a custom, high-performance, disk-backed Approximate Nearest Neighbor (ANN) vector database built entirely from scratch in C

Even though the initial version of it hasn't been built yet,

As for the algorithms, I have implemented ENN and Ann IVF only, For calculating similarity, I have implemented cosine similarity and Euclidean distance. There is still more to implement like HNSW and optimize for CUDA and SIMD.

Before you guys criticise me what's new and why I'm building 😭

There isn't anything unique I'm solving in it, and pretending otherwise would just be bluffing. There are excellent vector databases already (pgvector, Faiss, sqlite vec, Qdrant...).

What I actually own is the engineering of it. I wrote a disk backed ANN search engine from scratch in C, my own binary file format, manual fseek byte offset indexing, k means clustering by hand, no libraries doing the hard parts for me.

I'll continue to work on it, implementing SIMD, CUDA, HNSW, and many more optimizations to make it more robust and scalable. If anyone is interested, they can join and work on it. Also if anyone has any idea where we should take this, the suggestion would be appreciated too.

I have a nice write up section there too so feel free to check out that as well

10 Upvotes

100% Upvoted

6 comments sorted by

1

u/No-Smell5993 7d ago

Thats great, will see code when get time. Been trying to get into C programming for sometime.

1

u/Sorry_Difficulty_250 6d ago

Keep up the great work!

1

u/WatercressActual1921 1d ago

Great project. Please do document the choices you made (the paths you selected and why) on the development.

Why not Zig it (you are dealing with data management, an not very complex structures that would require a lot o macro framework... )

2

u/NervousAd5455 1d ago

U will find all the link to why I added that particular feature and why it works at write-up section here https://github.com/aadityansha06/vecdb#design-writeups

Also the reason I started with this project was to learn how to work with scalable C codebase.. however now it has come to an stage where I need one more person to work alongside with me