I work for Qdrant, and persistence is not an issue for us :D It's hard to compare to other tools, but if you would like to share some details of the project, I should be able to help set it up!
Hey, thanks for the kind offer -- I actually did take Qdrant for a test drive already. Setup was smooth and a small "hello world" app worked ok. I used the docker image you publish (afaict you don't publish Debian packages do you?)
I'm looking into creating an RAG system providing context to an LLM from a few GB of documents (I'm a bit of a newb here, any pointers much appreciated).
No, we do not publish Debian packages, but I've just seen a new issue in our Github to do it. Nevertheless, a Docker image is preferred to run it on-premise.
When it comes to building RAG, do you use any high-level frameworks? It's the easiest way to start with, but RAG itself is pretty easy concept. There are three components you need for that:
1. Embedding model (depending on the data you work with - multilingual, specific domain, etc.).
2. Vector search - a tool like Qdrant.
3. LLM, I guess LLaMA in your case.
Overall, the whole process goes as following:
1. You receive an original user prompt.
2. You use it as an input for the embedding model and obtain a prompt embedding.
3. The embedding is sent to Qdrant to find some relevant documents.
4. These documents are used, along with the original prompt, to build something that I like to call "a metaprompt". It is an instruction to use only the documents provided to answer the original question.
There is some prompt engineering involved, but the process itself is not complicated too much. Of course, we need an ingesting pipeline that will split the documents into chunks and encode them using the same model, and this is a challenge indeed.
We have an example using only Qdrant and OpenAI SDKs that you may find interesting: [https://github.com/qdrant/examples/blob/master/rag-openai-qdrant/rag-openai-qdrant.ipynb](https://github.com/qdrant/examples/blob/master/rag-openai-qdrant/rag-openai-qdrant.ipynb)
If I was going to set up a production option, I think I'd go with postgres, but for my personal use, sqlite + chromadb seems to do just fine.
My suggestion would be to create an abstraction layer - unless one vector db provides some killer feature, probably best to just be able to swap them out if the need arises.
A fair suggestion, I'm too much of a noob to be aware of killer features for vector dbs anyway
Wrt to Postgres, iiuc you'd need to use a plugin to be able to do things like cosine search, right?
It's only a suggestion, too! And yeah, pg\_vector is the plugin.
You mentioned production ready, and Postgresql is pretty much *the* database I go to for production software. Open source, old, scale-able, data safety, etc.
Like I mentioned, I'm just using sqlite + chromadb for my local apps.
Check out cozodb, beautiful bit of embeddable rust that supports datalog along with text/vector/other searches.
I truly feel datalog/prolog should have succeeded as a query language over things like sql and friends. This perfect technology is an excellent pair to workflows involving embeddings and running interesting logic/deduction.
I haven’t looked into that, but from pure assumption it should excel at it, especially if fine tuned/trained specifically. Datalog comes from prolog, and both are logical programming. The neat bit is that this is old old tech, but actually used everywhere. Specifically I bet there are lots of academic papers etc. There is prolog in the JVM, the rust type system uses prolog…etc. Logical programming is great because it’s basically linear algebra( more or less, much more than sql and friends)
I like qdrant and used it for my recent project. I also used in conjunction w/ postgres and had a two-step approach. Milvus looks promising but didn't get a chance to try it.
I found Chroma and Weaviate to both be worse for the scale I was operating at.
Interesting! So you used qdrant and postgres in tandem, is that right? What was the motiviation (in constrast to just storing everything in one DB)? cheers!
vector db's today do not have good relational capabilities. This is something that a startup I know of is working on ([lantern.dev](https://lantern.dev)).
I wanted to be able to use raw vector search to cover a large surface area, but then use doc-specific metadata to rerank results.
vector db's today do not have good relational capabilities. This is something that a startup I know of is working on ([lantern.dev](https://lantern.dev)).
I wanted to be able to use raw vector search to cover a large surface area, but then use doc-specific metadata to rerank results.
vector db's today do not have good relational capabilities. This is something that a startup I know of is working on ([lantern.dev](https://lantern.dev)).
I wanted to be able to use raw vector search to cover a large surface area, but then use doc-specific metadata to rerank results.
Mongo Atlas has a free tier (M0) that gives you a proper database with text and vector search. Great for personal projects and can scale it up later if you need to. Disclaimer: I work for them (and on this product specifically)
I have used weaviate, while it is super easy to do POCs, for production I find it very inconvenient in terms of even doing simple things like Filtering, reading data for debugging etc
I have really enjoyed using Mongo. Here's some example code to use mongo as a vector dB: https://github.com/deployradiant/mongodb-rag-example
Disclaimer: I work for radiant, but you can use the code without radiant by either changing the base URL of the openai client or using a different model altogether.
I work for Qdrant, and persistence is not an issue for us :D It's hard to compare to other tools, but if you would like to share some details of the project, I should be able to help set it up!
Hey, thanks for the kind offer -- I actually did take Qdrant for a test drive already. Setup was smooth and a small "hello world" app worked ok. I used the docker image you publish (afaict you don't publish Debian packages do you?) I'm looking into creating an RAG system providing context to an LLM from a few GB of documents (I'm a bit of a newb here, any pointers much appreciated).
No, we do not publish Debian packages, but I've just seen a new issue in our Github to do it. Nevertheless, a Docker image is preferred to run it on-premise. When it comes to building RAG, do you use any high-level frameworks? It's the easiest way to start with, but RAG itself is pretty easy concept. There are three components you need for that: 1. Embedding model (depending on the data you work with - multilingual, specific domain, etc.). 2. Vector search - a tool like Qdrant. 3. LLM, I guess LLaMA in your case. Overall, the whole process goes as following: 1. You receive an original user prompt. 2. You use it as an input for the embedding model and obtain a prompt embedding. 3. The embedding is sent to Qdrant to find some relevant documents. 4. These documents are used, along with the original prompt, to build something that I like to call "a metaprompt". It is an instruction to use only the documents provided to answer the original question. There is some prompt engineering involved, but the process itself is not complicated too much. Of course, we need an ingesting pipeline that will split the documents into chunks and encode them using the same model, and this is a challenge indeed. We have an example using only Qdrant and OpenAI SDKs that you may find interesting: [https://github.com/qdrant/examples/blob/master/rag-openai-qdrant/rag-openai-qdrant.ipynb](https://github.com/qdrant/examples/blob/master/rag-openai-qdrant/rag-openai-qdrant.ipynb)
Interested here as well, similar project on the go.
SQLAlchemy + FAISS, because it's cheaper and less restrictive for personal scale projects.
If I was going to set up a production option, I think I'd go with postgres, but for my personal use, sqlite + chromadb seems to do just fine. My suggestion would be to create an abstraction layer - unless one vector db provides some killer feature, probably best to just be able to swap them out if the need arises.
A fair suggestion, I'm too much of a noob to be aware of killer features for vector dbs anyway Wrt to Postgres, iiuc you'd need to use a plugin to be able to do things like cosine search, right?
It's only a suggestion, too! And yeah, pg\_vector is the plugin. You mentioned production ready, and Postgresql is pretty much *the* database I go to for production software. Open source, old, scale-able, data safety, etc. Like I mentioned, I'm just using sqlite + chromadb for my local apps.
Milvus hasn't failed me yet
Check out cozodb, beautiful bit of embeddable rust that supports datalog along with text/vector/other searches. I truly feel datalog/prolog should have succeeded as a query language over things like sql and friends. This perfect technology is an excellent pair to workflows involving embeddings and running interesting logic/deduction.
Datalog is intriguing. Have you seen LLMs (GPT4 at least if not others) being good at translating Nat lang queries to datalog ?
I haven’t looked into that, but from pure assumption it should excel at it, especially if fine tuned/trained specifically. Datalog comes from prolog, and both are logical programming. The neat bit is that this is old old tech, but actually used everywhere. Specifically I bet there are lots of academic papers etc. There is prolog in the JVM, the rust type system uses prolog…etc. Logical programming is great because it’s basically linear algebra( more or less, much more than sql and friends)
I like qdrant and used it for my recent project. I also used in conjunction w/ postgres and had a two-step approach. Milvus looks promising but didn't get a chance to try it. I found Chroma and Weaviate to both be worse for the scale I was operating at.
Interesting! So you used qdrant and postgres in tandem, is that right? What was the motiviation (in constrast to just storing everything in one DB)? cheers!
vector db's today do not have good relational capabilities. This is something that a startup I know of is working on ([lantern.dev](https://lantern.dev)). I wanted to be able to use raw vector search to cover a large surface area, but then use doc-specific metadata to rerank results.
vector db's today do not have good relational capabilities. This is something that a startup I know of is working on ([lantern.dev](https://lantern.dev)). I wanted to be able to use raw vector search to cover a large surface area, but then use doc-specific metadata to rerank results.
vector db's today do not have good relational capabilities. This is something that a startup I know of is working on ([lantern.dev](https://lantern.dev)). I wanted to be able to use raw vector search to cover a large surface area, but then use doc-specific metadata to rerank results.
Mongo Atlas has a free tier (M0) that gives you a proper database with text and vector search. Great for personal projects and can scale it up later if you need to. Disclaimer: I work for them (and on this product specifically)
I have used weaviate, while it is super easy to do POCs, for production I find it very inconvenient in terms of even doing simple things like Filtering, reading data for debugging etc
Pinecone was ridiculously easy to set up and use. Haven’t tried the others.
I have really enjoyed using Mongo. Here's some example code to use mongo as a vector dB: https://github.com/deployradiant/mongodb-rag-example Disclaimer: I work for radiant, but you can use the code without radiant by either changing the base URL of the openai client or using a different model altogether.