Viewing a single comment thread. View all comments

IntrepidTieKnot t1_j547lq2 wrote

Reply to comment by Ok-Cartoonist8114 in [D] is it time to investigate retrieval language models? by hapliniste

I made a tool that chops documents in chunks, creates embeddings for the chunks via GPT-3 and stores the embeddings in a REDIS database. When I make a query, I create an embedding for that and look up my stored embeddings via cosine similarity.

My question is: isn't that the same as your tool does? In other words: what can you do with Cherche what I cannot do like I described? Is it that I don't need GPT-3 for the same result? Or what is it?

2

Ok-Cartoonist8114 t1_j54l5yh wrote

Your pipeline is fine! Cherche is not fancy, it just allow to create hybrid pipelines that rely both on language models and lexical matching which can help a lot. Also Cherche is primarly design for computing embeddings with Sentence Transformers which have a better ratio <precision / number of parameters>.

3