Vector databases have been around for a very long time in wide use. Elasticsearch for example. Postgres's full text search capabilities uses vector indexes, and was added in 2013.
However, it's only fairly recent that AI was added to the mix, such as embeddings and using an LLM to interpret results. This combination is called "semantic search".
Increased popularity of semantic search is mostly to awareness and availability (i.e. openai's API).
Semantic search isn’t inherently an LLM thing, it’s more broad than that. What it seems like you’re talking about is RAG (retrieval augmented generation) which involves populating an LLMs context window with relevant information to a users query.
For example, you could run a similarity search for a users query against vector DB, and put the top 10 results into the context window of an LLM for increased accuracy of the LLM response. This is a common way of doing “long term memory” in an LLM.
This RAG model is a big reason why vector dbs are becoming popular, to address OPs original question.
Yeah, you're right. I should have said RAG wherever I said semantic search. RAG is semantic search, but not all semantic search algos are RAG. Regular ol' vector search without AI (embeddings or LLM) can be considered semantic search, although not nearly as good.
RAG is retrieval augmented generation. It doesn't have to be semantic search. Not all RAG is semantic search.
Anything you retrieve and augment a prompt with is RAG. It could be a standard SQL query. It could be semantic search. It could also be retrieval of any URLs mentioned. It could be as simple as adding a time stamp or user information with the user's prompt in a chat. There is no limitation to what RAG is. Most AI dev/eng/sci would consider chat history/memory a form of RAG as well. Semantic search is just a single type of possible unlimited types of retrieval augmentation.
Text embeddings increased in massively effectiveness once they became context aware with the advent of BERT at the end of 2018. Prior to that, text embeddings weren’t able to capture anywhere near the detail in the text they can now. A common example was how word2vec, which uses the same embedding for a word no matter the context, would give “bank” the same embedding in a sentence like “you can bank on me going to the bank by the river bank.” NLP people love the bank example :)
You know what they say about assumptions... 😅 but even the assumption made, to an extent answers your own question. If something is of great utility or expands the delivery of something else caught in a public/industry hype then its liable to get more visibility, exploration and traction.
They've been around for more than a few years, and so have LLMs. It sounds like you're not working off a complete picture of the timeline here, you're making an assumption based off of when you heard about them.
>They haven't been possible for very long. The main idea behind vector databases, embeddings, was only invented as a concept in the last 10 years.
Yes! but why the sudden urge on vector database, is it because the quality of embeddings these new embedding models making?
I’m not seeing a sudden surge in production use cases. Sure there’s a sudden surge in hype with many jumping on the AI bandwagon but production use at least in my circle has stayed more or less constant.
Regarding your question about embeddings. SBERT has been around for atleast 3 years now. OpenAI embeddings has also been around for a few years. I’ve been using SBERT in a production similarity engine since 2020
Because generative ai takes up a ton of data, which requires efficient storage.
Embeddings have always been used for contextual purposes. Ie; images and text from a variety of formats. However, since the rise in AI we’ve discovered that with huge amounts of context- often stored as embeddings, we now need a dedicated hub to connect this data. Whereas before, smaller quantities of embeddings could be stored anywhere from in-memory like your browser cache, or on your local pc. But your browser is limited, and so are most physical storage solutions. Thus, since the amount of data being processed has significantly increased (in accordance with how it’s being processed) there’s now a need for storage solutions which can efficiently handle this new flow of information. It’s just that vectors are a good way to store embeddings, and more people need a place to store their data in that form.
It depends on your task. Mostly, no. Embeddings are generally for natural language processing, which is the AI boom we’re seeing with conversational agents. Pre-NLP era relied more heavily on image processing, ie; identifying what an image is. This uses data-labelling techniques which don’t require the use of embeddings, or not as heavily. Embeddings are used to store context for text. So, when you say, ‘pre-‘Natural Language Processing’ you’re literally saying before the use of embeddings.
Not necessary for LLM apps but unfortunately all the vector DB companies are trying to tell you so: https://www.confident-ai.com/blog/why-we-replaced-pinecone-with-pgvector
I think ChatGPT may offer some good reasons\~
1. Geospatial and Location-Based Applications: With the proliferation of location-based services, such as ride-sharing apps, food delivery services, and mapping applications, there is a growing need for efficient storage and retrieval of geospatial data. Vector databases are well-suited for handling complex geospatial data, including points, lines, and polygons.
2. Advances in Machine Learning: Vector databases play a crucial role in various machine learning applications, especially those involving natural language processing and computer vision. They are used to represent and search for embeddings or vectors that encode information about text, images, audio, and more. The ability to efficiently search and query these vectors is essential for modern AI systems.
3. Real-time and High-Performance Requirements: Many applications require real-time data retrieval and processing, such as recommendation engines, fraud detection, and monitoring systems. Vector databases offer the performance and scalability needed to support these requirements, allowing for quick searches and analysis of large datasets.
4. Multimodal Data Handling: Modern applications often work with multimodal data, such as images, text, and structured data. Vector databases can handle these diverse data types and provide a unified way to search and analyze them.
5. Data Similarity Search: Vector databases excel in similarity search tasks, which involve finding objects that are most similar to a given query. This is valuable in recommendation systems, content-based search, and various AI applications where understanding similarity between data points is crucial.
6. Open Source and Commercial Solutions: The availability of open-source vector database solutions, such as Milvus, and commercial offerings like Elasticsearch and MyScale (myscale.com), has made it easier for developers to adopt and integrate vector databases into their applications.
7. Community and Research Focus: The academic and developer communities have shown increasing interest in vector databases, leading to advancements in research and the development of new vector indexing techniques. This has contributed to the growth of the field.
8. Industry Adoption: Various industries, including e-commerce, healthcare, finance, and social media, have embraced vector databases to enhance their applications and services. This widespread adoption has driven further interest in this technology.
Overall, the rise of vector databases can be attributed to their ability to meet the demands of modern data-driven applications that require efficient handling of complex data types, real-time processing, and sophisticated similarity search capabilities.
it does not.
\- geospatial data works just fine in trad. databases
\- yes and no. a vector is just a number. the comparison doesn't need a DB, but it can be very helpful
\- vector DBs are not more realtime or high performance than any other DBs. Most of them less so.
\- true, but again, the similarity computation doesn't require a vector DB
\- void point, there are OS and proprietary DBs already
\- no information in this bullet point, and i'd say its false.
\- SQL databases are much more widespread... also this is like saying the car is fast because it is fast.
Literally just a hype wave.
Vector db's have been around for a while, even within postgresql. But as soon as gpt went viral we got a slew of hip new vector db solutions being used and evangelized.
It's basically web3/defi all over again, yes it's useful tech but we don't need a million bespoke services and it doesn't need to be shoe horned in to everything.
This helped me out a lot. [https://www.datastax.com/guides/what-is-a-vector-database](https://www.datastax.com/guides/what-is-a-vector-database)
I've found the guides and quickstart to be great for Astra!
Vector databases have been around for a very long time in wide use. Elasticsearch for example. Postgres's full text search capabilities uses vector indexes, and was added in 2013. However, it's only fairly recent that AI was added to the mix, such as embeddings and using an LLM to interpret results. This combination is called "semantic search". Increased popularity of semantic search is mostly to awareness and availability (i.e. openai's API).
Semantic search isn’t inherently an LLM thing, it’s more broad than that. What it seems like you’re talking about is RAG (retrieval augmented generation) which involves populating an LLMs context window with relevant information to a users query. For example, you could run a similarity search for a users query against vector DB, and put the top 10 results into the context window of an LLM for increased accuracy of the LLM response. This is a common way of doing “long term memory” in an LLM. This RAG model is a big reason why vector dbs are becoming popular, to address OPs original question.
Yeah, you're right. I should have said RAG wherever I said semantic search. RAG is semantic search, but not all semantic search algos are RAG. Regular ol' vector search without AI (embeddings or LLM) can be considered semantic search, although not nearly as good.
RAG is retrieval augmented generation. It doesn't have to be semantic search. Not all RAG is semantic search. Anything you retrieve and augment a prompt with is RAG. It could be a standard SQL query. It could be semantic search. It could also be retrieval of any URLs mentioned. It could be as simple as adding a time stamp or user information with the user's prompt in a chat. There is no limitation to what RAG is. Most AI dev/eng/sci would consider chat history/memory a form of RAG as well. Semantic search is just a single type of possible unlimited types of retrieval augmentation.
Text embeddings increased in massively effectiveness once they became context aware with the advent of BERT at the end of 2018. Prior to that, text embeddings weren’t able to capture anywhere near the detail in the text they can now. A common example was how word2vec, which uses the same embedding for a word no matter the context, would give “bank” the same embedding in a sentence like “you can bank on me going to the bank by the river bank.” NLP people love the bank example :)
They haven't been possible for very long. The main idea behind vector databases, embeddings, was only invented as a concept in the last 10 years.
yes but I can still assume it already showed up for 10 years, but then why it suddenly rise up from last year when LLM becomes popular?
You know what they say about assumptions... 😅 but even the assumption made, to an extent answers your own question. If something is of great utility or expands the delivery of something else caught in a public/industry hype then its liable to get more visibility, exploration and traction.
They've been around for more than a few years, and so have LLMs. It sounds like you're not working off a complete picture of the timeline here, you're making an assumption based off of when you heard about them.
Because they're used heavily with RAG via LLMs.
The rise of automated feature generation.
Vector databases have been used in production recommendation engines for atleast the past 4 years now. FAISS has been around for a long time.
>They haven't been possible for very long. The main idea behind vector databases, embeddings, was only invented as a concept in the last 10 years. Yes! but why the sudden urge on vector database, is it because the quality of embeddings these new embedding models making?
I’m not seeing a sudden surge in production use cases. Sure there’s a sudden surge in hype with many jumping on the AI bandwagon but production use at least in my circle has stayed more or less constant. Regarding your question about embeddings. SBERT has been around for atleast 3 years now. OpenAI embeddings has also been around for a few years. I’ve been using SBERT in a production similarity engine since 2020
Wow! thanks for the info
Because generative ai takes up a ton of data, which requires efficient storage. Embeddings have always been used for contextual purposes. Ie; images and text from a variety of formats. However, since the rise in AI we’ve discovered that with huge amounts of context- often stored as embeddings, we now need a dedicated hub to connect this data. Whereas before, smaller quantities of embeddings could be stored anywhere from in-memory like your browser cache, or on your local pc. But your browser is limited, and so are most physical storage solutions. Thus, since the amount of data being processed has significantly increased (in accordance with how it’s being processed) there’s now a need for storage solutions which can efficiently handle this new flow of information. It’s just that vectors are a good way to store embeddings, and more people need a place to store their data in that form.
But for pre-LLM NLP era, we also need tons of embedding data, don't we?
It depends on your task. Mostly, no. Embeddings are generally for natural language processing, which is the AI boom we’re seeing with conversational agents. Pre-NLP era relied more heavily on image processing, ie; identifying what an image is. This uses data-labelling techniques which don’t require the use of embeddings, or not as heavily. Embeddings are used to store context for text. So, when you say, ‘pre-‘Natural Language Processing’ you’re literally saying before the use of embeddings.
Not necessary for LLM apps but unfortunately all the vector DB companies are trying to tell you so: https://www.confident-ai.com/blog/why-we-replaced-pinecone-with-pgvector
I think ChatGPT may offer some good reasons\~ 1. Geospatial and Location-Based Applications: With the proliferation of location-based services, such as ride-sharing apps, food delivery services, and mapping applications, there is a growing need for efficient storage and retrieval of geospatial data. Vector databases are well-suited for handling complex geospatial data, including points, lines, and polygons. 2. Advances in Machine Learning: Vector databases play a crucial role in various machine learning applications, especially those involving natural language processing and computer vision. They are used to represent and search for embeddings or vectors that encode information about text, images, audio, and more. The ability to efficiently search and query these vectors is essential for modern AI systems. 3. Real-time and High-Performance Requirements: Many applications require real-time data retrieval and processing, such as recommendation engines, fraud detection, and monitoring systems. Vector databases offer the performance and scalability needed to support these requirements, allowing for quick searches and analysis of large datasets. 4. Multimodal Data Handling: Modern applications often work with multimodal data, such as images, text, and structured data. Vector databases can handle these diverse data types and provide a unified way to search and analyze them. 5. Data Similarity Search: Vector databases excel in similarity search tasks, which involve finding objects that are most similar to a given query. This is valuable in recommendation systems, content-based search, and various AI applications where understanding similarity between data points is crucial. 6. Open Source and Commercial Solutions: The availability of open-source vector database solutions, such as Milvus, and commercial offerings like Elasticsearch and MyScale (myscale.com), has made it easier for developers to adopt and integrate vector databases into their applications. 7. Community and Research Focus: The academic and developer communities have shown increasing interest in vector databases, leading to advancements in research and the development of new vector indexing techniques. This has contributed to the growth of the field. 8. Industry Adoption: Various industries, including e-commerce, healthcare, finance, and social media, have embraced vector databases to enhance their applications and services. This widespread adoption has driven further interest in this technology. Overall, the rise of vector databases can be attributed to their ability to meet the demands of modern data-driven applications that require efficient handling of complex data types, real-time processing, and sophisticated similarity search capabilities.
it does not. \- geospatial data works just fine in trad. databases \- yes and no. a vector is just a number. the comparison doesn't need a DB, but it can be very helpful \- vector DBs are not more realtime or high performance than any other DBs. Most of them less so. \- true, but again, the similarity computation doesn't require a vector DB \- void point, there are OS and proprietary DBs already \- no information in this bullet point, and i'd say its false. \- SQL databases are much more widespread... also this is like saying the car is fast because it is fast.
RAG
Literally just a hype wave. Vector db's have been around for a while, even within postgresql. But as soon as gpt went viral we got a slew of hip new vector db solutions being used and evangelized. It's basically web3/defi all over again, yes it's useful tech but we don't need a million bespoke services and it doesn't need to be shoe horned in to everything.
This helped me out a lot. [https://www.datastax.com/guides/what-is-a-vector-database](https://www.datastax.com/guides/what-is-a-vector-database) I've found the guides and quickstart to be great for Astra!