Awhile ago I’d written about how traditional search and newer Large Language Models (LLMs) handle representing text so they can do their job. (LLM / Text Vectors for Product Managers) It was heavy on theory, but I’m not sure the examples were as good as they could be. So I’m trying again.
The following examples let you step through the process for Search, and then for LLMs. Because of the screen size of the demo, some of it may overlay parts of the blog design and some parts may require a little scrolling. This is as intended to get the desired presentation on the page.
Part 1: TF-IDF Vectorization
Watch how a word gets converted into a TF-IDF vector through a step-by-step calculation process.
You’re going to want to use a word that’s in the mini text corpus below, so stick with the default for now.
Part 2: LLM Embedding Vectorization
See how a neural network transforms a word into a dense embedding vector through learned representations.