How engines work
Training data
Training data is the body of text a model learned from. It sets the model's default understanding of a category and which brands it associates with what, before any live retrieval happens.
In more detail
Because training happens in batches, information that entered the web after a model was trained reaches answers through retrieval rather than through the model itself.
Guides that cover this
Related terms
Knowledge cutoff
A knowledge cutoff is the date after which a model's training data ends. Anything that changed after that date reaches an answer only if the engine retrieves it live at the time of asking.
Large language model
A large language model is a system trained on very large amounts of text to predict and generate language. It is the component that writes an answer engine's response, drawing on what it learned in training plus anything retrieved at the moment of asking.
AI crawler
An AI crawler is an automated agent that collects web pages for an AI company, either to inform model training or to build the search index an assistant retrieves from. Each is identified by its own user-agent string in robots.txt.
Ready to Dominate AI Search?
Get started now
Allswap