Back to basics: Artificial intelligence and states

When using an artificial intelligence model from a frontier lab like Anthropic or OpenAI, the experience can be seamless and it feels as if those massive black boxes, like Claude Opus 4.8 or GPT 5.5, can do so many things. The models can feel really personal and it is impressive that they can remember things said previously in a conversation or “do” tasks like send an email.

The truth is that those models aren’t the ones making all of this possible. This a clever illusion created by the software infrastructure built around them. The infrastructure is the software that enables a large language model (LLM) to be useful. You could take Fable 5, remove everything around it and you’ll see quickly that it is just something that answers the message you sent which is nothing much.

In this article, we’ll talk about that infrastructure around the models. How it enables a stateless AI to do what we assume they should be able to do today. We’ll also address some technical constraint the software has to account for and how it works around so the user experience doesn’t diminished over time.

Stateless models problem

As I mentioned above, AI models are stateless, but what does it mean? To answer this, we will have to understand what are large language models fundamentally and what you have when you only have it.

We can see models like enormous statistical functions. We’ll call that function f(x) where x is a user input that it needs to treat. When given an input, it outputs human readable text. This is basically a high level view of what an LLM is. f(x) doesn’t remember what was the input you gave before nor what it generated as an output.

This is made even worse with the fact that f(x) is non-deterministic, meaning you can give the same value for x over and over again and the function will output something different every time. This is great for it to feel “natural”, but it can range from annoying to really bad for everything else. This also means that f(x) can’t assume that, if you gave x = "What is the temperature of the sun?", its output was "5 800 K" or "10 000 °F" or "That is a great question! The temperature of the sun is the same as your office when you're put in a stressful situation.". f(x) takes an input, generates an output and that is it.

This is what is meant when we say that a model is stateless. It means it is mostly unusable, so it falls on the infrastructure around to actually make it usable and “remember” things. This is called making it stateful.

Solving the state problem

Now we know that it isn’t the model that solves the state issue, but the software infrastructure around it. We might think that it is a complicated solution that requires complex software to be put in place, but in practice, it is a simple solution that works.

Let’s say we have a conversation with the model. This is the classic way of using it, but we established that f(x) does not remember anything every time we pass it a new input. To illustrate how to solve this, we’ll use our example from earlier:

1. User: x = "What is the temperature of the sun?"
2. AI: f(x) -> "5 800 K"
3. User: x = "Ok, but what is this in °C?"

This is where we reach our problem. Our latest user input references an earlier message, but the AI is completely clueless about it. To solve this, we basically have to make x the whole conversation every time you call f(x). It was built into the model that it will receive this sort of turn-by-turn conversation pattern as an input so it’ll answer the last message.

Inserting a whole conversation again and again would be tiresome for the user, so this falls on the infrastructure acting as the middle man between the AI and the user interface. This makes the rest of the sequence:

1. Infrastructure: x = {User: "What is the temperature of the sun?", AI: "5 800 K", User: "Ok, but what is this in °C?"}
2. AI: f(x) -> "5526.85 °C"

What we give as an input on top of the actual input, the history, is called the context window of the model. Everything a model needs to know to create the right output must be in that context if it wasn’t “trained” into the model itself. Every file contents, messages in a conversation, ways to tell the infrastructure to do something its behalf etc is put in there.

The context window

Now that we’ve established what the context window is, we can see an obvious problem with it. From the user perspective, it can grow exponentially and it doesn’t really have a limit. To understand why this is a problem, we’ll have to understand what a token is.

The token is how we can quantify the data being sent as an input and the data being outputted (which becomes part of the next input). Each input and outputs is cut down into those tokens and, since we accumulate them over time, it means our input is always bigger and bigger.

Tokens are also the way we quantify the size of the context window. A bigger context window means more tokens can be used as context, but it also means the model will allocate more system memory to fit the maximum possible context window. Memory isn’t a plentiful resource on most systems so we can’t have an infinite context. It’ll be capped.

Context management

So this is the problem: an AI needs a big context window to “remember” things, which means it needs a lot of tokens and this takes a lot of system memory. We can’t have unlimited memory, so we can’t have unlimited context windows.

This brings the responsibility of managing that context on the infrastructure. If we have a 32000 tokens context window, we can’t give it 34000 tokens in context. The infrastructure needs a way to keep the context under the maximum number of tokens and two ways to do this exist. Make a sliding window or do rolling summaries.

Sliding window

The sliding window is the simplest of both solutions. The idea is that older messages in the context are probably not needed for the model to answer your next input, so we can forget them.

For creating the sliding window, the infrastructure will not fetch the whole conversation for every new input. It’ll just fetch N messages from it. If N = 5, this means that the context created will always be the five latest messages.

See full picture by clicking here ↗

This is great since it is really simple to implement and the assumption we make is somewhat true for most use cases. A user probably doesn’t need the model to “remember” everything since that user probably won’t reference the first ever message in the conversation. Also, the model should be able to make up what the conversation is about with context clues from the given context.

When we do long running tasks where we need the model to remember everything or almost, this is where the sliding window fails for the same reason it works for most use cases. The context is constantly cut and the model has no idea what happened before the context it was given.

Rolling summaries

For long tasks where the model needs a big context, but the resources aren’t enough to support it, rolling summaries are better. With it, we want to keep as much of the context, while cutting down all the unnecessary details that inflates the token count.

To achieve this, at a certain point, the infrastructure will ask the model to summarize the conversation before giving it the newest user input. The generated summary will be used as the new start of the context instead of fetching the whole conversation again.

See full picture by clicking here ↗

This works since the model will summarize maybe 30000 tokens into 2000 tokens, meaning we saved 28000 tokens in context. If the summary is good, we should have lost almost nothing in actual context we would want the model to remember. This method is the best of both worlds when it comes to managing long contexts. There’s only two small trade offs with rolling summaries.

The first is that it is done automatically by the infrastructure when it sees it might go over context. This means we have to do two requests to the model to treat one. Normally, on a server dedicated to this in a data center, it probably won’t be an issue, but it will considerably increase response times, on an already slow request, for local computers that don’t have that kind of power.

The second is that a problem comes from the nature of a summary. It is just a summary. This means we’ll loose some of the finer details of the conversation unless we do a detailed summary, but a detailed summary will save less tokens. A good rolling summary needs to strike a balance between the two.

Conclusion

As we saw in this article, the infrastructure around an AI model does a lot of heavy lifting for the model to be usable. A lot of the responsibilities it has are on managing the state and the context of the model.

They don’t always need to be bigger and smarter for them to become more usable or “better” if the infrastructure gives it the right context for the job it needs to do. Also, it will not just be new models, but those software that will probably be what pushes AI more and more locally instead of leaving it in data centers.