Token truncation strategies

When a document exceeds the context window, the model stops processing new information. This forces a choice between truncation or selective ingestion. We prioritize keeping the start and the most recent sections of the document.

The default approach to overflow

Standard language models have a fixed limit on the number of tokens they can process in a single pass. If a user uploads a textbook that exceeds this limit, the system cannot simply process the entire file at once. Many applications choose to truncate the end of the text. This is a poor solution for study tools. Students rarely need the middle of a chapter more than the summary at the end. Our current implementation attempts to preserve the structure of the document while discarding redundant whitespace and boilerplate content before passing the text to the model.

Selective summarization

We implement a tiered approach to document ingestion. If the document length exceeds the capacity of the model, we perform a first pass on individual segments. We create a compressed representation of each segment. This process relies on identifying high-density information areas. We avoid arbitrary cut-offs that might split a sentence or a complex definition. By compressing the periphery of the document while maintaining the integrity of the specific paragraphs the user is currently querying, we keep the model focused. This reduces the risk of hallucinations that occur when a model is forced to fill in gaps created by abrupt text truncation.

Constraints on memory

Maintaining state within a limited window requires constant trade-offs. We choose to keep a rolling buffer of the user's chat history at the front of the context window. This ensures the assistant maintains continuity across a long session. The trade-off is that the source document gets pushed further down the stack as the conversation progresses. We monitor this pressure closely. When the source document becomes too thin to be useful, the system triggers a request for the user to specify a new focus area within the text. We do not attempt to store the entire document in the short-term working memory. Instead, we use an index to fetch relevant excerpts on demand. This hybrid approach allows for longer documents without sacrificing the depth of the answers. It remains a balancing act between retrieval accuracy and the inherent constraints of the current model architecture. We continue to refine the threshold for these triggers to minimize the impact on the user experience.