Over the past three years, Buoyant Ventures has been digging into efficient computing, the intersection of AI and climate where software can help address growing energy and infrastructure constraints. In our last post, we explored why those constraints are making optimization inevitable. We are now looking more closely at the AI inference stack to understand where efficiency creates value and which opportunities can support durable businesses.

We started with context: the information a model uses to answer a question or complete a task. Most conversations about context focus on improving accuracy. We wanted to understand whether better context could also reduce compute (and associated energy use), and whether customers would pay for those savings.

Our research and conversations with builders and operators pointed to a more complicated answer. The efficiency opportunity is real, but enterprise demand is often driven by a more immediate need for governance and data sovereignty: control over what agents can access and share.

This is part one of a three part series on the AI inference stack.

Why Context Matters

Token prices are falling quickly. Epoch AI found that the price per token at a fixed level of performance declined between 9x and 900x per year across the benchmarks it studied. But the cost of a token tells us only part of the story. Longer reasoning, repeated attempts and additional agent steps can consume those savings. The more useful measure is the cost of reaching a correct answer, accounting for both inference cost and the likelihood of success. The cost of a correct answer is also falling, though at a slower pace than tokens. 

Despite this, a model's usefulness depends on the information it receives. For an enterprise agent, that might include internal documents, customer records, live data and the results of earlier actions. Finding the right information, and deciding how much to send, shapes both the quality and cost of its work.

Larger context windows allow models to read more, but they do not guarantee better answers. Chroma's 2025 Context Rot research, which evaluated 18 models, found that performance became less reliable as inputs grew longer. Models given 375x more tokens produced a worse answer. Similar-looking but unhelpful information could also distract models from the answer.

This is where better context can improve the economics of a correct answer. Selecting relevant information and removing unnecessary material can reduce both the tokens processed and the need to try again. The opportunity is to improve accuracy and efficiency together. As token prices continue to fall, context companies will need to show that they reduce the total work required to deliver a reliable result.

Efficiency Opportunities in the Context Layer

Our research organizes the context layer around three functions:

Prepare: Make raw data usable. Extraction tools turn messy PDFs, calls and logs into structured content. Semantic layers define business metrics and relationships so an agent can use the company's definition of “revenue,” for example, without reconstructing it from raw tables.

Select: Find the information needed for each question. Private retrieval surfaces relevant internal documents, while web search brings in current external information. Better selection reduces how much unnecessary material reaches the model.

Manage: Keep an agent's working context useful as a task progresses. This includes compressing prompts and removing stale tool outputs or repeated information from long sessions.

Access and permissions run across all three functions. An agent needs to find useful information while respecting the same restrictions that apply to the person using it. Those controls become especially important when agents connect to multiple company systems.

Together, these functions determine how much work happens before the model responds, and how much work the model has to do itself.

Where the Efficiency Gains Live

The most visible savings often sit closest to a model, by measuring the reduction in number of tokens sent. Those savings are relatively straightforward to demonstrate by comparing the input before and after compression, for example.

But the larger opportunities sit earlier in the process. A well-prepared document can support many later queries. A shared business definition can prevent repeated searches and incorrect database queries. More precise retrieval can keep entire documents out of the prompt. In some cases, a structured lookup can answer a question without a model call.

These benefits are harder to attribute savings to, but they are still important. The most efficiencies can be gained by sending as little as possible to the model. 

‍This is why we focus on cost and time per successfully completed task. Token savings at one step matter only if quality holds and the overall task becomes more efficient. For Buoyant, that is also the connection back to climate: reducing the compute required to deliver a useful result. A lower token bill alone does not establish an equivalent reduction in energy use.

A Crowded Market With Room for Specialists

The need for context capabilities does not automatically translate into demand for standalone context companies. Competition comes from across the ecosystem: 

• Frontier models increasingly build in retrieval, search, caching, memory and compaction.
• Open-source frameworks such as LangChain, LlamaIndex, Haystack and LLMLingua make many capabilities available without a dedicated vendor.
• Data and cloud platforms such as Snowflake, Databricks, MongoDB and Elastic can add parsing and retrieval where enterprise data already lives.
• In-house engineering teams can capture meaningful gains through retrieval and configuration choices alone.
• Application-layer companies such as Buoyant portfolio company HData use context to improve industry-specific products.

‍

At the same time, our research suggests enterprises are wary of putting all their context into one vendor's stack. A company may use a broad platform alongside a specialist that performs better for a particular workflow. That creates an opening for providers whose controls work across models and tools, including environments where sensitive data must remain on company infrastructure.

Market Map

We mapped the context ecosystem across prepare, select, and manage, with cross-cutting categories at the bottom that span multiple functions.

• Extraction: Converts messy files, calls and logs into clean, structured content that agents can use. Threats: data and cloud platforms such as Snowflake and Databricks are bundling document parsing into their own products. Example: Reducto, which turns complex documents like PDFs and spreadsheets into structured, AI-ready data.
• Semantic Layers: Map metrics, entities and relationships so agents query business meaning instead of raw tables. Threats: data platforms like Snowflake, Databricks and dbt offer their own semantic layers. Example: Mobioos, which applies the semantic layer idea to software, mapping a company’s business intent onto its codebase so coding agents understand why code exists.
• Private Retrieval: Finds the internal documents relevant to each query, usually through vector or hybrid search. Threats: the category is consolidating and being absorbed into data platforms, and agents increasingly search files directly without a separate retrieval step. Example: turbopuffer, a serverless vector database built on object storage and pitched on lower cost than traditional vector databases.
• Web Search: Fetches and ranks live web content for an agent’s query. Threats: model providers now include built-in web search in their APIs. Example: Ceramic, which runs its own independent web index of more than 40 billion pages, designed for AI agents rather than human search, with a focus on low-latency, low-cost search for agents that run many searches per task. It’s also the default provider for Cloudflare’s Web Search API.
• Prompt Compression: Shrinks prompts by stripping low-signal tokens before each model call. Threats: frontier models cache repeated prompts natively, compression can break that cache, and open-source tools like LLMLingua offer similar functionality for free. Example: The Token Company, which builds compression models that remove low-signal tokens from prompts before they reach the model, with the compressed result sent to any LLM. 
• Agent Sessions: Cut the waste that builds up as an agent works, such as repeated file reads, loops and stale tool outputs. Threats: model platforms are adding native compaction, and agent frameworks handle subagents themselves. Example: Parsec, a context-curation layer for coding agents that scores every chunk of an agent’s trace for whether it will be needed later and cuts the rest. In its own published benchmark, Parsec was the only layer tested that both saved cost and solved more tasks than the baseline. 
• Enterprise Context Platforms: Connect, organize and retrieve across all of a company’s data, spanning Prepare and Select. These companies often add layers of governance. Threats: data platforms and incumbents are expanding across the same functions, and frontier models are building their own enterprise connectors. Example: Miriel, which connects structured and unstructured data, including documents, databases, video and sensor streams, so agents can answer questions across all of it, with permissions enforced at the data layer.
• Access & Permissions: Connect agents to company apps and control what they can see and do, at every step. Threats: model providers are shipping their own connectors, and identity providers are adding agent permissions. Example: Arcade, which handles OAuth authentication, token management, permission enforcement, tool execution and audit logging so agents can safely take actions in systems like Gmail, Slack, GitHub and Salesforce. 

The Opportunity Ahead

Better context can help AI complete useful work with less compute. 

But put simply: we went looking for efficiency and found governance. Customers may buy first for control and visibility, creating the foundation to measure and eventually prioritize efficiency.

The open question is how much of that value independent companies can retain as models and platforms absorb more of the stack. We are looking for businesses that can demonstrate better economics today and give customers a reason to stay as the technology evolves.

-

This is the first post in our AI inference stack series. Next, we will explore Routing. 

Are you building in context, governance or efficient AI infrastructure? We'd love to hear from you!

A big thank you to our research associate, Sarah Devermann, for her work on this research, and to the experts and companies who shared their perspectives and helped shape our thinking.