A Production Pipeline for IChatClient: Microsoft.Extensions.AI Middleware in Order

Most .NET AI code starts with a single IChatClient and a call to GetResponseAsync. The part that decides whether it survives production is everything wrapped around that call: caching, tool invocation, tracing and logging. Microsoft.Extensions.AI gives you a builder for exactly this, and the order you add the pieces changes behaviour.

This post builds one small pipeline, explains the order, and notes what the 10.x releases added. I compiled and ran the code below against Microsoft.Extensions.AI 10.11.0, the newest version I could restore on November 7, using a stand-in model so it needs no keys.

The pipeline

using Microsoft.Extensions.AI;
using Microsoft.Extensions.Caching.Distributed;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;

IChatClient inner = /* your provider's client, adapted with AsIChatClient() */;

using var loggerFactory = LoggerFactory.Create(b => b.AddSimpleConsole());

IChatClient client = new ChatClientBuilder(inner)
    .UseDistributedCache(new MemoryDistributedCache(Options.Create(new MemoryDistributedCacheOptions())))
    .UseFunctionInvocation()
    .UseOpenTelemetry(loggerFactory, sourceName: "blog.ai")
    .UseLogging(loggerFactory)
    .Build();

var options = new ChatOptions { Tools = [AIFunctionFactory.Create(GetOrderStatus)] };
var reply = await client.GetResponseAsync("Where is order A-1001?", options);
Console.WriteLine(reply.Text);

[System.ComponentModel.Description("Gets the status of an order.")]
static string GetOrderStatus(string orderId) => 
quot;Order {orderId} has shipped.";

ChatClientBuilder wraps the client in the order you call it, and the first call becomes the outermost layer. A request passes through the layers top to bottom, and the response comes back the other way.

Why this order

• Cache first (outermost). A cache hit returns before anything else runs, so no model call, no tool call and no trace for a repeated question. Put it lower and you would still pay for function invocation on every hit. Use a distributed cache in production so every instance shares answers.
• Function invocation next. This layer sees the model ask for a tool, runs your AIFunction, sends the result back and loops until the model answers. Everything below it sees each of those model calls separately, which is what you want for tracing.
• OpenTelemetry below that. Each individual model call becomes its own span. If you place it above function invocation you get one span for the whole exchange, which hides how many round trips a single question cost.
• Logging last. It sits closest to the model and records what actually went over the wire.

If a trace looks wrong, the first thing to check is the order of your Use calls.

Two things to decide on purpose

Caching and non-determinism. Caching only makes sense when the same input should give the same answer. Do not cache calls that depend on the current time, the signed-in user, or live data returned by a tool. Cache the layer you can defend, and consider separate pipelines for cacheable questions and tool-driven ones.

What you log. UseLogging and the tracing layers can record prompts and responses. Treat that as personal data and decide what is allowed in logs before you turn it on in production. The OpenTelemetry GenAI conventions are also still marked as in development according to the sources I read, so expect attribute names to move between versions.

What the 10.x releases added

From the release notes quoted in a downstream pull request:

• 10.4.0 added hosted file, web search and reasoning content types, stabilised the MCP and tool approval APIs, and added streaming latency metrics to the OpenTelemetry instrumentation.
• 10.4.1 added experimental realtime client sessions and text to speech.
• 10.11.0 is the version I used. I did not find release notes for it, so I am not claiming specific changes.

The tool approval API is worth knowing about now. It is the building block that lets an agent pause before a risky call. I use it in my post on human approval for agent tools.

A quick test you can run

Because everything is behind IChatClient, you can test the pipeline with a fake client that counts how often it is called. I ran two checks that way on 10.11.0:

• Two layers added with Use, each recording its name. The calls ran first-added, then second-added, then the model, so the first layer you add is the outermost.
• The distributed cache layer in front of a counting fake, three requests where two were identical. The fake was called twice, not three times, so the repeated question never reached the model.

Counting inner calls is the cheapest way to prove your order is right, and it fits in a unit test.

What I could not verify

I ran the pipeline against a stand-in model, not a real provider, so I have not measured latency or token use. The compiled code and its output are real; the cost and quality of a real model are outside what this post tested.

Sources

• Microsoft.Extensions.AI.Abstractions on NuGet (10.11.0)
• Release notes for 10.4.0 and 10.4.1, quoted in a dotnet-presentations pull request
• OpenTelemetry client builder extensions, source
• OpenTelemetry GenAI semantic conventions, practical guide