What an agentic AI workshop for developers using LangChain should cover
Most teams arriving with LangChain in production already know the API surface. They have read the docs. They have a working prototype. What they do not have is a way to tell whether a change made the system better, and that is where the whole day should go.
So we spend the morning on evaluation and the afternoon on failure. Tutorials on composing chains are free and plentiful. Nobody needs to pay two trainers to read documentation aloud.
Evaluation first, because everything else depends on it
Without a test set you are tuning by anecdote. The fastest path we know:
- Collect twenty to fifty real inputs from your logs. Real ones, including the ugly ones users actually sent.
- Write the expected outcome for each by hand. Painful, and there is no shortcut.
- Score with the cheapest check that works. Exact match or a regex where possible, a model grader only where output is genuinely open-ended.
- Run it in CI. A prompt change is a code change and deserves the same gate.
Teams resist the hand-labelling. Then they change a system prompt, feel that it improved things, and ship a regression on a case class they forgot about. The set pays for itself in about two weeks.
Failure handling is the real engineering
An agent loop in production fails in ways a demo never shows. Tool calls that return malformed arguments. A retrieval step that returns nothing and a model that answers anyway. Loops that run until a token budget dies. Timeouts on a downstream service that surface as a confident, empty answer.
We work through each on the team's own code. Hard iteration caps. Structured output validation with a schema, so a bad tool call fails loudly instead of flowing downstream. A distinct path for the empty-retrieval case, because "I do not have that information" is a valid product response and most implementations never emit it.
Tracing belongs here too. If you cannot see the exact sequence of calls for a bad production response, you are debugging blind. Whether you use LangSmith or plain structured logging matters less than having the trace at all.
When to drop the framework
We say this openly, and it is not a criticism of LangChain. A good number of production systems we see would be simpler as direct API calls plus a small amount of your own orchestration. The framework earns its place when you genuinely swap providers, or when you need the integrations and the tracing.
The signal to reconsider is debugging cost. When your team spends more time reading library internals than reading their own logic, the abstraction is charging more than it delivers. We run that assessment on real code during the session and sometimes the honest answer is: keep it, you use it properly.
What to do next
Before booking anything, pull thirty real inputs from your logs and label them. Do it this week. If you already have that set, tell us and we will skip the morning and spend the whole day on failure handling and cost. If you do not, building it together is the single highest value thing we can do with your team.