4.7 LLM agents, tool use and evaluation
An agent is a language model called in a loop: it writes a tool call, ordinary code runs the tool, and the result goes back into the next prompt until the model answers or a budget stops it.
01From chat to agent: one full transcript
A chat model answers in one pass: text in, text out. It cannot look anything up, run code or change anything outside the conversation. An agent wraps the same model in a loop. At each turn the model either answers, or asks the surrounding program to run a tool and show it the result, and then it is called again. Before any architecture, here is one complete run of the viewport's first task, exactly as the loop sees it.
System prompt and tools. A short instruction (you are a support assistant, use tools when they help) plus four tool definitions, each a name, one sentence and a JSON schema for the arguments: calculator(expression), search_docs(query, k), lookup(table, key) and send_email(to, body). Together about 310 tokens.
User. Dinner for 7 came to €1,284.50. Add an 18% tip and split it evenly. How much does each person pay?
Model call 1 (335 tokens in). Instead of an answer, the model writes a thought and a tool call. Thought: Add the 18% tip, then split by 7. I will use the calculator. Action: {"tool": "calculator", "args": {"expression": "1284.50 * 1.18 / 7"}}
Runtime. The program, not the model, parses that JSON, checks it against the calculator's schema, runs the tool and appends the result to the conversation as an observation: {"result": 216.53}. (1,284.50 × 1.18 = 1,515.71, and 1,515.71 / 7 = 216.53.)
Model call 2 (372 tokens in: everything above plus call 1 and its observation). Thought: The calculator returned 216.53. Answer: {"answer": "Each person pays €216.53."}. The loop sees an answer instead of a tool call and stops.
The viewport walks the same loop. The packet is emerald at Thought while the model writes , amber while the JSON action travels through the gate to the tool rack, and cyan as the observation comes back. The column in the centre is the context : dark for the system prompt and tools, light for the task, then one plate per model output and per observation. The gauge at the lower right is the step budget , and the panel at the top right is the trajectory . When the run ends, the HUD reads 2 of 6 steps and 757 tokens: the 335 + 372 input tokens of the two calls plus the 32 + 18 tokens the model wrote.
ReAct: Synergizing Reasoning and Acting in Language Models, Yao et al., 2023, Sec. 2
02Function calling: schemas, JSON arguments and validation
Function calling (tool use) is the API contract behind call 1. You send tool definitions with the request: a name, a description and a JSON Schema for the parameters. The model's reply is either text or a tool call: the tool name plus the arguments as a JSON string, with an id. Your code parses the string, validates it, runs the tool and sends the result back in a tool-result message carrying the same id. A model may request several calls in one turn (parallel tool calls) when they do not depend on each other.
The schema for the search tool: {"name": "search_docs", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "k": {"type": "integer"}}, "required": ["query"]}}. The description matters as much as the types: it is the only documentation the model reads, so it should say what the tool does, when to use it and what each field means.
The model writes the arguments token by token, so they can be malformed JSON, miss a required field, use a field name from some other API (expr instead of expression), or put a string where an integer belongs ("k": "3"). Many providers offer a strict or structured-output mode that constrains decoding to the schema; it removes most syntax and type errors, but it cannot make the values right (a well-typed query for the wrong thing is still wrong).
validate_call returns. Try k = true: in Python True is an int, but a JSON boolean is never an integer, a classic validator bug.Make the model call the calculator with the wrong field name, under a budget of 3 steps.
Return validation errors to the model rather than silently dropping the call: a capable model usually fixes expr to expression on the next turn, and it can only do that if the observation says which field was wrong. When it does not, as here, the step budget is what stops the loop (step 4).
Maths, CS & AI Compendium, Henry Ndubuaku, LLM agents and tool use
03The ReAct loop, planning and context
ReAct (reason + act) interleaves reasoning and acting. At step the model reads the context, writes a thought and an action; the environment answers with an observation; the triple is appended and the model is called again.
The viewport now runs the multi-hop task: how much does customer ACME-42 pay per year? No single tool knows. Call 1 runs lookup(customers, ACME-42) and observes {"plan": "Team", "seats": 12}. Call 2 builds its argument from that observation: lookup(plans, Team) returns {"price_per_seat_month": 24}. Call 3 computes 12 * 24 * 12 = 3456, and call 4 answers €3,456 per year. Each argument depends on an earlier observation, so the run cannot be written down completely in advance.
Replay the four-call run and watch the context column.
310 tokens of system prompt and tool schemas, 11 of task, and 100 of history: three calls of 25, 29 and 28 tokens plus observations of 7, 7 and 4. Every call re-reads all of it.
- 1
S: tokens every call carries (system prompt, tools, task). m: tokens one step adds (the model's call plus its observation).
Planning. ReAct decides one step at a time. A plan-and-execute agent first writes the whole plan as a list of steps, then executes it and re-plans when an observation breaks it; that costs fewer model calls when the plan holds and is easier to audit, but it is brittle on tasks like this one, where call 2's argument comes from call 1's result. Reflection adds a critique call after a failure. All three are the same loop with different prompts.
Memory. The context is the agent's short-term memory, bounded by the context window and paid for on every call. Long tasks keep it in check by keeping only the last few turns verbatim, summarising older turns into a short note, or moving facts to long-term memory: a store the agent writes to and later searches with a retrieval tool (the vector search of lesson 4.5).
ReAct: Synergizing Reasoning and Acting in Language Models, Yao et al., 2023, Sec. 2 and 3
The rest of this lesson is in Pro
- 04Failure modes and controls: budgets, retries, timeouts
- 05Least privilege, prompt injection and tool servers
- 06Multi-agent patterns and their cost
- 07Evaluating agents: the trajectory, not only the answer
- Python challenge: Build the agent loop, its validator and its judge
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is the difference between an AI agent and a chatbot?
- A chatbot produces one reply per message from what is in its context. An agent runs the model in a loop and lets it act: at each turn the model can request a tool (a search, a calculation, an API call), the surrounding program runs it and feeds the result back, and the model continues until it answers or hits a limit such as a step budget. The model only writes text; the program around it executes the actions, which is where permissions, retries and limits are enforced.
- How does function calling work in large language models?
- You send the model a list of tools, each with a name, a description and a JSON Schema for its arguments. Instead of plain text the model can reply with a tool call: the tool's name and a JSON string of arguments. Your code parses and validates that JSON, runs the function, and sends the result back as a tool-result message, after which the model continues. Strict or structured-output modes constrain the arguments to the schema, but your code should still validate values and never execute arguments as code.
- How do you evaluate an LLM agent?
- Measure task success on a set of realistic tasks, and also check the trajectory: whether the agent called the right tools with the right arguments, avoided forbidden or unsafe actions, finished within its step and token budget, and grounded its answer in tool results. Run each task several times because agents are stochastic, and report reliability across repeats, cost and latency per task. Use code checks where possible and an LLM judge with a rubric for open-ended parts, calibrated against human labels because judges have position and verbosity biases.
Sources
- ReAct: Synergizing Reasoning and Acting in Language Models, Yao et al., 2023
- Maths, CS & AI Compendium, Henry Ndubuaku
- AWS Certified Generative AI Developer – Professional exam guide, Amazon Web Services
- Developing AI Apps and Agents on Azure (study guide), Microsoft
- OWASP Top 10 for LLM Applications, OWASP Foundation