Tool calling

Tool calling (also called function calling) is a model's structured output capability where it produces JSON-shaped tool invocations instead of free-form text when the use case calls for action. The model sees a list of available tools (with JSON schemas), decides which to call, and emits {"name": "search_web", "args": {"query": "..."}}. The runtime parses, executes, and feeds the result back as a new user-role message.

What tool calling enables: agents (multi-step reasoning + action loops), structured extraction (forcing the model to emit JSON conforming to a schema), MCP clients (the Model Context Protocol exposes tools as a standard interface). Modern open-weight models with strong tool calling: Qwen 2.5 Coder, Llama 3.3, DeepSeek V4, Mistral Small 3 — all train on tool-using corpora and emit tool calls reliably.

Operator caveats that matter: (1) tokenizer alignment — some quantization formats subtly damage tool-call output structure; verify your AWQ/GGUF quant produces clean JSON before committing. (2) temperature — keep ≤0.4 for tool-calling agents; >0.6 causes JSON parse errors as the model invents tool names. (3) runtime parser — vLLM's tool-call parsing was buggy until 0.6.x; SGLang shipped it cleanly later. (4) schema complexity — large JSON schemas burn KV cache; keep tool definitions terse.

An operator wiring up a local research agent with Qwen 2.5 Coder 32B behind an MCP server notices the model occasionally emits malformed JSON for tool calls — a missing closing brace, or a hallucinated tool name close to but not matching the schema. Two changes fix most of it: dropping temperature from 0.7 to 0.3 (high temperature was letting the model improvise tool names instead of copying them exactly from the schema), and re-quantizing from a Q3_K_M GGUF up to Q5_K_M, since the more aggressive quant had been subtly corrupting the structured-output head. They also trim the tool schema definitions from verbose multi-paragraph descriptions down to one-line summaries, since every tool definition sits in the context window on every single turn and was eating into the effective budget available for actual reasoning and retrieved documents.