Why LLMs without tool calling capability can call tools?

2026-05-062026-05-17

LLM

OGP Why LLMs without tool calling capability can call tools?

Tool calling looks like a model feature, but the model only ever predicts the next token. The "capability" is really two things that live outside the model: a trained output pattern and a runtime that parses it. That's why an LLM without native tool calling can still call tools.

With native tool calling

Pass tool definitions to the provider; they are flattened into a special system prompt.
Invoke the LLM.
The model emits trained tool-call tokens (e.g. <|python_tag|>{...}<|eom_id|>); the runtime parses them and returns stop_reason: "tool_use".
Execute the tool on your server.
Send the result back as a tool result and re-invoke.

Some providers go a step further and run tool execution themselves — for example, Anthropic's web search tool. You declare it, but the API performs the searches and the model returns a final cited answer in a single request, with no client-side tool_use → tool_result round-trip.

Web search tool

Example

The Anthropic Messages API takes tools as a top-level tools array. Each tool has a name, a description, and a JSON-schema input_schema:

json

{
  "model": "claude-sonnet-4-20250514",
  "max_tokens": 1024,
  "tools": [
    {
      "name": "get_weather",
      "description": "Get current weather for a city. Use this whenever the user asks about weather, temperature, or conditions in a specific location.",
      "input_schema": {
        "type": "object",
        "properties": {
          "city": { "type": "string" },
          "unit": { "type": "string", "enum": ["c", "f"], "default": "c" }
        },
        "required": ["city"]
      }
    }
  ],
  "messages": [
    { "role": "user", "content": "What's the weather in Tokyo?" }
  ]
}

When the model decides to use the tool, the response ends with stop_reason: "tool_use" and contains a structured tool_use content block. You execute the tool and reply with a matching tool_result content block in the next turn.

Without native tool calling

Describe the tools in the prompt and instruct the model to emit a fixed pattern (e.g. <tool>...</tool>).
Register the closing tag as a stop sequence — optionally enforce the schema with constrained decoding.
Invoke the LLM. Decoding halts when the closing tag appears; finish_reason is just "stop".
Detect the pattern in the text, parse it, and execute the tool.
Send the result back in the next prompt and re-invoke.

The two flows differ only in steps 2–3, and both differences live in the runtime — not in the model.

The diagram puts pattern detection in the app, but with a hosted provider's stop parameter that detection moves into the provider runtime — the same mechanism as native tool calling, just on a custom string instead of a special token. Either way, the boundary lives outside the model.

Example

A minimal prompt that lets any model — even one without native tool calling — call tools. The same XML-tag convention is widely used in the open-source ecosystem and recognized by major inference servers out of the box.

System prompt

markdown

You are an assistant that can answer directly OR call tools.

<tools>
[
  {
    "name": "get_weather",
    "description": "Get current weather for a city. Use this whenever the user asks about weather or conditions in a specific location.",
    "parameters": {
      "city": { "type": "string" }
    }
  }
]
</tools>

## Rules

- If a tool is needed, output ONLY: <tool_call>{"name": "...", "arguments": {...}}</tool_call>, then stop.
- Otherwise, answer the user in plain text.
- After a <tool_result>...</tool_result> message, use the result to answer in plain text.

## Examples

User: How is the weather in Tokyo today?
Assistant: <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>

User: What's the weather like in Paris?
Assistant: <tool_call>{"name": "get_weather", "arguments": {"city": "Paris"}}</tool_call>

User: What is weather?
Assistant: Weather is the state of the atmosphere at a particular place and time — temperature, humidity, wind, precipitation, and so on.

What actually differs

Both flows produce the same outcome. The real difference is who owns the boundary and who guarantees the format:

Native — the runtime detects a special tool-call token, and fine-tuning guarantees the format.
Non-native — your app detects a stop sequence, and the prompt (plus optional constrained decoding) guarantees the format.

Add constrained decoding to the non-native path and the two converge in reliability — what remains is just where the boundary lives.

Strip everything else away and tool calling is one loop: prompt → pattern → stop → execute. Whether you wire it up yourself or hand it to a provider's runtime is an implementation detail. AI agent framework SDKs have become the default, and they absorb this abstraction so completely that it stops feeling like an abstraction at all — which is exactly why the boundary between "native" and "non-native" has gotten so blurry.