Add comprehensive documentation and templates for LangChain skill development
- Introduced a new reference document for streaming output issues, detailing the differences between streaming APIs and providing solutions for common problems. - Created a structured output issues reference, outlining the use of `with_structured_output`, `response_format`, and schema enforcement strategies. - Added a user query convention guide to standardize structured questions for skill authors, including block types for queries and data gathering. - Implemented a template for a chat model, encapsulating API key management, request payload construction, and response handling. - Established a symlink for the LangChain dev guide in the Claude skills directory for easier access. - Initialized a skills lock file to manage dependencies and versions for the LangChain dev guide.
This commit is contained in:
yuxuanhui
@@ -0,0 +1,137 @@
|
||||
# Structured Output Issues
|
||||
|
||||
A focused reference for `with_structured_output`, `response_format`, and provider-side schema enforcement.
|
||||
|
||||
## Choose the API path first
|
||||
|
||||
LangChain exposes two related but different structured-output APIs. Diagnose and configure the one the application actually uses:
|
||||
|
||||
- **Model wrapper — `model.with_structured_output(...)`**: the model integration selects a steering method such as `json_schema`, `function_calling`, or `json_mode`, then parses the model response into the requested schema. Defaults vary by integration and version, so set `method` explicitly when behavior matters.
|
||||
- **Agent — `create_agent(response_format=...)`**: passing a schema directly lets the agent select a strategy using model capability metadata plus LangChain fallback and tool-compatibility checks. Both strategies parse and validate structured responses; `ToolStrategy` can retry configured structured-output errors, while `ProviderStrategy` surfaces provider or schema validation failures. These strategies are not aliases for the model wrapper's `method` argument.
|
||||
|
||||
Use the automatic agent strategy when the model profile is accurate:
|
||||
|
||||
```python
|
||||
agent = create_agent(
|
||||
model=model,
|
||||
tools=tools,
|
||||
response_format=MySchema,
|
||||
)
|
||||
```
|
||||
|
||||
Force a strategy only when provider capability is known and the automatic choice is unsuitable:
|
||||
|
||||
```python
|
||||
from langchain.agents.structured_output import ProviderStrategy, ToolStrategy
|
||||
|
||||
native_agent = create_agent(
|
||||
model=model,
|
||||
tools=tools,
|
||||
response_format=ProviderStrategy(MySchema),
|
||||
)
|
||||
|
||||
tool_agent = create_agent(
|
||||
model=model,
|
||||
tools=tools,
|
||||
response_format=ToolStrategy(MySchema),
|
||||
)
|
||||
```
|
||||
|
||||
## Issue 1: Unstable structured output, often returns None
|
||||
|
||||
- **Symptom**: `model.with_structured_output(schema)` returns `None`, an empty object, missing fields, or a parsing error. The same prompt can work on one run and fail on the next. Small, open-source, or quantized models tend to be less reliable with complex schemas.
|
||||
- **Cause**: The cause depends on the configured method and provider capability:
|
||||
- With provider-native structured output, the provider may reject an unsupported schema or model before generation.
|
||||
- OpenAI-family `with_structured_output(..., method="function_calling")` binds the schema as a tool and **forces the schema tool** by passing its name as `tool_choice`. When the provider supports and honors forced selection, the model is not free to answer in prose instead.
|
||||
- If an adapter strips or ignores `tool_choice` — for example through `disabled_params={"tool_choice": None}` — the model is again **free to skip the schema tool**. A natural-language answer then gives the tool parser nothing to deserialize, so the parsed result can be `None`.
|
||||
- A returned tool call can still fail schema validation because its arguments are malformed, incomplete, or have the wrong types.
|
||||
- **Solution**:
|
||||
|
||||
1. **Inspect the raw response before changing methods.** `include_raw=True` separates "no schema tool call" from "tool call failed validation":
|
||||
|
||||
```python
|
||||
structured_model = model.with_structured_output(
|
||||
MySchema,
|
||||
method="function_calling",
|
||||
include_raw=True,
|
||||
)
|
||||
result = structured_model.invoke(prompt)
|
||||
|
||||
print(result["raw"].tool_calls)
|
||||
print(result["parsed"])
|
||||
print(result["parsing_error"])
|
||||
```
|
||||
|
||||
2. **Choose a model-level method by documented provider capability.** There is no universal fallback order that works for every integration:
|
||||
|
||||
- Prefer provider-native `json_schema` when the selected model and provider document schema-constrained output:
|
||||
|
||||
```python
|
||||
structured_model = model.with_structured_output(MySchema, method="json_schema")
|
||||
```
|
||||
|
||||
- Use `function_calling` when the provider supports tool calling and forced `tool_choice`. Keep `with_structured_output` so LangChain both binds the schema tool and attaches the output parser:
|
||||
|
||||
```python
|
||||
structured_model = model.with_structured_output(
|
||||
MySchema,
|
||||
method="function_calling",
|
||||
)
|
||||
```
|
||||
|
||||
Calling `bind_tools` directly returns an `AIMessage` with tool calls; it does not provide the schema parsing performed by `with_structured_output`.
|
||||
|
||||
- Use `json_mode` when the provider guarantees JSON syntax but not schema conformance. Spell out field names, types, and constraints in the prompt, then validate and retry in application code:
|
||||
|
||||
```python
|
||||
structured_model = model.with_structured_output(MySchema, method="json_mode")
|
||||
```
|
||||
|
||||
3. **Configure agents through agent strategies.** For `create_agent`, use a direct schema for automatic selection, `ProviderStrategy` for known native support, or `ToolStrategy` for tool-based structured output and its validation-retry loop. Do not diagnose `create_agent(response_format=...)` solely through the model wrapper's three `method` values.
|
||||
|
||||
- **Lessons learned**:
|
||||
- Check `raw.tool_calls` and `parsing_error` before deciding whether the failure is tool selection or schema validation.
|
||||
- Provider-native schema enforcement is usually the strongest option when the exact model supports it, but capability must be verified rather than inferred from OpenAI-compatible transport alone.
|
||||
- Weak model + complex schema is the most failure-prone combination. Split large schemas into smaller calls when possible.
|
||||
|
||||
## Issue 2: `with_structured_output(..., method="function_calling")` fails because the model does not support `tool_choice`
|
||||
|
||||
- **Symptom**: Calling `model.with_structured_output(schema)` or `model.with_structured_output(schema, method="function_calling")` raises a provider-side 400 error like "`<model>` does not support this `tool_choice`". This often happens with `ChatOpenAI`-style integrations or derived classes such as `ChatDeepSeek` against OpenAI-compatible providers. A typical example is `deepseek-v4-flash` in thinking mode: it supports tool calling but rejects forced `tool_choice`.
|
||||
- **Cause**: For `function_calling`, LangChain's OpenAI-family chat models try to make structured output more reliable by forcing the schema tool to be called. Internally, `with_structured_output` binds the schema as a tool and usually passes `tool_choice=<schema_tool_name>`. That works for providers that support forced tool selection, but some OpenAI-compatible backends only allow free-form tool calling and reject explicit `tool_choice`. When you access those models through `ChatOpenAI`, `ChatDeepSeek`, or another subclass inheriting the same behavior, the adapter forwards `tool_choice` and the provider errors before generation starts.
|
||||
- **Solution**: Disable forwarding of `tool_choice` for that model instance, so LangChain still binds the schema tool but does not send the unsupported parameter:
|
||||
|
||||
```python
|
||||
from langchain_deepseek import ChatDeepSeek
|
||||
from pydantic import BaseModel
|
||||
|
||||
class User(BaseModel):
|
||||
name: str
|
||||
age: int
|
||||
|
||||
model = ChatDeepSeek(
|
||||
model="deepseek-v4-flash",
|
||||
disabled_params={"tool_choice": None},
|
||||
extra_body={"thinking": {"type": "enabled"}},
|
||||
)
|
||||
|
||||
structured_model = model.with_structured_output(
|
||||
User,
|
||||
method="function_calling",
|
||||
)
|
||||
|
||||
print(structured_model.invoke("My name is John and I am 25 years old."))
|
||||
```
|
||||
|
||||
Why this model-wrapper workaround works: OpenAI-family `with_structured_output(..., method="function_calling")` calls `_filter_disabled_params(...)` before binding the schema tool. Setting `disabled_params={"tool_choice": None}` strips the forced selection from this model-wrapper path.
|
||||
|
||||
This setting does not configure agent `ToolStrategy`. Stock `create_agent(response_format=ToolStrategy(...))` still forces structured-tool use through `model.bind_tools(..., tool_choice="any")`, and ordinary `bind_tools` does not consult `disabled_params`. If an agent's provider rejects forced tool selection, use a documented native `ProviderStrategy`, a compatible model or integration, or an adapter that explicitly handles this provider limitation.
|
||||
|
||||
Removing forced selection restores compatibility but also makes the model free to skip the schema tool. Use `include_raw=True` to detect that case and choose another method by capability:
|
||||
- If the provider documents native schema-constrained decoding for this model, use `method="json_schema"`.
|
||||
- If it supports only JSON syntax enforcement, use `method="json_mode"`, describe the schema in the prompt, and validate plus retry in application code.
|
||||
- If neither is available, keep unforced `function_calling` only with an explicit retry or repair path for missing tool calls.
|
||||
- **Lessons learned**:
|
||||
- This failure mode is not "the schema is wrong" and not "tool calling is unsupported" in general. The narrower issue is that the backend rejects forced tool selection via `tool_choice`.
|
||||
- `ChatOpenAI` compatibility is only a transport-level starting point. Once you connect it to non-OpenAI providers, verify which OpenAI request parameters they actually support, especially for reasoning models.
|
||||
- When a provider says "`... does not support this tool_choice`", the fastest fix is usually model-level configuration (`disabled_params`) rather than patching LangChain source code.
|
||||
- Removing `tool_choice` trades reliability for compatibility. If outputs start coming back as `None`, see Issue 1 and switch to `json_schema`, `json_mode`, or an explicit retry path based on provider capability.
|
||||
Reference in New Issue
Block a user