Build a workflow from small decisions

Draw the decisions before the calls

Sketch the application before choosing a pattern. For a return request, code knows the order date and return window. The message supplies less tidy evidence: whether the customer wants a replacement, whether an item is damaged, and which order they mean.

Mark where interpretation is required. Then ask whether those judgments share their evidence, whether they need different evidence, and what the application should do with an uncertain result. The sections below work through those choices.

Speculative fan-out

A customer says “A replacement would be great; otherwise I’d like a refund.” You can ask about the requested remedy and the details of each remedy using the same message and order.

Write each conditional question with its premise included: “If the customer wants a replacement, which item are they referring to?” The request does not pass the returned route answer into the other question. Keep the premise explicit even on a joint schema model.

After the answers arrive, code selects a route and uses that route’s follow-ups. Questions for other routes may be irrelevant; their uncertainty should not block the chosen path.

This can save a repeated copy of the state and a network trip. It also asks questions that may go unused. Compare those costs for your request sizes instead of treating extra questions as free.

Composite scoring

You have six returns to review and only time for two. The order may depend on evidence quality, policy fit, account risk, and handling effort. Store each judgment separately so those priorities remain adjustable.

A single overall score is enough when you only need that one judgment. Use separate dimensions when you need to explain or change how the ranking is assembled.

Normalize the scales, choose weights, and sort in code. A weighted sum allows one strong factor to compensate for a weak one. Apply mandatory eligibility checks separately so a good score cannot cancel a disqualifying condition.

Confidence-gated cascade

Some returns can be routed from the first message. Others need an attachment, a follow-up, or a reviewer. A cascade gives those unresolved cases a next destination.

Keep the first stage small. Send cases that fail its tested acceptance rule to a stage with useful additional evidence or capability. Calling a larger model with the same missing facts may not resolve the problem.

Count the cost and delay of every stage reached. The lab assumes each stage resolves a chosen share of the cases it receives. The shares are inputs, not predictions.

Select, do not generate

The customer mentions two order numbers but wants to return only one. A parser can find both numbers; interpreting which one the customer means is a different task.

Collect candidate values in code, offer them as Choice options, and keep an option for no match. Once a candidate is selected, use the original value rather than asking a model to reproduce it.

Test candidate collection before testing selection. A missing candidate cannot win. Even a correct selection still needs authorization before you change the associated order.

import re
from typesafe_sdk import Choice, TypeSafeClient

message = "Returning the jacket from order R-20931. My other order R-20877 is fine."
candidates = re.findall(r"\bR-\d{5}\b", message)          # ["R-20931", "R-20877"]

criteria = dict.fromkeys(candidates)               # every candidate, no description
criteria["none"] = "No order number in the message is the one being returned"

with TypeSafeClient() as client:
    r = client.system_one(
        state={"message": message},
        questions={"which": Choice(
            instructions="Which order number does the customer want to return?",
            criteria=criteria,
        )},
    )

picked = r.choices["which"]
order_id = None if picked.choice == "none" else picked.choice  # copied, never retyped

For a large catalogue, you can narrow candidates in stages. Keep alternatives when an early branch is uncertain, and measure whether the extra paths improve your final selections.

Reuse the evidence within a request

Suppose a return message needs six independent checks. Six separate calls repeat the message six times. Jev and Kev can include all six checks over a shared state prefix. Clef scores a joint schema; Laya forms per-question encoder rows. One HTTP call does not guarantee identical compute or isolation across implementations.

Split when a later request needs newly fetched evidence, a different state, or room beyond the current request budget. For latency, distinguish sequential calls from concurrent calls: parallel network requests can finish sooner, but still repeat the input.

Many questions and many items are different experiments

The lab above adds questions to one unchanged state. Packing many customer messages into that state changes the evidence every question sees. Test that growth separately from question count; fitting the context budget does not establish reference accuracy.

{
  "state": {"messages": ["First return…", "Second return…"]},
  "questions": {"route_1": {
    "type": "choice",
    "instructions": "Which queue fits `messages[1]`?",
    "criteria": {"returns": "Returning an item", "other": "No fit"}
  }}
}

{
  "state": {"messages": {"case_a": "First return…", "case_b": "Second return…"}},
  "questions": {"route_b": {
    "type": "choice",
    "instructions": "Which queue fits `messages.case_b`?",
    "criteria": {"returns": "Returning an item", "other": "No fit"}
  }}
}

These are request excerpts; add a model to send them. IDs such as route_b match answers for code and are not visible to the model. Put the complete item reference in the instructions.

Numeric paths are documented and useful, but the model still has to resolve them. Our long-array experiments favoured named or quoted items; the 48-item results are a reason to compare forms on your own workload. Quoting duplicates text in the questions. Naming adds keys. Neither form guarantees isolation from adversarial content.

1. Compare your single-item request with the batch template at size one.
2. Sweep larger sizes on tuning data, keeping wording and reference form fixed.
3. Rotate item positions and composition; inspect errors by position and class, as well as failures.
4. Measure the time until the complete batch returns. Dividing it by item count gives amortized time, not each customer’s wait.

Shared definitions save repetition but change the question

A store taxonomy can live once in state, with each question pointing to it. The model must then resolve an extra reference. Compare that wording against inline criteria before treating token savings as an improvement.

Our shared-definition follow-up saved input tokens at eight items, while shifting an urgency Noul’s probabilities and changing errors differently at two cutoffs. At one item, indirection can cost more. Re-evaluate the exact shared form and select its thresholds afresh; the study notes give the observed scope.

Choose fan-out versus route-then-ask using full workflow measurements. Unused branch questions consume tokens; staged calls can avoid them while resending state and adding a sequential wait. Use a later call when a first answer is needed to retrieve evidence or construct candidates.

Choosing a pattern

These compose. A real return-handling flow is a fan-out request whose branch answers feed a composite priority, gated by a cascade for the ambiguous cases, with order numbers pulled by selection. Next, evaluate the frame and action policy, then use the build guide to connect them to software.