Part 11 of 12 · Pitfalls and limits

Give each task the right owner

Some failures need better evidence. Others need an ordinary function. Use these exercises to decide what belongs in code, what needs interpretation, and what needs further investigation.

The examples are written for this guide. TypeSafe’s version-specific limitation notes are linked where relevant.

Keep exact rules out of the judgment

A refund deadline is a date calculation. Whether a customer is asking for a refund is an interpretation. Keep those tasks separate: calculate the deadline exactly, and use the model only for the meaning of the message.

Deterministic code still needs correct inputs and tests. Its advantage here is that an explicit rule can be checked directly. A model adds uncertainty without helping when the rule already settles the answer.

Before adding a question, name the part that cannot be resolved by a calculation, parser, or lookup. If the remaining task needs more evidence or a written explanation, plan that work explicitly.

Lab

Code or model?

Twelve tasks from four products. Assign each to the bucket that should own it. The reasons matter more than the tally.
CodeSystem OneGenerative model or person
Answered
0 / 12
Correct
0
Several of these are close calls on purpose. When the reason says "with a roster it is code", that is the general rule: the moment you can enumerate the answer, the enumeration wins and the model only handles what falls outside it.

Turn limitations into test cases

The Jev 1.13 limitation notes describe weaknesses specific to that release. For a working application, turn those notes into tests of your own inputs.

Evidence

Can it find the relevant facts?

Try missing context, long records, irrelevant passages, and text that attempts to influence the verdict.

Meaning

Is the boundary explicit?

Try negation, conditional requests, overlapping labels, and instructions that contradict the criteria.

Exactness

Should code do this part?

Move counting, dates, numeric comparisons, and exact copying into functions you can test.

The lab turns those concerns into concrete rewrites. Each rewrite is a hypothesis to test on your own inputs.

Lab

Jagged edges

Pick a failure mode. Left: a request that trips it. Right: the rewrite. The fix is almost always "move the exact part to code and leave the model the judgment".
Trips it
Rewrite

Examples are original; the failure modes and the "instead" advice are the documented ones. Where a rewrite shows code, the language is Python for brevity and the same split applies in any language.

Typed is not true

An answer can fit its schema and still describe the input incorrectly. A valid category tells you that the interface worked; it does not prove that the classification worked.

Calibration is a goal to verify on your evaluation set. A well-calibrated group of 0.8 predictions succeeds about 80% of the time, but a new language, topic, or question can change that relationship.

A concentrated distribution deserves the same scrutiny. Missing context or misleading criteria can produce a clear-looking wrong answer. Compare the prediction with an independently checked outcome before trusting a threshold.

Practical consequence

Start by comparing suggestions with independently reviewed outcomes. Keep the failure cases, including high-confidence errors. Use them to decide whether a question is ready for automatic use.

Named fields do not establish trust

Separate customer text from trusted facts so the request can be inspected. Keep authorization and allowed actions in code. The Jev 1.13 notes document that adversarial content can steer answers.

Our two injection probes preserved neighbour labels with keyed and quoted references. That narrow result does not establish a general defense. Test attacks on the target’s own judgment and on neighbours, vary wording and position, and isolate requests when residual cross-item risk is unacceptable. A separate request reduces shared context; it does not make its content trustworthy.

A Noul checking for suspicious text is another fallible prediction. It cannot grant permission, prove isolation, or guarantee redaction. The experiment matrix keeps that distinction testable.

Test what should change and what should stay stable

Add relevant history and check whether the decision follows it. Paraphrase a request without changing its meaning and check whether the application’s action stays stable. Change batch position, add an unrelated neighbour, remove a referenced field, and simulate a failed call.

Inspect both label changes and cutoff crossings. Preserve each regression case with the frame and model version that produced it. A failure near an action boundary matters even when the winning label is unchanged.

Jev’s operational limits

Capacity belongs to the model and serving surface. The provider comparison covers Clef, Kev, and Laya; do not apply this table to them.

This Jev 1.14 snapshot was checked October 15, 2026. Default text limits remain the same as 1.13. Experimental images and long context need separate opt-in headers; check the linked live references when configuring a deployment.

LimitDocumented valueWhat it means for youSource
Context per request64k tokensShared by the state and every question in the request. Fan-out is bounded by this, not by a question count.Models
State plus longest question32k tokensThe state is ingested once; the biggest single question sets the second ceiling. Trim state before you trim questions.Models
Opt-in long context262k for both limitsAdd TypeSafe-Danger-Zone-Long-Context: true. Slower and less accurate; evaluate separately.Models
Choice optionsup to 255Enough for a full team list or taxonomy level. Past that, split the pick into a coarse step and a fine step.Choice
Score levels2 to 10Add a level only when you can describe it as a distinct situation. Validate the level count before sending a request.Score
Rate limitsPer organizationRead your organization’s token/s and request/s limits in the console. Exceeding either returns 429; SDKs back off and honour retry-after.Models
Price$0.042 per Mtok inputOutput tokens are free. Cost is driven by state size times request count, which is why batching questions per state is the main lever.Models
InputText; opt-in PNG/JPEGImages need assets and the images header. Box requires 1.14. Image support is experimental; audio/video need preprocessing.Images
LanguageEnglish primaryEnglish is strongest; other scripts are supported at lower accuracy. Measure on your own text and keep an eye on confidence.Models
CustomizationNone per accountNo fine-tuning or adapters. You shape behaviour through state, instructions, and criteria only.Models
Model namesjev-1.14.0 · jev-1.13.0 · aliasesOn October 15, latest selects 1.13 and preview selects 1.14. Pin the evaluated version and log the returned ID.Models
Errors400 · 401 · 413 · 415 · 422 · 429 · 529Request rules/context, key, size, encoding, schema, rate limit, overloaded. Fix rejected inputs; back off on 429/529.API reference

What you can control locally: trim irrelevant evidence, batch compatible questions, bound retries, and keep a review path. Test these behaviours with oversized inputs, missing fields, and a simulated service failure.

Anti-patterns

Each of these shows up in a first integration. Each is worth testing before connecting a judgment to a real action.

  1. Asking for an explanation. There is no field for one. A question like "why is this urgent?" has no answer space, so it cannot be a Choice, Score, or Noul. Ask for the judgment and let code or a generative model produce prose if a person needs it.
  2. Reading a Noul near 0.5 as "medium". A Noul is the probability that the statement is true. At 0.5 the model is saying it cannot tell, not that the property is half present. If you want intensity, use a Score with described levels.
  3. Putting dependent questions in one request. A request containing several questions does not create a sequential workflow. If a later decision needs a returned answer or newly retrieved evidence, construct another request. Check question isolation separately; Clef’s joint schema scoring differs from Jev/Kev isolation.
  4. Stuffing the state. Sending the whole record "just in case" is the documented context-rot failure. Send the fields the question needs, name them, and reference them by path in the instructions.
  5. Treating confidence as permission. Confidence measures how peaked the distribution is. The threshold at which you act is a cost decision you make in code, and it should differ per action.
  6. Overlapping Choice options. A Choice still returns one winner when several labels apply; its distribution need not expose every applicable label. Define the intended tie-break or use one Noul per label for applicability. A harmless preference among several acceptable answers can tolerate lower confidence.
  7. Score levels that describe adjectives instead of situations. "Low, medium, high" gives the model nothing to match against. Each level should read like a scene the model can recognise in the state.
  8. No escape option. A Choice with no "none of these" forces a wrong pick when nothing fits. Noul criteria are optional; explicit true/false descriptions can help define a difficult boundary. Use an input-validity or presence judgment when the application needs it separately, and evaluate it too.
  9. Copying a cookbook threshold. A cutoff tuned on someone else's documents says nothing about your invoices. Thresholds are measured, not inherited. Put every one of them in a labelled evaluation before it gates an action.
  10. Pinning a model version with no plan to move. Pinning is right when thresholds are tuned against a version, but a pinned integration also never gets future model improvements. Pin, log the version from the response, and schedule a re-evaluation when a new release ships.