Make the question answerable
A useful question names the evidence, defines the boundary, and produces an answer your application can use. Practise those three moves with a reading group choosing papers.
Fictional abstracts and illustrative answers. Nothing on this page calls a model.
Anatomy of a question
Before writing a request, complete this sentence: “Given this material, I need to know ___ so the application can ___.” That sentence connects the question to a visible behaviour.
The shape of the answer
choice for one of a set, score for a position on ordered levels, noul for the probability of yes. Pick by what the code will do with the answer.
The judgment
A specific question or a statement to judge. A string, or an object or array when the judgment has parts worth labelling, such as what to focus on and what to ignore.
The answer space
For a Choice, the named options; for a Score, the levels in order from low to high; for a Noul, an optional gloss on what yes and no mean. Each entry may itself be a string or a structured object.
The question ID is a lookup key for your application. Put the full meaning in instructions. Keep the wording and criteria under version control so that a change can be evaluated before it changes production decisions.
"code_released": {
"type": "noul",
"instructions": "Does `paper.abstract` state that code, models, or data from this work are publicly released?",
"criteria": {
"true": "The abstract says code, checkpoints, data, or probes are released or available",
"false": "No release is mentioned, or only a promise to release later"
}
}Separate the decisions you want to tune
A reading group asks “Should we discuss this paper?” The answer depends on the group’s interests, the kind of contribution, and whether members can try the work themselves. If those factors matter separately, ask about them separately.
For example, measure relevance, identify the contribution, and check whether the abstract mentions released code. The group can then change its selection rule without repeating every judgment.
Do not split merely to increase the question count. Keep a judgment together when it has one useful meaning, such as whether an abstract addresses any of the group’s listed interests.
Decomposition lab
The same abstract, asked two ways. Toggle between the vague question and its decomposition and compare what the code can do with each. Answers are illustrative and omit the confidence field.State (JSON object; the abstract is fictional):
Criteria draw the boundaries
Imagine two group members labelling the same abstract. One uses “systems” for anything involving a computer; the other reserves it for infrastructure research. A label alone has not established a shared boundary.
Start with a plain description of each option. Add an exclusion or example to resolve an observed confusion, then compare against that baseline. Jev supports structured instructions and criteria. For portability, start with plain descriptions and verify richer value types on the destination API. Structured criteria organize meaning; keys such as what, not_for, and examples are descriptive conventions, not special API operators.
- Describe what the option covers, in terms that match your inputs.
- Exclude what it does not cover, naming the neighbouring option that should win instead. Exclusions are where two fuzzy options stop overlapping.
- Exemplify when examples help express a boundary. Draw them from a labelled pool separate from evaluation data; keep near duplicates and paraphrases with their source.
Check whether every realistic input has a place to go. “Outside our topics” handles irrelevant papers; “not enough information” handles incomplete abstracts. Use separate checks when you need to distinguish absence of evidence from evidence of absence.
Criteria sharpener
Compare three versions of the same question. The invented distributions show the intended effect of clearer boundaries; a real rewrite needs evaluation to see whether it helps.Shape the state so questions can point
Give the model a tidy evidence packet: the abstract, the reading group’s interests, and any context the judgment genuinely needs. Named JSON fields keep those roles visible.
A backticked path such as `paper.abstract` names the target in natural language. It is interpreted by the model, not enforced as a programmatic pointer. In large collections, compare numeric paths with named items and quoted content; item batching explains the evaluation.
Remove unrelated attachments and compute exact facts first. If publication age matters, calculate it in code and provide the result. Retain enough surrounding text to preserve qualifiers and negations.
Path builder
Click any key in this state to get the backticked path a question would use, dropped into a sample instruction.Ask whether each extra description earns its place
A longer criterion can fix a boundary or teach the model an irrelevant association. Compare the complete question with a version that removes one exclusion, example, or note. Measure the resulting decisions and token usage on the same cases.
Keep a description for its effect on the task, not simply because it makes confidence rise. If nothing improves, the smaller frame may be preferable. After choosing the wording, validate once on untouched data. The evaluation chapter explains ablation and paired comparisons.
Ensure instructions and criteria describe the same judgment. Noul criteria are optional; add descriptions of true and false when they clarify a boundary, rather than treating them as a required defense against every ambiguity.
The token budget
The meter below estimates how state size and question count affect a request. It uses the Jev 1.14.0 default text limits and price checked October 15, 2026 as a snapshot: 64k tokens total, 32k for state plus the longest question, and $0.042 per million input tokens. Check current model details before budgeting a deployment. This meter excludes image tokens and the opt-in 262k mode; use the long-context builder to inspect that header.
State and token meter
Paste a state and see an estimated token count against the documented limits, plus what a thousand requests would cost at the documented price. The estimate uses about four characters per token; the API's usage field reports the real count.With questions this shape, the answers become material for the patterns in Part 8: ask many at once and have code discard the answers it does not need (speculative fan-out), or weight several Scores into one ranking you can retune without another request (composite scoring). Part 11 lists what still belongs in code rather than in a question (code or model?).