Make the question answerable
A useful question names the evidence, defines the boundary, and produces an answer your application can use. Practise those three moves with a reading group choosing papers.
Fictional abstracts and illustrative answers. Nothing on this page calls a model.
Anatomy of a question
Before writing a request, complete this sentence: “Given this material, I need to know ___ so the application can ___.” That sentence connects the question to a visible behaviour.
The shape of the answer
choice for one of a set, score for a position on ordered levels, noul for the probability of yes. Pick by what the code will do with the answer.
The judgment
A specific question or a statement to judge. A string, or an object or array when the judgment has parts worth labelling, such as what to focus on and what to ignore.
The answer space
For a Choice, the named options; for a Score, the levels in order from low to high; for a Noul, an optional gloss on what yes and no mean. Each entry may itself be a string or a structured object.
The question ID is a lookup key for your application. Put the full meaning in instructions. Keep the wording and criteria under version control so that a change can be evaluated before it changes production decisions.
"code_released": {
"type": "noul",
"instructions": "Does `paper.abstract` state that code, models, or data from this work are publicly released?",
"criteria": {
"true": "The abstract says code, checkpoints, data, or probes are released or available",
"false": "No release is mentioned, or only a promise to release later"
}
}Separate the decisions you want to tune
A reading group asks “Should we discuss this paper?” The answer depends on the group’s interests, the kind of contribution, and whether members can try the work themselves. If those factors matter separately, ask about them separately.
For example, measure relevance, identify the contribution, and check whether the abstract mentions released code. The group can then change its selection rule without repeating every judgment.
Do not split merely to increase the question count. Keep a judgment together when it has one useful meaning, such as whether an abstract addresses any of the group’s listed interests.
Decomposition lab
The same abstract, asked two ways. Toggle between the vague question and its decomposition and compare what the code can do with each. Answers are illustrative and omit the confidence field.State (JSON object; the abstract is fictional):
Criteria draw the boundaries
Imagine two group members labelling the same abstract. One uses “systems” for anything involving a computer; the other reserves it for infrastructure research. A label alone has not established a shared boundary.
Write down what qualifies, what belongs elsewhere, and a representative example. Structured criteria can help organize that information. More detail is useful only when it resolves a real ambiguity.
- Describe what the option covers, in terms that match your inputs.
- Exclude what it does not cover, naming the neighbouring option that should win instead. Exclusions are where two fuzzy options stop overlapping.
- Exemplify with two or three phrases that look like the real data. Examples steer strongly, so they must be representative.
Check whether every realistic input has a place to go. “Outside our topics” handles irrelevant papers; “not enough information” handles incomplete abstracts. Use separate checks when you need to distinguish absence of evidence from evidence of absence.
Criteria sharpener
Compare three versions of the same question. The invented distributions show the intended effect of clearer boundaries; a real rewrite needs evaluation to see whether it helps.Shape the state so questions can point
Give the model a tidy evidence packet: the abstract, the reading group’s interests, and any context the judgment genuinely needs. Named JSON fields keep those roles visible.
Use a path such as `paper.abstract` in the instructions so the target is unambiguous. A path tells the model which text you mean.
Remove unrelated attachments and compute exact facts first. If publication age matters, calculate it in code and provide the result. Retain enough surrounding text to preserve qualifiers and negations.
Path builder
Click any key in this state to get the backticked path a question would use, dropped into a sample instruction.The token budget
The meter below estimates how state size and question count affect a request. It uses the published Jev 1.13 limits and price as a snapshot: 64k tokens total, 32k for state plus the longest question, and $0.042 per million input tokens. Check current model details before budgeting a deployment.
State and token meter
Paste a state and see an estimated token count against the documented limits, plus what a thousand requests would cost at the documented price. The estimate uses about four characters per token; the API's usage field reports the real count.With questions this shape, the answers become material for the patterns in Part 6: ask many at once and have code discard the answers it does not need (speculative fan-out), or weight several Scores into one ranking you can retune without another request (composite scoring). Part 8 lists what still belongs in code rather than in a question (code or model?).