Know what an answer can tell you
A review comment can ask for a change without using the word “change.” That is a useful job for a model. Deciding whether an entire release is safe needs much more evidence. Learn where to draw that boundary.
Try it with fictional code reviews. The charts use synthetic data, not measured model performance.
Start with the evidence you actually have
Imagine sorting review comments into “change requested” and “observation.” A sentence such as “Could this loop wait before trying again?” carries a request even without an explicit label. The model can interpret that meaning from the comment.
Now consider “Can we release this service?” The comment alone cannot establish test coverage, deployment readiness, or the effect of a migration. Fetch those facts first. Ask focused questions about the evidence you have, then apply your release rules in code.
A useful boundary is whether the supplied material supports the judgment. The name comes from Kahneman’s System 1, fast intuitive judgment as opposed to slow deliberate reasoning. It describes the kind of question the model is built for.
The whole contract fits on one line
Think of the request as an envelope with two compartments: the material to read (state) and the judgments to make (questions). You assign each question an ID so your application can find its answer.
The model does not use the ID as part of the question. A key named blocking cannot rescue instructions that only say “Is it?” Write “Does this comment request a change before merge?” Each request must include its own context; earlier calls and answers to neighbouring questions are not memory.
Check the numbers against outcomes
Suppose you collect 100 judgments assigned a 0.8 probability of “yes.” If roughly 80 are actually yes, that group is well calibrated. This says something about the group, not which individual answers will fail.
Accuracy asks how often the winning answer is correct. Calibration asks whether probabilities match observed frequencies. A model can rank cases well while assigning probabilities that are too extreme. The chart below lets you separate those two effects.
TypeSafe describes calibration as a training objective. Whether it holds for your questions, language, and inputs is something to measure. Keep a labelled evaluation set that resembles the work your application will do.
Calibration explorer
A synthetic set of 400 yes/no judgments. Each dot is a bin of reported probabilities; its height is how often those answers came true. A calibrated model sits on the diagonal. The sliders distort the reported probabilities without touching the outcomes.The overconfidence slider preserves which side of 0.5 each answer lands on, while moving its probability toward an extreme. Classification accuracy can stay the same even as the probabilities become less reliable. Noise can change the classification too. Check both effects when evaluating your own questions.
Evaluate the behaviour, not the training label
TypeSafe calls its training approach RLCD: reinforcement learning for calibrated decisions. Its aim is to make the returned probabilities useful for software. TypeSafe’s AI primer explains the approach.
For an integration, the practical test is simpler: do these answers help your application make better decisions? Compare mistakes, review workload, cost, and response time on the same cases. A training label cannot answer those questions for you.
Generative models can also return schema-constrained output. Use generation when you need a draft or an explanation; compare decision models when you need judgments over defined answers.
Share context; keep judgments independent
Imagine three reviewers each receiving the same comment. One checks whether a change is requested, one classifies the topic, and one checks the tone. Each has the source material, but none receives the others’ verdicts. That is how to think about questions sent together.
- Share the evidence. Put questions about the same input in a single request when the request budget permits.
- Keep each question understandable alone. “How severe is the bug you just found?” assumes an answer the second question cannot see. “If this report describes a bug, what impact does it describe?” supplies its own premise.
- Wait when the evidence changes. If an answer determines which file you fetch next, build the next request after that fetch.
Batching can avoid repeated state and network trips. Extra questions still consume tokens and capacity. Measure the response time of your actual request.
Add a question, nothing else moves
Toggle questions in and out of one request about the same review comment. The request body updates; the answers already on screen do not change. Answers are illustrative; token counts are estimates.Show the request body
Snap judgment or not?
For each review task, choose an owner: deterministic code, a focused model judgment, or a person or model that can investigate and explain. Look for the evidence required, not just the wording of the question.
Sort the questions
Pick one answer per row. The feedback explains the tell.Put the contract to work
Next, choose an answer type that fits what your code needs. Then use observed outcomes to decide which answers can be used automatically.