How To Use Decision Models for Evals

main.content table { display:block; width:100%; max-width:100%; overflow-x:auto; } #evaluation-prompt pre { background:var(--bs-body-bg); color:var(--bs-body-color); padding:18px; border:1px solid var(--bs-border-color); border-radius:6px; } #evaluation-prompt pre code { color:inherit; white-space:pre-wrap; overflow-wrap:anywhere; } Decision models have exploded onto the scene recently, starting with TypeSafe’s Jev, followed by Perplexity’s Decider, Liquid’s d1, Cloudflare’s Clef, and OpenAI’s Decisions API 1 . These models promise fast, inexpensive classification. Decision models are an appealing low-cost alternative to LLM judges 2 for automated evals. But whether they’re the right choice depends on the cost, performance, and latency requirements of your use case. We’ve also noticed people using the default decision threshold of 0.50 without checking whether it works well for their task. This can leave a lot of performance on the table. In this post, we’ll start by showing how to tune thresholds to get the best results from decision models. With those thresholds tuned, we’ll compare cost and performance to help you assess which model fits your use case. Finally, we’ll explore how to interpret model scores when deciding whether to act, and how calibration can make them more useful as confidence estimates. What we tested We used 100 traces, or logs of conversations and actions, from an apartment leasing agent. The task is to identify traces containing an error, such as a failed booking or a missed handoff to a human. 3 We used the same dataset and broad failure-finding task in our prior post on automated evals . Here, each decision model returns a failure probability for each trace. We tested Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions (GPT-6 Luna) 1 . Each model received the same evaluation criteria. The appendix shows the prompt and the structured version used for Jev. We compare models on two dimensions: Performance: We compare each model’s predictions with traces that people have reviewed and labeled as passing or failing. This measures how well the model catches failures while avoiding false alarms. Cost: We calculate API cost from token usage and the provider’s prices. 4 For brevity, we leave latency aside because it depends on factors outside this experiment’s control. 5 Understanding decision thresholds We assign each trace one of two labels, pass or fail. This is a binary pass/fail eval , which we recommend because it encourages clear criteria and is easier to align with human preferences. Each decision model returns a score from 0 to 1 that estimates how likely a trace is to contain a failure. The APIs turn this score into a decision using a default decision threshold of 0.50 . Traces with scores at or above the threshold are flagged as failures. Using this default without checking it against your data can give you subpar results. For example, a model could assign every failure a score of 0.40 and every passing trace a score of 0.10. Even though this model classifies traces perfectly at a threshold of 0.30, the default threshold of 0.50 would catch nothing. In practice, scores for failures and passing traces will overlap. Lowering the threshold catches more failures but can also raise more false alarms. Choosing a threshold means balancing: Precision: The share of flagged traces that contain a known failure. Recall: The share of known failures that the model catches. The right tradeoff depends on your business goals. For example, criminal trials set a high bar for conviction to avoid convicting innocent people, emphasizing precision. Cancer screening may emphasize recall, accepting more false alarms to miss fewer cases. 6 For this comparison, we’ll give precision and recall equal weight using the F1 score , their harmonic mean. F1 ranges from 0 to 1, and higher is better. The curves below show how each model scores failures (orange) and passing traces (blue). Taller parts of a curve mean more scores from that class fall in that range. Scores at or above the dashed threshold line are flagged. Use the toggle to compare 0.50 with Optimized for best F1 , which chooses a threshold for each model. 7 (() => { const frame = document.querySelector('.eval-score-frame'); const fit = () => { try { const content = frame.contentDocument.querySelector('main'); const resize = () => { frame.style.height = Math.ceil(content.getBoundingClientRect().height + 4) + 'px'; }; resize(); new ResizeObserver(resize).observe(content); } catch () { /* Keep the fallback height for file previews. */ } }; frame.addEventListener('load', fit); if (frame.contentDocument?.readyState === 'complete') fit(); })(); OpenAI separates the two classes best on this task, which is also reflected in its higher F1 score. It’s also important not to blindly maximize F1 without thinking about your minimum requirements for the classifier to be useful. For example, if this classifier is used for an automated alerting system, I would want a minimum of 80% on both precision and recall so that I’m minimizing false alarms while catching a sufficient number of failures. The chart below shows how precision and recall change at different thresholds. 8 The shaded region marks where both reach at least 80%. (() => { const frame = document.querySelector('.precision-recall-frame'); const fit = () => { try { const content = frame.contentDocument.querySelector('main'); const resize = () => { frame.style.height = Math.ceil(content.getBoundingClientRect().height + 4) + 'px'; }; resize(); new ResizeObserver(resize).observe(content); } catch () { /* Keep the fallback height for file previews. / } }; frame.addEventListener('load', fit); if (frame.contentDocument?.readyState === 'complete') fit(); })(); Perplexity, Liquid d1-ultra, and OpenAI Decisions meet our requirements, but we still need to choose a decision threshold within the shaded region for each model. In the next section, we’ll discuss how to tune those thresholds. Tuning the decision threshold For this example, we’ll optimize F1. For our candidates, we’ll restrict tuning to thresholds that meet both 80% minimums on the training data whenever possible. We leave the other models unconstrained to illustrate the effect of tuning. 9 With labeled traces and saved model scores, you can ask a coding agent to tune the thresholds. Below is an example prompt you can modify for your use case: Using my saved model scores and binary labels (1 means failure), tune a decision threshold to maximize F1 while requiring at least 80% precision and 80% recall. Use stratified five-fold cross-validation, choosing thresholds on the training examples and measuring performance on the remaining examples. If comparing models, use the same splits for each. Only select thresholds that meet both minimums on the training examples. If none qualifies, report that rather than relaxing either requirement. Report validation precision, recall, and F1, along with any splits where no threshold qualified or validation performance fell below my minimums. Compare with the default threshold of 0.50. Reuse the saved scores without making new model calls. Here’s how tuning for F1 changed each model’s performance. We group the three candidates from the precision-recall chart together and show the other models in gray to illustrate how much tuning can help. (() => { const frame = document.querySelector('.threshold-impact-frame'); const fit = () => { try { const content = frame.contentDocument.querySelector('main'); const resize = () => { frame.style.height = Math.ceil(content.getBoundingClientRect().height + 4) + 'px'; }; resize(); new ResizeObserver(resize).observe(content); } catch (_) { / Keep the fallback height for file previews. / } }; frame.addEventListener('load', fit); if (frame.contentDocument?.readyState === 'complete') fit(); })(); Our candidate models gained less from F1 tuning than Liquid d1 and Clef Flash, which fell below our minimums. However, even a small shift in precision and recall can affect business costs or the user experience, so choose the tradeoff deliberately. In practice, it’s unlikely that precision and recall are equally important (as we assumed in this post with F1). For example, the F-beta score lets you give precision more weight with beta below 1, or recall more weight with beta above 1. Optimizing a different metric could lead to different thresholds and results. Performance versus cost The chart below compares each model’s F1 (after tuning) against cost, focused on our remaining candidates: (() => { const frame = document.querySelector('.cost-comparison-frame'); const fit = () => { try { const content = frame.contentDocument.querySelector('main'); const resize = () => { frame.style.height = Math.ceil(content.getBoundingClientRect().height + 4) + 'px'; }; resize(); new ResizeObserver(resize).observe(content); } catch (_) { / Keep the fallback height for file previews. / } }; frame.addEventListener('load', fit); if (frame.contentDocument?.readyState === 'complete') fit(); })(); The dashed line shows the Pareto frontier among our remaining candidates. For any given F1 score, it identifies the cheapest candidate that meets or exceeds that score. This particular chart is a bit silly as there are only three options, but it’s a useful illustration for how to think about cost vs. performance. Perplexity was the cheapest of these candidates at 0.82 cents per 100 eval runs, with an F1 of 0.79. OpenAI Decisions reached an F1 of 0.87 at 3.72 cents per 100 eval runs, about 4.5 times the cost. Can we trust the probabilities? Beyond pass/fail evals, you may want to use a decision model’s score as a confidence estimate. But can you take that number literally? For a score of 0.80 to mean an 80% chance of failure, roughly 80% of traces with similar scores should contain failures. The chart below shows each model’s probability calibration : how closely its predicted probabilities match observed outcomes. 10 Each point represents about 20 traces with similar scores. A point on the diagonal means the predicted probability matches the failure rate in that group. Above the line, the model underestimates the risk of failure. Below it, the model overestimates that risk. If the predicted probabilities don’t match the observed outcomes, we can adjust them using another model. The second panel shows one method, Platt scaling , which fits a simple curve to make that adjustment. 11 (() => { const frame = document.querySelector('.calibration-frame'); const fit = () => { try { const content = frame.contentDocument.querySelector('main'); const resize = () => { frame.style.height = Math.ceil(content.getBoundingClientRect().height + 4) + 'px'; }; resize(); new ResizeObserver(resize).observe(content); } catch (_) { / Keep the fallback height for file previews. */ } }; frame.addEventListener('load', fit); if (frame.contentDocument?.readyState === 'complete') fit(); })(); Perplexity, Liquid d1, and Clef Flash move closer to the diagonal after calibration. For example, one group of Clef Flash traces had an average predicted failure probability of about 8%, but 60% contained failures. Jev and Liquid d1-ultra were already well calibrated on this dataset and needed little adjustment. Platt scaling isn’t always the best fit. For example, OpenAI’s probabilities got slightly worse with the version of Platt scaling we used. 12 You can direct your coding agent to perform calibration with the following prompt: Using my saved failure probabilities and binary labels (1 means failure), test whether Platt scaling improves each model’s probabilities. Use stratified five-fold cross-validation with the same splits for every model. Fit the calibration mapping on each training split and apply it to the remaining traces. Fit on log-odds, clipping probabilities away from 0 and 1 first. Plot predicted probabilities against observed failure rates before and after calibration. Report Brier scores (lower is better), and explain which models improved or got worse. Reuse the saved scores without making new model calls. Prompt tuning and evaluator scope We kept the eval task in this post broad to make the differences between models easier to see. In practice, we recommend building a separate judge for each failure mode . With a narrower task and more specific guidance, we expect the performance gaps between models to shrink. I like to start with a larger model that has acceptable performance on a task and then try smaller models to see if they can match the larger model’s performance for less cost. Prompt tuning can also be helpful if you have high-quality labels. 13 I discuss how to craft a good judge prompt in this post . Putting these ideas into practice For this particular dataset, OpenAI had breakout performance at a higher cost. Perplexity and Liquid d1-ultra offer lower-cost options with lower performance. We have observed that model performance varies substantially across tasks and datasets, which means it’s important to measure things yourself. To recap: Once you’re satisfied with your judge’s prompt , set minimum precision and recall requirements for your use case. Rule out models that cannot meet these. Tune each model’s decision threshold before comparing performance. Choose a precision-recall tradeoff that fits your business goals. Be intentional, don’t just use F1 score because this post did! If you want to use scores as confidence estimates, check whether the probabilities match observed outcomes. Calibration can help, but you need to measure whether it improves them. Weigh performance against cost and latency for your workload. This process is iterative. You may improve the prompt, then need to tune the threshold again or revisit calibration. In this post, we focused on threshold tuning and calibration because we’ve seen many people use decision models off the shelf without considering these factors. You might also be wondering how decision models stack up to LLMs. OpenAI Decisions uses GPT-6 Luna, which delivered our highest F1 at the highest cost, suggesting that harder eval tasks may justify paying more for an LLM. In a broader comparison , Randy Olson found that decision models could match LLMs on classification and routing at lower cost, while LLMs pulled ahead on tasks requiring knowledge and reasoning. None of the ideas discussed in this post are new. Choosing a precision-recall tradeoff, tuning thresholds and comparing performance vs. cost are part of evaluating any classifier. To learn more about these fundamentals, I recommend Machine Learning with Python Cookbook, 2nd Edition by Kyle Gallatin and Chris Albon. Finally, if you’re a data scientist or ML engineer wondering where you belong in the AI age, these are skills you already have. It’s time to get revenge 🙂. Appendix: Evaluation prompt Perplexity, Liquid d1 and d1-ultra, Clef Flash, and OpenAI Decisions received the prompt below. Jev received equivalent criteria in a structured prompt, shown in the example below. The scores and labels used in the charts are available as JSON. Decide whether messages contains at least one substantive failure, risk, or quality problem in the AI assistant's behavior or tool use. This is one complete production trace from an AI that handles apartment-leasing customer support over phone or messaging. The trace includes the conversation plus any tool calls and tool outputs. Evaluate the AI assistant and its tool use, not the customer. Treat system messages inside messages as the instructions governing the assistant's behavior and allowed capabilities. Inspect the full trace, including final turns and all tool calls and outputs. Base the judgment only on evidence visible in this trace. Do not assume external policies, channel behavior, or business requirements that are not present. Do not assume every trace failed. A short, ambiguous, or inconclusive trace is not an issue unless visible evidence establishes a substantive failure. Return the probability that at least one substantive issue is present; the issue does not need to be named or clustered. Count as an issue when supported by the trace: the assistant contradicts an applicable system instruction or tool result; makes an unsupported factual or operational claim; misses, misuses, repeats, or falsely claims completion of a required tool action, transfer, booking, cancellation, follow-up, or confirmation; materially misunderstands or leaves an explicit user request unresolved; or exposes internal actions, placeholders, or broken content to the customer in a way that materially harms the interaction. Do not count as an issue: a minor stylistic preference with no meaningful effect; a failure caused only by the customer's behavior; internal tool metadata that was not exposed to the customer and did not cause an incorrect response; a hypothetical concern requiring information outside the trace; or insufficient evidence and inconclusive conversations. Note Example: A structured prompt This is how we structured the prompt for Jev’s requests, with separate fields for instructions and evaluation criteria. { "version" : "nurtureboss-any-issue-v1" , "question_id" : "has_substantive_issue" , "type" : "noul" , "instructions" : { "task" : "Does messages contain at least one substantive failure, risk, or quality problem in the AI assistant's behavior or tool use?" , "context" : "This is one production trace from an AI that handles apartment-leasing customer support over phone or messaging. The trace contains the complete conversation plus any tool calls and tool outputs." , "evaluation_rules" : [ "Evaluate the AI assistant and its tool use, not the customer." , "Treat system messages inside messages as the instructions governing the assistant's behavior and allowed capabilities." , "Inspect the full trace, including the final turns and all tool calls and outputs." , "Base the judgment only on evidence visible in this trace. Do not assume external policies, channel behavior, or business requirements that are not present." , "Do not assume every trace failed. A short, ambiguous, or inconclusive trace is not an issue unless the visible evidence establishes a substantive failure." , "Return the probability that at least one substantive issue is present. The issue does not need to be named or clustered." ] }, "criteria" : { "true" : { "definition" : "At least one substantive problem is supported by evidence in the trace." , "examples_of_qualifying_evidence" : [ "The assistant contradicts an applicable system instruction or a tool result." , "The assistant makes an unsupported factual or operational claim." , "A required tool action, transfer, booking, cancellation, follow-up, or confirmation is missing, incorrect, repeated, or falsely claimed as complete." , "The assistant materially misunderstands or leaves an explicit user request unresolved." , "Customer-facing text exposes internal actions, placeholders, or broken content that materially harms the interaction." ] }, "false" : { "definition" : "No substantive assistant problem can be established from the visible trace." , "do_not_count_as_an_issue" : [ "A minor stylistic preference with no meaningful effect." , "A failure caused only by the customer's behavior." , "Internal tool metadata that was not exposed to the customer and did not cause an incorrect response." , "A hypothetical concern that requires information outside the trace." , "Insufficient evidence or an inconclusive conversation." ] } } } Footnotes Versions tested: Jev 1.13 ( announced September 15, 2026 ), Perplexity Decider 1.1 27B , Liquid d1 and d1-ultra , Cloudflare Clef Flash , and OpenAI Decisions . OpenAI Decisions uses GPT-6 Luna. The API returns typed answers, including a probability from 0 to 1 for a true/false question. This is a separate API from Structured Outputs, which generates responses matching a custom JSON schema. We include it because it serves the same classification task.↩︎ An LLM judge uses a prompt to evaluate an AI system’s output or behavior. We recommend asking it for a binary pass/fail decision about a specific failure mode. This encourages clear criteria and makes it easier to compare the judge’s decisions with human labels.↩︎ We scored 98 production traces after excluding two internal QA examples from the 100-trace dataset. The frozen reference contains 61 known failures and 37 traces without a labeled failure: 39 failures from the original human review plus 22 discoveries from Braintrust and Arize that we manually validated. Models received masked traces without human labels or reviewer notes. Perplexity, Liquid, Clef Flash, and OpenAI received the same instructions and serialized inputs; Jev received equivalent criteria through its structured API. The original conversations are private.↩︎ API cost covers 100 eval runs, one per trace. We use Liquid’s prices as of October 17, 2026 , $0.02 per million input tokens for d1 and $0.042 for d1-ultra. The experiment recorded d1 at $0.04; we have repriced its saved token usage. Other models use the rates in the experiment’s operational metrics. Performance excludes two internal QA traces. Threshold tuning uses saved scores and requires no additional model calls.↩︎ Latency depends on how you batch or parallelize requests, your arrangement with the vendor, and your distance from its infrastructure, among other factors. Batching and concurrency can also change cost and throughput. See TypeSafe’s multi-question requests and client performance guidance . Its launch discussion also notes the location of its latency tests relative to its service. Mean response times were 4.33 seconds for Jev, 7.85 for Perplexity, 7.38 for Liquid d1, 1.56 for Liquid d1-ultra, 2.09 for Clef Flash, and 2.77 for OpenAI. These came from separate runs and do not establish comparative throughput or tail latency.↩︎ These examples illustrate how the consequences of mistakes affect the tradeoff. In U.S. criminal trials, proof beyond a reasonable doubt sets a high bar for conviction. In cancer screening, a missed case can delay care, while a false alarm can lead to anxiety and unnecessary procedures. A positive screening result usually requires further testing. The NCI explains both risks ; the appropriate balance depends on the screening test and population.↩︎ The density plots show unconstrained F1 tuning, before we introduce minimum precision and recall requirements. The later comparison charts apply those requirements to candidate models where feasible. All use the same five-fold cross-validation splits. Thresholds were chosen on training traces, and precision, recall, and F1 were measured on separate traces. We average those measurements across repeated splits. In the optimized density plots, the dashed line shows the average selected threshold. The displayed scores come from the individual splits, rather than applying that average threshold to the full dataset. The density curves use Gaussian smoothing with bandwidth 0.07 and reflection at 0 and 1. Each class has an area of one. Smoothing affects only the display.↩︎ A precision-recall curve plots precision against recall as the decision threshold changes. It shows the tradeoffs available for a model across thresholds. Scikit-learn’s precision-recall guide explains how to read and plot these curves.↩︎ We identify candidates from the full-sample precision-recall curves. In each training split, we choose the highest-F1 threshold among those reaching at least 80% precision and 80% recall. When no threshold qualifies, we use the unconstrained F1 optimum and record the split as infeasible. This occurred in 36 of 100 training splits for Perplexity and 29 for Liquid d1-ultra, and never for OpenAI. Non-candidates always use unconstrained F1 tuning. We keep all validation predictions, including those from fallback splits, in the reported averages. The minimums apply during selection and do not guarantee the same performance on new examples. The threshold-selection audit records each split without trace IDs.↩︎ A calibrated failure probability of 0.80 means that, among many examples receiving scores near 0.80, roughly 80% contain failures. Platt scaling and isotonic regression use labeled examples to learn an adjustment from model scores to probabilities. Platt scaling fits a sigmoid curve. Isotonic regression fits a more flexible mapping that preserves score order, but is more prone to overfitting with small datasets. Neither guarantees improvement on new examples. Scikit-learn’s probability calibration guide explains both methods and how to assess them.↩︎ The reliability chart groups each model’s scores into five groups of 19 or 20 traces, sorted by probability. The adjusted scores are each trace’s mean out-of-fold probability from the saved cross-validation results. We fit Platt scaling to the log-odds of the original probabilities. Each mapping is fitted without the traces it predicts. The chart summarizes this small sample and does not establish an exact failure rate for a future trace. The plotted calibration results are available as JSON.↩︎ On the saved out-of-fold probability summaries, Jev’s Brier score was 0.187 before adjustment and 0.188 afterward. Liquid d1-ultra’s was 0.173 and 0.172, while OpenAI’s was 0.159 and 0.164. Brier score measures average squared probability error, so lower is better. It reflects discrimination as well as calibration. These results describe the log-odds Platt fit shown in the chart. An exploratory refit directly on probabilities, using the same folds, gave OpenAI a Brier score of 0.154. This illustrates sensitivity to the fitting method. It does not establish which method will work best on new data.↩︎ GEPA uses feedback from model runs to propose prompt revisions and test them against labeled examples. It retains candidates with different strengths, so a prompt that helps a difficult example can remain in consideration. Incorrect labels can steer this search toward the wrong changes. Keep a separate test set outside prompt development for the final check. In our small manual Jev experiment, F1 at the default threshold rose from 0.78 with a simple prompt to 0.80 with a checklist of failure types. That experiment did not use GEPA or establish how much systematic optimization would help. Our FAQ on choosing an eval model discusses prompt optimization and model comparison.↩︎