The results
in words.
Plain-language facts for each setup and condition, followed by the observations and limits that matter when choosing a model.
[ REPORT / 01 ]
A numbered record of the outcomes.
One fixed follow-up comparison.
The original CLM fixtures and game protocol were reused for a pinned public Kev-4B checkpoint and a fresh Jev-1.13.0 run. Narrow-decision and BFCL request hashes match the earlier CLM report. CLM values in this report are historical, not new CLM runs.
A serving comparison, with different network paths.
Both clients ran on the same US-KS-2 GPU pod. Kev used local loopback HTTP; Jev used hosted HTTPS. Client latency includes network transit, server caching, and model execution. It cannot isolate architecture speed.
Released Kev settings, without workload tuning.
Kev used bf16 precision, fused kernels, CUDA graphs, its checkpoint temperature, and the default four-state prefix cache. No fine-tuning, new calibration fit, or date preprocessor was used. The checkpoint, code revision, dependencies, and hardware are recorded in the audit data.
Choose one of 15 tools.
Kev answered 40/40 correctly (100.0%); Jev answered 40/40 (100.0%). The historical CLM result on the identical requests was 50.0%. Median client latency was 37.33 ms for Kev and 125.80 ms for Jev.
Keep one of four trajectories.
Kev answered 24/24 correctly (100.0%); Jev answered 24/24 (100.0%). The historical CLM result on the identical requests was 58.3%. Median client latency was 32.37 ms for Kev and 124.31 ms for Jev.
Judge tool-result relevance.
Kev's positive F1 was 0.95, with 100.0% recall, 2 false positives, and 0 false negatives. Jev's F1 was 0.85, with 100.0% recall, 7 false positives, and 0 false negatives. Brier probability-error scores were 0.0520 for Kev and 0.1092 for Jev; lower is better. Historical CLM F1 was 0.18.
Check subgoal completion.
Kev's positive F1 was 0.98, with 100.0% recall, 1 false positives, and 0 false negatives. Jev's F1 was 1.00, with 100.0% recall, 0 false positives, and 0 false negatives. Brier probability-error scores were 0.0220 for Kev and 0.0065 for Jev; lower is better. Historical CLM F1 was 0.57.
Select files for context.
Kev's positive F1 was 0.85, with 94.4% recall, 5 false positives, and 1 false negatives. Jev's F1 was 0.67, with 94.4% recall, 16 false positives, and 1 false negatives. Brier probability-error scores were 0.0529 for Kev and 0.0641 for Jev; lower is better. Historical CLM F1 was 0.36.
The overall narrow score hides class imbalance.
Kev answered 360/369 correctly, while Jev answered 345/369. But 207 of the 225 file judgments are negative: rejecting every file scores 92% in that category. Positive F1, recall, and the cost of false positives are more useful than a single combined accuracy.
Related prompt variants test consistency.
Semantic answers stayed the same in 71/72 Kev pairs and 71/72 Jev pairs when option order or incidental text changed. These are related copies of base tasks, not independent examples. Consistency alone does not mean the answer is correct.
A fresh hosted run can differ under the same version label.
The earlier Jev run scored 350/369; the new run scored 345/369. Both identified themselves as jev-1.13.0, but 5 predictions changed on the same frozen requests. The visible version label does not guarantee identical outputs. This is why this report uses fresh Jev calls instead of copying its earlier answers; the cause of the variation was not established.
New support tickets.
Kev routed 40/40 correctly and Jev routed 40/40. Median client latency was 31.83 ms for Kev and 122.14 ms for Jev. The same 40 states, four team descriptions, fixed order, and serial protocol were used for both models.
Exact ticket replay.
Kev routed 40/40 correctly and Jev routed 40/40. Median client latency was 31.42 ms for Kev and 124.02 ms for Jev. The same 40 states, four team descriptions, fixed order, and serial protocol were used for both models.
A replay is a cache-dependent condition.
No client answer cache was used. Kev retains four state prefixes, so the 40-ticket replay cycles beyond its default cache capacity. CLM's earlier exact-replay benefit came from a different embedding and answer-cache design. Those replay timings should not be treated as a pure model-compute comparison.
BFCL: name + description.
Kev chose the official function in 1215/1,253 cases (96.97%); Jev did so in 1225/1,253 (97.77%). Median client latency was 35.34 ms for Kev and 130.69 ms for Jev. Historical CLM accuracy on the identical requests was 76.86%.
BFCL: parameters in prose.
Kev chose the official function in 1059/1,253 cases (84.52%); Jev did so in 1236/1,253 (98.64%). Median client latency was 65.75 ms for Kev and 134.55 ms for Jev. Historical CLM accuracy on the identical requests was 58.42%.
BFCL: full json schema.
Kev chose the official function in 1216/1,253 cases (97.05%); Jev did so in 1235/1,253 (98.56%). Median client latency was 70.65 ms for Kev and 137.22 ms for Jev. Historical CLM accuracy on the identical requests was 47.49%.
Candidate representation changes the measured result.
Moving from name-plus-description to full JSON changed Kev accuracy by +0.08 percentage points and Jev by +0.79 points. All three views were specified before this run. The description and prose adapters also omit some schema information, so this does not isolate formatting alone.
Prose parameters are a substantial Kev weakness in this adapter.
Kev scored 84.52% with parameters in prose, 12.45 points below its description score and 12.53 below full JSON. Jev scored 98.64%. The earlier CLM ordering was different: prose performed better than JSON. A candidate adapter that helps one model does not necessarily help another; this run does not establish the cause of the difference.
Kev: confidence on BFCL descriptions.
Top probability was at least 0.9 in 933/1,253 cases (74.5% coverage). Of those, 1 were wrong (0.11% error among accepted cases). The 10-bin confidence-calibration error was 0.0542, and full-distribution Brier was 0.0583. These are descriptive measurements on this evaluation set; the 0.9 threshold is not an independently validated production gate.
Jev: confidence on BFCL descriptions.
Top probability was at least 0.9 in 1176/1,253 cases (93.9% coverage). Of those, 10 were wrong (0.85% error among accepted cases). The 10-bin confidence-calibration error was 0.0093, and full-distribution Brier was 0.0350. These are descriptive measurements on this evaluation set; the 0.9 threshold is not an independently validated production gate.
BFCL scores here cover function selection.
The suite uses the official multiple (200) and live_multiple (1,053) cases, with official function-name answers and original candidate order. Arguments, execution, parallel calls, memory, and agentic BFCL tasks are not scored. These values are not the overall BFCL leaderboard score or an exact replication of the CLM team's chart.
A model alone is an action policy.
Over five original 60-second T-Rex courses, Kev survived 0/5 with 48 deaths; Jev survived 2/5 with 5 deaths. Planner agreement was 93.3% versus 99.2%; median answer arrival was 35.0 ms versus 133.7 ms. The shield recorded 0 interventions for Kev and 0 for Jev.
A deterministic shield changes the system.
Over five original 60-second T-Rex courses, Kev survived 5/5 with 0 deaths; Jev survived 5/5 with 0 deaths. Planner agreement was 59.6% versus 98.8%; median answer arrival was 66.1 ms versus 132.9 ms. The shield recorded 3536 interventions for Kev and 27 for Jev.
Operational faults are recorded separately.
The four full game runs recorded 3 API errors and 0.000 dropped host seconds in total. Narrow-decision and BFCL runs recorded 0 retries. Per-seed late/discarded answers are available in the raw game results. These timing effects are part of the deployed action-policy outcome.
A hosted API interruption was recovered.
An earlier Jev prose-schema attempt returned HTTP 520 after at least 400 successful cases. That incomplete attempt was excluded from scores, and the full view was rerun from the beginning after adding 520 to bounded retry handling. Completed suites were preserved. The interrupted attempt is recorded in the run ledger and provenance.
Static decisions and live control test different things.
The fixed suites send identical requests to both models. In the game, actions and response times affect future states, so prompts diverge. High agreement with the planner does not by itself establish that the chosen action arrives in time or that a complete agent is reliable. Shielded survival measures a combined model-and-planner system.
The fixtures are a limited, public sample.
The synthetic fixtures were published in the earlier report, and BFCL is a public benchmark. Training overlap was not audited. The task pairs and file judgments are related, and five game seeds are a small sample. These results describe these released deployments on these inputs; they do not establish generalization to unseen real agent traces.
The published Kev results use other datasets.
The model card reports 83.8% accuracy on a locked out-of-domain test and 81.7% on its out-of-domain development split. Those are different data and protocols from these synthetic tasks and the BFCL slice. No numerical agreement with the card can be claimed from these runs. The current checkpoint is also distinct from older Kev releases in the card.
Model cost and report hosting use different billing models.
The pod's GPU quote was $0.72/hour, including setup and idle time while allocated; storage adds to that. The recorded Jev results total 9,662,661 input tokens, about $0.406 at TypeSafe's published $42 per billion input tokens. Warm-up calls and the interrupted attempt are not included in that sum. These are usage estimates, not an invoice. The report serves frozen assets and makes no live model calls.
The temporary resources were removed.
The evaluation instance was deleted after the raw results were copied and verified. Resource inventories were checked for remaining instances, serverless endpoints, and network volumes. No evaluation resource remains allocated. The report continues to work using its frozen files.
[ REPORT / 02 ]
Conclusion.
Kev matched Jev on tool choice, trajectory selection, and ticket routing in these small fixtures, scored higher on relevance and file selection, and scored slightly lower on subgoal completion. On full-schema BFCL function selection, Kev scored 97.05% versus Jev's 98.56%; with prose parameters, that changed to 84.52% versus 98.64%. Kev's local deployment returned answers faster. Without the safety shield, Kev survived 0/5 game courses and Jev 2/5. These results support choosing a decision model together with its input adapter, checking false-positive costs, and evaluating the complete deployed system. They do not establish reliability for an open-ended agent or for unseen production workloads.
[ REPORT / 03 ]