AI for biology has several evidence levels. Keep them separate
Biological agents can turn existing measurements into useful leads. A map of the evidence shows where interpretation ends and validation begins.
Topic
Evaluation
AI writing, AI reviewing and the verification bottleneck
Citation matching, technical criticism, executable checks and formal proofs answer different questions. A map of what AI review can establish.
From fuzz tests to proofs: inside an AI verification workflow
A maintainer’s guide to the artifacts behind AI-written proofs, from fuzz targets and binary hashes to assumptions and release changes.
What the AISI incident reveals about evaluation boundaries
AISI’s incident separates task success, permitted action and containment. Its follow-up experiments explain why each needs its own evidence.
Coding-agent productivity needs a systems denominator
Coding-agent activity is easy to count. Useful delivery requires accounting for review, infrastructure, retries and the quality of accepted changes.
FlashProve and the limits of a cryptographic AI receipt
FlashProve makes model execution a checkable claim. Its research prototype also exposes the limits of what an AI receipt can guarantee.
GPT-6.1 Terra brings a specialised pitch to computer work
Terra targets visual reasoning and screen-based tasks. Its release makes the model, action environment and saved-result checks separate questions.
Model merging versus distillation: what the new evidence compares
Merging changes parameters; distillation trains on behavior. P²-MOPD explores a hybrid, with evidence bounded by its models, teachers and training budget.
Muse Spark Open and Apertus 1.5 expose different kinds of openness
Weights, training evidence, licence terms and reproducible evaluations are separate criteria. Two releases make the differences concrete.
The sleep framework puts forgetting at the centre of AI learning
Sleep consolidates temporary information into model parameters. Its experiments clarify what retention means and what the extra computation buys.
TorchNeuron’s public beta: what PyTorch teams should verify
TorchNeuron makes Trainium a native PyTorch device. Operator fallbacks, static shapes and numerical checks still decide migration readiness.
World models, video generators and simulators serve different jobs
Interactive generation, cinematic video and physics simulation expose different controls and evidence. Here is how to match them to a task.
Fact check: LiteLLM's Rust benchmark claims, tested against its own repo files
LiteLLM's homepage and benchmark blog credit its Rust gateway with about 7x lower overhead, 9x less memory and about 4.5x more requests per dollar. Fifteen claims, set against the CSV and JSON files LiteLLM published, with neutral verdicts, including the ones that check out.
LiteLLM's Rust gateway benchmark, checked against LiteLLM's own data
LiteLLM says its Rust gateway has about 7x lower overhead, 9x less memory and 4.5x more requests per dollar than the next-closest gateway. LiteLLM's own benchmark files show a latency and throughput near-tie, a mislabeled widest-pair memory chart, a modeled cost and a config gap, plus Rust leads on CPU and memory.
LiteLLM vs Bifrost latency: tracing the 7x overhead figure
LiteLLM says its Rust gateway has roughly 7x lower overhead than Bifrost. The concurrency sweep in LiteLLM's own benchmark repo shows Bifrost lower at concurrency 1 and 64, Rust lower at 16, a tie at 4, and throughput gaps under 1%. The headline number comes from a separate file with no stated concurrency.
Logging on vs logging off: the configuration gap in LiteLLM's benchmark
LiteLLM's benchmark blog says no gateway logged or persisted anything. The Bifrost v1.6.4 config it published never sets enable_logging or a log store, and the defaults then switch on logging, a SQLite store and built-in plugins. That is inferred from v2.4.0 code, not confirmed on v1.6.4.
A Rust beta vs an old Bifrost build: the version problem in software benchmarking
LiteLLM's Rust gateway benchmark compared an early Rust beta, which its own blog says is still landing streaming, against Bifrost v1.6.4, an old build. The homepage never names the version. Why version parity decides what a benchmark can claim, what the record shows, and a checklist for readers.
Agent benchmarks are now 44% of new AI benchmarks, and most are too small to rank close models
An analysis of 352 sized benchmarks in The Frontier Wire's AI benchmarks dataset finds agent benchmarks are about 4.5 times smaller than the rest. That makes many leaderboard gaps statistical noise. The data, the method, and a table of the smallest score gap worth taking seriously at each benchmark size.
LLM observability: what to capture, trace and alert on in production
A practical explainer on LLM observability: the per-call record, time to first token, OpenTelemetry GenAI conventions, trace structure for agents, sampling and PII redaction, Prometheus alert thresholds, feedback joins and evals on production traffic, plus the open-source llm-observability-stack agent skill.
How to use Jev: where a System One model fits next to an LLM
TypeSafe's Jev returns typed answers with probabilities instead of text. Its documentation, the first developer reports and early independent papers show where it fits in a pipeline, what the published price works out to, and why its probabilities need checking on your own data.
System One models, explained: typed answers and calibration
TypeSafe's Jev returns typed answers with probabilities instead of text. The idea borrows Kahneman's System 1, a calibration objective called RLCD, and a strong claim about hallucination. The concepts, the evidence behind them, and where each one stops.
System One models vs LLMs: most AI calls should decide, not talk
Routing, gating, judging and choosing an agent's next step are decisions, not writing. Splitting decision from generation is the right architecture, and TypeSafe's Jev is the first product built for it. Trust will come from independently measured calibration, not a promise that it cannot hallucinate.
TypeSafe launches Jev, a model that returns decisions instead of text
TypeSafe AI came out of stealth on September 15 with $40 million in seed funding and Jev, a model that answers typed questions about a block of text with probabilities instead of prose. The pricing is unusual, the speed claims are large, and the first independent tests are mixed.
How to compare LLM evaluation frameworks without getting sold to
Every evaluation framework claims to measure quality. The differences show up in who can write the test, whether a trajectory can fail, and what happens to a bad score in production. Here are the criteria, the research that justifies them, and nine current tools read against them.
Stop calling it hallucination
The word implies a glitch in an otherwise reliable mind. The published evidence says something plainer: models are trained and graded to guess, and the remedy depends on saying so.
Reading a model card like an auditor
The model card was proposed as a one-to-two-page disclosure. Current cards range from a 135-page system card to a benchmark table with a licence. Reading one well means checking it against the original template, the EU AI Act and the Stanford transparency index.