Ornith-1.5 35B-A3B Review for Local Coding Agents

This Ornith-1.5 35B review starts with a limit: I have not completed a local benchmark run on my own hardware yet. So I will not turn the official score table into an editorial verdict. That is how teams fool themselves.

The useful question is narrower: can a local coding-agent team build a repeatable test around this model before it touches shared development work? Demos show the ceiling. Production shows the floor.

The official Ornith-1.5-35B-A3B model card lists it as an MIT-licensed reasoning model, about 35B total parameters, about 3B active parameters per token, with a 262,144-token context window. The card also says serving needs Transformers >= 5.8.1, vLLM >= 0.19.1, or SGLang >= 0.5.9.

The 30-second answer: treat Ornith-1.5-35B-A3B as a promising MoE coding model for local coding agent pilots, not as a drop-in team default. Before running further, narrow the variables.

Understand the 35B-A3B Deployment Profile

Mixture-of-experts architecture and active parameters

The “35B-A3B” label matters. This is not a dense 35B model where every token activates the full parameter set. The model card describes Ornith-1.5-35B-A3B as a mixture-of-experts model with about 3B active parameters per token.

That helps explain why the model is interesting for local inference. Active parameters affect per-token compute, while total parameters still affect storage, memory placement, and loading behavior. A good single output does not mean the production workflow is ready. For coding agents, the real test is tool-call order, patch quality, context discipline, and whether it keeps following the task after 40 minutes of repo work.

Runtime, memory, and context assumptions

The official full-precision serving recipe targets 2x80GB GPUs for 256K context headroom. The same card points local users to llama.cpp, Ollama, Atomic.chat, Hermes Agent, and OpenClaw through OpenAI-compatible endpoints.

For local pilots, the GGUF status is the practical detail. The Ornith-1.5-35B-A3B-GGUF repository currently lists BF16, Q4_K_M, Q5_K_M, Q6_K, Q8_0, plus an mmproj file. Do not record “GGUF” as the test condition. Record the exact file name.

The context claim also needs care. The card says 262,144 tokens and describes YaRN scaling for longer windows. That is not permission to stuff a whole monorepo into every prompt. Long context is a workload to test, not a badge to repeat.

Build a Reproducible Local Coding Test

Hardware, runtime, quantization, and test controls

A useful Ornith local benchmark starts with boring fields. Boring is good here. Boring lets someone rerun the same failure next week.

Field	Record it exactly
Hardware	GPU/CPU, VRAM or unified memory, RAM, OS, driver
Runtime	llama.cpp, vLLM, SGLang, Ollama, exact version or commit
Model file	HF repo, revision SHA, quant file, tokenizer/template
Context	requested context, real prompt tokens, max output tokens
Concurrency	parallel sessions, batch size, cache settings
Sampling	temperature, top_p, top_k, repeat penalty
Task set	repo name, commit SHA, issue/task ID, expected tests
Failures	patch failure, tool-call failure, loop, timeout, bad edit

Use official benchmarks as orientation, not as your result. The Ornith-1.5 blog explains the self-improvement loop and reported evaluations. For coding-agent relevance, cross-check task types against Terminal-Bench 2.1 and SWE-bench Verified, but run your own repo tasks.

Repository tasks, tool calls, and failure capture

I would start with 12 tasks, not 120. Four small edits, four debugging tasks, two long-context navigation tasks, and two “agent hygiene” tasks: read docs, inspect files, modify only the right place, run tests, report honestly.

For a local coding agent, capture the transcript. Did it call tools before answering? Did it inspect the failing test? Did it edit unrelated files? Did it keep a failed assumption alive after the repo contradicted it? The model was not lazy when it made the wrong edit. It may have worked very hard to repeat the mistake.

Interpret Performance by Workload

Code editing, debugging, and long-context tasks

Score by workload, not by one blended number.

Code editing should measure patch acceptance and diff cleanliness. Debugging should measure whether it finds the failing boundary before rewriting. Long-context work should measure retrieval discipline: can it use 80K or 160K tokens without looping, forgetting constraints, or inventing file state?

Tool calls need their own lane. The model card describes reasoning and tool-call parsing through OpenAI-style fields, and vLLM documents --enable-auto-tool-choice in its OpenAI-compatible server docs. SGLang also documents parser choices in its tool parser guide. Pin these settings, or the comparison becomes fog.

Latency, memory use, and output reliability

For production, speed matters, but stable speed matters more. Track cold load time, first-token latency, tokens per second, peak memory, and failure after context growth. If one run is fast and the next one stalls under the same conditions, that belongs in the result.

My cutoff for a pilot would be simple: if the model cannot pass at least 10 of 12 fixed repo tasks with clean transcripts, no shared endpoint yet. If it passes but fails under concurrency, keep it as a single-user workstation model. This conclusion only fits this task volume.

Frequently Asked Questions

Which test artifacts should teams publish internally?

Publish the prompt, repo commit, expected tests, full transcript, runtime command, quant file, context size, sampling settings, final diff, pass/fail result, and one failure note. The failure note is not optional. That is where the next operator learns.

How should a team handle an upstream checkpoint change?

Pin the Hugging Face revision SHA for both the main checkpoint and GGUF repo. When upstream changes, rerun the same fixed task set before updating the shared alias. Do not let “latest” silently replace “tested.”

Who should own the local inference service?

The owner should be the team that can read both sides: infra metrics and coding-agent behavior. If infra owns only uptime, bad patches slip through. If developers own only output quality, memory and concurrency rot quietly.

When is a shared local endpoint safer than desktop installs?

A shared endpoint is safer when multiple people need the same model file, same template, same logging, and same rollback. Desktop installs are fine for exploration. Team work needs one language for failures.

What rollback plan is needed before a pilot?

Keep the previous endpoint alive, preserve the old model alias, freeze a known-good quant, and define the switch-back trigger before testing starts. For this Ornith-1.5 35B review, my trigger would be tool-call regression, repeated unauthorized edits, or unstable behavior beyond the agreed context range.

The model is worth testing. But the review is not the benchmark table. The review is whether your team can reproduce the result, catch the failures, and roll back before one bad local coding agent becomes everybody’s problem.

Previous posts: