Ornith API Deployment: vLLM, SGLang, Ollama

It’s Dora. I had the route almost ready: base URL, model name, SDK wrapper, fallback slot. The blank cell was the one that mattered: does the Ornith API behave like the endpoint my coding agents already expect, or am I about to ship another one-off integration that someone will have to babysit next month.

I paused here. Ornith-1.0 is interesting, but this note is not a Hugging Face model-card audit. I rechecked the Ornith-1.0-35B model card on July 8, 2026 for deployment fields only: runtime versions, context parameters, parser names, and example serving commands. Benchmarks stay model-card claims. Routing decisions need local evidence.

Short version: vLLM and SGLang are the main paths if you need an OpenAI-compatible endpoint. GGUF, llama.cpp, and Ollama are useful for local or internal testing. Production routing starts only after compatibility, reasoning output, tool calls, latency, logs, and rollback behavior are proven in your own stack.

Ornith API Deployment Paths

OpenAI-compatible serving goal

The goal is not “run the model.” That part is too easy to write down and too easy to misunderstand.

The useful goal is narrower: expose Ornith-1.0 behind an OpenAI-compatible API that existing SDKs, coding agents, eval harnesses, and gateway logic can call without special treatment. The model card’s current serving examples point to /v1/chat/completions, with reasoning_content separated from final content and tool calls surfaced through standard tool_calls.

That matters because agentic coding systems usually have more than one dependency on the API shape. They expect message roles. They expect streaming to behave. They expect function calls to come back in a predictable field. They expect failures to be retryable or classifiable. If one of those assumptions breaks, the model may still be “working,” but the product route is not.

vLLM’s OpenAI-compatible server documentation is the reference I would use for the vLLM side of that contract. SGLang documents the same direction through its OpenAI-compatible APIs, and its Ornith-specific cookbook gives a more model-aware deployment matrix.

Here is the decision boundary I use:

Path	Best use	Main risk
vLLM	Production-style HTTP serving, OpenAI SDK compatibility, prefix caching, gateway integration	Parser and long-context settings must match the model behavior
SGLang	Model-specific serving recipes, tool parser control, reasoning parser control, deployment matrix tuning	Cookbook defaults may differ from the model-card quickstart
GGUF + llama.cpp	Local server, edge testing, CPU/GPU mixed local inference	Throughput and tool behavior need separate validation
Ollama	Fast local trial, developer workstation testing	Good for internal use first, not a production route by default

Good enough. That is the most honest assessment I can give at the path-selection stage.

vLLM, SGLang, GGUF, llama.cpp, and Ollama roles

As of the July 8, 2026 recheck, the Ornith 35B model card lists these current runtime requirements: Transformers >=5.8.1, vLLM >=0.19.1, and SGLang >=0.5.9. It also shows a 262144 context setting in both serving paths: --max-model-len 262144 for vLLM and --context-length 262144 for SGLang.

For vLLM, the model-card recipe uses vllm serve deepreinforce-ai/Ornith-1.0-35B, a served model name of Ornith-1.0-35B, --tool-call-parser qwen3_xml, and --reasoning-parser qwen3. I would treat that as the first smoke-test recipe, not as a finished production manifest.

For SGLang, the model-card recipe uses python -m sglang.launch_server, --tool-call-parser qwen3_coder, and --reasoning-parser qwen3. The SGLang Ornith-1.0 cookbook also lists Ornith checkpoints and says the 35B recipes use tensor parallelism 2 in that deployment matrix. That is why I would not copy one TP value blindly. The right TP value belongs to your node, memory budget, context target, and concurrency target.

The GGUF path is different. The Ornith-1.0-35B-GGUF model page lists GGUF files and repeats the local runtime examples. The model card shows llama-server -hf deepreinforce-ai/Ornith-1.0-35B-GGUF --port 8000 -c 262144 for llama.cpp, and ollama run hf.co/deepreinforce-ai/Ornith-1.0-35B-GGUF for Ollama.

That gives you a clean split. vLLM and SGLang are endpoint candidates. GGUF is a local inference artifact. llama.cpp can still expose an OpenAI-compatible server, and its HTTP server README documents OpenAI-compatible routes, but I would keep it in the internal lane until the same tool-call and reasoning tests pass.

Ollama is useful when I want to know whether the model is worth more work before I burn time on the serving stack. Its OpenAI compatibility docs make it easy to test existing clients against a local service. I still would not treat “runs in Ollama” as “ready for production routing.” Different sentence.

Build a Production Serving Plan

GPU sizing, context length, endpoint shape, and SDK compatibility

I do not start GPU sizing with a GPU number. I start with the workload.

For Ornith 35B, write down the real request shape first: average repo context, worst-case prompt, expected output length, tool-call count per task, concurrent sessions, streaming requirement, and timeout budget. Then run a staged load test with the exact serving stack you plan to operate.

The model card’s vLLM example mentions an 8×80GB GPU node with tensor parallelism 8. SGLang’s cookbook matrix gives different TP defaults for its selected hardware recipes. Those are not contradictions to smooth over. They are deployment assumptions. Keep them labeled.

Context length needs the same discipline. The model-card examples use 262144. That does not mean every public route should expose 262K tokens. Long context raises KV-cache pressure, queue time, memory fragmentation risk, and failure cost. For coding agents, I usually test three tiers:

Tier	Purpose	What I watch
Small	Quick edits and single-file tasks	First-token latency, tool-call validity
Medium	Repo-level questions and patch planning	Cache reuse, completion quality, timeout rate
Long	Large codebase or trace-heavy tasks	Memory pressure, queue behavior, failure recovery

Endpoint shape should be boring. That is the point.

Use a stable base_url, a served model name that does not change with every weight reload, and a key path that your gateway can rotate. If the model name is Ornith-1.0-35B in vLLM but deepreinforce-ai/Ornith-1.0-35B in a client config, decide which one your platform owns. Then freeze it behind a route alias.

One fewer switch. Sounds small. Adds up fast.

Tool-call support, reasoning output handling, and smoke tests

Ornith-1.0-35B is a reasoning model. The current model card says assistant output starts with a <think>...</think> block unless the serving parser separates it. In production, I do not want raw thinking tags leaking into final user-visible content or agent logs that were built for final answers only.

The smoke-test set is small, but I would not skip it:

/v1/models or equivalent model discovery returns the served model name expected by the client.
A basic chat completion returns final content without parser errors.
Reasoning output lands in reasoning_content, or your application intentionally strips it.
A deterministic tool prompt returns one valid tool_calls object with valid JSON arguments.
A no-tool prompt does not invent a tool call.
Streaming and non-streaming responses use the same message fields.
A long-context request fails cleanly when it exceeds your configured limit.
The gateway records latency, request ID, model name, parser configuration, and stop reason.

Found the pattern on the third try: most failed “model” launches are really contract failures. The model answered. The client expected a different shape. The agent crashed somewhere in the gap.

For tool calls, save both the raw response and the parsed object during testing. For reasoning, save whether the parser was enabled, which parser name was used, and whether the final answer still includes thinking tags. If that sounds tedious, good. It is cheaper than debugging a silent production regression.

Operational Checks Before Routing Traffic

Latency budget, logging, failure modes, and fallback candidates

Before traffic moves, define the latency budget in the unit your users feel.

For coding agents, p95 total task time usually matters more than single-token throughput. First-token latency still matters because it affects perceived stall. Queue time matters because agent loops multiply waiting. Tool-call round trips matter because one coding task can make several calls before it writes a final answer.

I would log at least this:

Field	Why it matters
request ID	Ties gateway, server, and agent logs together
served model name	Catches accidental model swaps
runtime and version	Makes upgrade regressions traceable
context length used	Explains memory and latency spikes
parser settings	Explains reasoning and tool-call differences
finish reason	Separates success, length stop, tool handoff, and failure
fallback route	Proves whether rollback actually happened

Failure modes need names. “Bad output” is not a useful incident category.

Use buckets like parser failure, malformed tool arguments, timeout, OOM, queue saturation, context overflow, client schema mismatch, reasoning leakage, and regression against eval set. Once the buckets exist, routing decisions become less theatrical. You are no longer arguing whether the model “feels ready.” You are looking at failure rates.

Fallback candidates should use the same API shape where possible. If the primary route is Ornith 35B behind OpenAI-compatible chat completions, the fallback should not require a new client adapter during an incident. It can be another self-hosted coding model, a hosted provider model, or a smaller internal model for degraded mode. The important part is that fallback is tested before you need it.

This is where my data ends: I would not put a local endpoint on a public production route until I have at least one clean upgrade rehearsal. Upgrade vLLM or SGLang in staging. Re-run the same compatibility tests. Compare reasoning fields, tool-call shape, latency, and failure categories. Then decide.

FAQ

Who owns API compatibility tests after serving-stack upgrades?

The platform owner owns them. Not the model researcher. Not the application team that happens to notice the breakage first.

The model owner can say which parser and template the model expects. The serving owner must prove that vLLM, SGLang, llama.cpp, or Ollama still returns the fields the application contract requires. The application owner should provide the fixture prompts and agent tasks.

In practice, I would keep a small compatibility suite beside the route config. Every serving-stack upgrade runs the same suite before the route is promoted. If the parser name changes, the test has to fail loudly.

What evidence should be saved before changing production routing?

Save the boring evidence.

Keep the model repo and commit or last-modified snapshot, serving runtime version, full launch command, parser settings, context length, GPU profile, request fixtures, raw responses, parsed responses, latency summary, error buckets, and rollback result.

Also save the reason for the change. “New model looks better” is not enough. “Tool-call validity improved from staging baseline while p95 stayed inside budget” is useful. “Long-context tasks stopped timing out after reducing route max context” is useful.

This conclusion has an expiration date. Models update fast. Serving stacks update faster.

When should a local Ornith endpoint stay internal only?

Keep it internal when the endpoint is useful for engineers but not yet stable for users.

That includes cases where GGUF runs locally but tool calls are unverified, reasoning tags leak into final output, latency swings under modest concurrency, logs cannot tie requests to failures, or fallback has not been rehearsed. It also includes the very common case where a team wants to expose the full 262144 context because the model supports it, but has not measured the memory and queue cost.

An Ornith API earns production traffic when the route is boring: predictable endpoint shape, known parser behavior, measured latency, named failure modes, and a fallback that has actually been used in a test. Until then, keep it inside the lab. Run it yourself. That will tell you more than anything I say.

Previous posts: