How to Deploy Inkling: vLLM and SGLang

The worst Inkling deployment ticket is also the shortest one: “deploy inkling model.”

I paused there. For a 7B model, that might be enough. For Inkling, it is not. According to the official Thinking Machines announcement, Inkling is a 975B-total-parameter MoE model with 41B active parameters and a 1M-token context window. The Hugging Face model card also lists multimodal input, BF16, NVFP4, and several inference paths.

That confirms the model exists. It does not confirm your deployment is ready.

This note is for infrastructure and inference teams planning to deploy Inkling model through vLLM, SGLang, or a related serving stack. It is not a one-click local tutorial.

Start With Deployment Reality

Weight scale, active parameters, and why this is not a one-click local tutorial

Inkling is sparse, but sparse does not mean light. The active parameter count affects per-token compute. The total parameter count still affects storage, loading, memory placement, parallelism**, restart time, and operational risk.**

This is where a lot of local-run articles go wrong. They see open weights and jump straight to a command. I would not do that here.

To run Inkling locally, first define the goal. A research run can tolerate manual retries and slow startup. An internal endpoint needs repeatable auth, logging, request limits, and owner rotation. A production service needs capacity approval, observability, fallback routing, rollback, and incident records.

Those are different jobs. Treating them as one job is how deployment notes become incident notes.

Research run, internal endpoint, and production service are different goals

For research, the useful record is small: checkpoint, framework version, prompt set, output samples, and known gaps.

For an internal endpoint, add schema control, queue limits, basic dashboards, restart behavior, and ownership. Someone has to answer when the endpoint stops responding. “Whoever launched it” is not an owner.

For production, the bar moves again. The team needs SLOs, GPU budgets, abuse limits, context policies, payload validation, and fallback rules. Multimodal support also means image and audio preprocessing become part of the service, not an afterthought.

Good enough is allowed for experiments. It is not allowed for shared infrastructure.

Confirm Framework Support

vLLM, SGLang, Transformers, custom kernels, and multimodal processors

Inkling vLLM support is publicly documented. The vLLM team’s Inkling note says both thinkingmachines/Inkling-NVFP4 and BF16 Inkling are supported, with multimodal input, 1M native context, tensor/data/expert/pipeline parallelism, prefix caching, speculative decoding, and disaggregated serving.

Inkling SGLang support is also public. The SGLang and Thinking Machines Day-0 support post covers BF16 and NVFP4 checkpoints, NVIDIA and AMD paths, speculative decoding, prefill-decode disaggregation, multi-LoRA serving, and HiCache.

That is useful. Still verify the exact build. Inkling needed custom runtime work for short convolution, relative attention, and shared-expert MoE behavior. A framework name alone is not enough.

Transformers support is better treated as a correctness and processor path, unless your team is prepared to build production serving around it. Tokenization is not the whole input layer. Inkling is multimodal, so image and audio payload rules matter.

What to verify before assuming support for NVFP4 or long context

Inkling quantization is a deployment decision, not a magic discount. NVFP4 should be verified against GPU generation, CUDA stack, container image, framework commit, kernels, and fallback behavior. If there is no official or framework confirmation for your exact environment, write “needs verification.”

Long context needs the same caution. A 1M-token context window is a model capability. It is not a reason to expose 1M tokens to every request. Long prefill, KV cache pressure, queue delay, and tail latency can change the service shape fast.

My minimum preflight list:

Checkpoint: BF16 or NVFP4.
Framework: vLLM, SGLang, Transformers, plus version or commit.
Hardware: GPU type, memory, interconnect, driver, CUDA.
Context policy: default limit, elevated limit, rejection behavior.
Modality policy: text only, image enabled, audio enabled, or staged.
Fallback: model, provider, or degraded route.

This is where my data ends. Anything not checked stays marked as must-verify.

Plan Infrastructure Requirements

GPU memory, tensor parallelism, storage, network, and cold start

Start with memory, but do not stop there. Inkling inference requirements include weights, KV cache, temporary buffers, multimodal processors, scheduler overhead, and any speculative decoding components.

The most concrete public packaging reference I found is NVIDIA’s NIM support matrix, which lists an Inkling NVFP4 SGLang configuration on 8 B200 GPUs with 192 GB GPU memory and 670 GB disk. I would use that as a reality check, not a universal minimum.

Tensor parallelism is also a network plan. Cross-node serving depends on interconnect quality. Weak topology can look like “the model is slow,” when the real issue is communication overhead.

Cold start needs its own budget. Large checkpoints, container pulls, cache misses, sharded weight placement, and kernel compilation can all delay the first useful token. Pre-warm, drain, restart, and rollback paths should be written down before launch.

KV cache, 1M context, concurrency, and throughput tradeoffs

Concurrency and context length compete for memory. A service tuned for short prompts can fall over when a few long-context jobs arrive. A service tuned for 1M context may waste capacity on normal chat traffic.

Split traffic early:

short text
long text
image-plus-text
audio-plus-text
batch jobs

Do not hide all of that behind one endpoint and hope the scheduler figures it out. Schedulers are good. They are not budget owners.

Throughput estimates should come from real traces: input length, output length, modality mix, cache hit rate, retry rate, and peak concurrency. Framework benchmarks are useful references, but not your capacity model.

Build a Serving Plan

Endpoint schema, batching, queueing, observability, and failure handling

An OpenAI-compatible chat endpoint is a practical starting point, especially with vLLM and SGLang. The schema still needs local decisions.

Define supported content parts. Decide whether images, audio, tool calls, and long-context requests are day-one features or staged features. Decide whether the served model name is the raw Hugging Face ID or an internal alias.

Batching should follow traffic class. Keep interactive requests away from large batch jobs. Put long-context requests in their own queue. Validate media payloads before they reach the model server.

Observability should record checkpoint revision, framework, GPU group, quantization format, context length, input/output tokens, modality, queue time, prefill time, decode time, finish reason, retry count, and fallback result.

Without those fields, incident review becomes guesswork. I do not recommend building production around guesswork.

Fallback models and staged rollout

Fallback must be specific. “Use a smaller model” is not a plan.

For coding traffic, fallback may need tool-call compatibility. For multimodal traffic, fallback may disable audio but keep image input. For long-context jobs, fallback may summarize first, then route to a smaller model.

Rollout should start with shadow traffic. Compare accepted-answer rate, refusal behavior, tool-call validity, latency, cost per accepted output, and error rate. Then move to internal users. Then low-risk production traffic. Then multimodal and long-context expansion.

Rollback should name the action: disable NVFP4, reduce context, turn off audio, move traffic to a provider, or revert to another model. Vague rollback plans fail under pressure.

FAQ

Who signs off on GPU capacity assumptions?

The infrastructure owner signs off on the technical capacity model. Finance or procurement signs off on committed GPU spend. The application owner signs off on the user-facing service tradeoff.

For Inkling, that approval should include checkpoint format, GPU class, expected concurrency, context policy, modality mix, queue behavior, and fallback route.

What incident record is needed after deployment instability?

Record the checkpoint, framework version, GPU setup, quantization format, request shape, modality, context length, queue depth, KV cache settings, error type, mitigation, and rollback decision.

For Inkling vLLM or Inkling SGLang failures, also separate prefill, decode, media preprocessing, kernel execution, distributed communication, and restart issues. They are not the same failure.

When should teams stop self-hosting and switch to a provider?

Stop self-hosting when the operational burden is larger than the control gained. Common signs: poor GPU utilization, rising on-call noise, slow framework upgrades, unstable multimodal edge cases, or capacity work that consumes more time than the product feature justifies.

Self-hosting can still make sense for sensitive data, custom routing, deep integration, or fine-tuned variants. For occasional base-model access, a provider route is usually cleaner.

Conclusion

To deploy Inkling model responsibly, start with the deployment reality, not the launch command. Confirm current vLL M, SGLang, Transformers, NVFP4, BF16, long-context, and multimodal support before publishing the plan. Then size the service around actual traffic, not the most flattering benchmark.

Open weights give teams control. They also hand teams the bill for every hidden assumption. That is the trade. Treat deploy Inkling model as an infrastructure decision first, and the tutorial part becomes much less dangerous.

Previous posts: