Implementation guide
How to Use a Local LLM Through an OpenAI-Compatible API
Connect the OpenAI SDK to a local LLM running on your own computer, keep machine selection out of application code, and understand the privacy boundary.
The minimum Python change
The client still speaks the OpenAI chat-completions contract. The base URL points to Optimaiz, the application token is read from an environment variable, and the provider extension makes the local execution requirement explicit.
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["OPTIMAIZ_APP_TOKEN"],
base_url="https://optimaiz.endlessriver.ai/v1",
default_headers={"x-optimaiz-trace-mode": "metadata"},
)
response = client.chat.completions.create(
model="local",
messages=[{"role": "user", "content": "Summarize this note."}],
extra_body={"provider": "optimaiz.local"},
)Why the provider field matters
A model name alone does not express every routing constraint. The same workload might be available on a local model and a hosted model. provider=optimaiz.local is an application-level policy: the request must enter the approved local-compute path.
Some tools only accept a base URL and model name. They can use the local alias returned by the OpenAI-compatible models endpoint. Application code that can send extension fields should prefer the explicit provider because it is easier to audit.
One computer first, a pool later
With one connected computer, that machine is the only eligible target. When more computers are approved, the same API call can use the pool. The application still sends no machine ID unless an operator deliberately pins the workload.
Optimaiz filters machines by availability, model artifact, capability, and policy. A ready target is preferred over a merely warm candidate. If the selected machine disappears before output begins, the request can be reassigned to another eligible machine.
The privacy boundary is not the same as direct loopback
A managed Optimaiz local call runs inference on your machine, but the prompt and response transit the Optimaiz cloud gateway. Metadata trace mode keeps prompt and response content out of long-term interaction history; it does not make the network path device-only.
When data must never leave the device, call the Helm loopback endpoint directly on 127.0.0.1 and disable cloud fallback. That path trades managed pooling and cloud routing for a stricter device-only boundary.
Production checks before moving traffic
Verify the actual model reported in response metadata, observe time to first token and total latency, and test what happens when the computer sleeps. Define whether a request may be reassigned before output, whether hosted fallback is allowed, and whether content may be retained.
- Keep application tokens in environment variables or a secret manager.
- Use stable workload keys instead of hard-coding machine identities.
- Test local-required and local-preferred policies separately.
- Treat post-output retry as unsafe unless the operation is explicitly idempotent.
Frequently asked questions
Can I use the standard OpenAI Python or TypeScript SDK?+
Yes. Point the SDK at the Optimaiz /v1 base URL and send the Optimaiz application token. Use provider optimaiz.local for managed local execution.
Do I choose which laptop runs each request?+
Not by default. Optimaiz selects an eligible ready computer; an operator can still prefer or pin a machine when needed.
Is an Optimaiz local call fully offline?+
No. Managed local calls transit the Optimaiz cloud gateway. Use the direct Helm loopback endpoint for data that must remain device-only.