How GPU for Agents works

01Overview

One command, gpu, rents a GPU from a cloud provider, serves a model or runs code on it, and makes sure it stops. Agents reach it through a skill and an MCP server. Applications call the same command with --json.

Two planes

You never have to stop the GPU yourself. The one thing a client has to handle is a machine that has been stopped: it calls gpu setup again and gets a fresh endpoint.

02Quickstart

Let the agent install it

Paste this into Claude Code, Codex, Gemini CLI or Cursor, naming the one you use:

Set up GPU for Agents for me. I'm using Claude Code. Fetch https://gpu4agents.jambu.ai/install.md with curl and follow its steps.

The agent runs the installer, gives itself the skill and the MCP server, and checks your provider account. Nothing is rented during installation.

Connect the account that pays for the GPUs

When the agent asks, open your terminal app (not the chat) and type gpu login. It takes about two minutes, and the key never passes through the conversation.

Connect a GPU provider, so your agent can rent GPUs on its own.
Your key stays on this computer; the agent never sees it.

  1) Vast.ai  built in; GPUs by the hour from a few cents, pay as you go
  2) Another provider I already use  RunPod, Lambda, my company's servers, ...: your agent adds it

Choose [1]: 1

Vast.ai, in 3 steps:
  1. Create a Vast.ai account: https://cloud.vast.ai/
  2. Add some credit ($5 to $10 is plenty to try it): https://cloud.vast.ai/billing/
     You pay only while a GPU runs, and idle ones are shut down for you.
  3. Create a key and copy it: https://cloud.vast.ai/manage-keys/
Already have an account? Skip to the key.

Paste your key here and press Enter (it stays hidden):

The key is checked with the provider before it is saved in ~/.gpu/credentials (mode 600). Pick "Another provider" and the command tells you what to ask your agent: it writes the adapter, then you run gpu login --provider <name>. gpu login --check says whether an account is connected, without changing anything.

Or by hand

$ curl -fsSL https://gpu4agents.jambu.ai/install.sh | sh -s -- --agent auto
# or: uv tool install --python 3.12 gpu4agents  /  pipx install gpu4agents
$ gpu init --model Qwen/Qwen2.5-7B-Instruct   # writes gpu.yaml
$ gpu validate                                # config, credentials, real capacity; fails before renting
$ gpu offers                                  # matching machines and their hourly price
$ gpu setup                                   # rent, install the watchdog, serve, wait until ready
$ eval "$(gpu env)"                           # OPENAI_BASE_URL, OPENAI_API_KEY, JAMBU_MODEL_ID

03Architecture

Three places are involved: your machine, the provider's API, and the rented GPU machine. Control goes through the gpu command and the watchdog. Data goes straight from your client to the model on the GPU.

What is in the package

The core never checks which provider it is talking to. It checks what the provider can do (provider.capabilities), and a test fails the build if core/ ever mentions a provider by name.

04Process flow: gpu setup

What happens between "rent a GPU" and a working endpoint. The first rent usually takes 5 to 15 minutes, mostly the image pull and the model download. Restarting a stopped machine took 41 seconds for a small model.

Through MCP, setup never blocks the agent

A first rent outlasts any client's tool timeout, so gpu_setup starts setup as a background job, waits up to 45 seconds, and answers status: "running" until the model is ready. Calling it again re-attaches to the same job and never rents a second machine. The job outlives the MCP server, so a restarted client finds it again.

05Lifecycle states

Every provider's states are mapped onto one set, so the CLI and the rule behave the same on any provider.

06The watchdog

The CLI process can't be in charge of shutdown: close the laptop and a local timer dies while the GPU keeps billing. So setup installs a watchdog on the machine itself, and the machine stops itself.

What counts as activity

Traffic is read from the server's own counters. The first read only sets a baseline, so old totals don't keep a machine up, and a counter that goes down means the server restarted, so it rebaselines. A failed probe never counts as activity.

When the provider call fails

• A failed stop never disarms the guard. It logs provider_error and guard_retry and tries again on the next tick.
• If the provider accepted the call but the machine is still running after 5 minutes, the watchdog issues it again (guard_reissue).
• After 30 minutes of failed stops it destroys the machine instead (stop_failed_escalation): a machine that bills forever costs more than a lost disk.

A second line of defense

The same rule runs from outside too. gpu guard evaluates it from cron or CI, and gpu sweep --install-timer runs every 5 minutes over every registered project, then stops orphans running longer than 24 hours and destroys orphans stopped longer than 24 hours. It only ever sees machines labeled jambu-; nothing else in the account is touched. It's a dry run until --apply.

07The shutdown rule

One pure function, evaluate(snapshot, config), decides whether a machine may keep billing. It checks in this order and stops at the first answer. Every automatic shutdown records its reason.

"STOP" is the cheapest action that ends billing: a stop where a stopped machine is free, a destroy with auto_destroy: true or when the provider can't stop. gpu status prints the decision it would take right now, so you can see the watchdog's next move before it happens.

08Data flow

What lives where, what crosses the network, and which secrets go where.

Files on your machine

Keep .gpu/ on persistent storage. A container that loses it forgets its instance and rents a second one, leaving the first to the watchdog and the sweep. Add .gpu/ and .env to .gitignore.

09Audit and cost

Every lifecycle action appends one JSON object to the event log, so any GPU cost can be traced back to a decision and a reason. The watchdog keeps its own log on the machine, and gpu status and gpu events merge it into yours.

$ gpu events                 # the lifecycle log, newest last
$ gpu --json status          # state, cost so far, idle time, the watchdog's next move
$ gpu ls                     # every jambu- machine in the account, and which project owns it

Each line carries ts, event and pid, plus the project's context (instance id, provider) and the event's own fields, such as the reason, the price or the decision detail.

Event vocabulary

10Integrating a coding agent

The agent gets two things: a skill that teaches it the rules, and a local MCP server that exposes the gpu commands as tools.

$ gpu agent install -t claude            # this project; --global for every project
$ gpu agent install -t auto --global     # every agent found on this machine
$ gpu mcp --project /path/to/project     # any other MCP client, over stdio

What keeps the agent in line

• The skill tells it to show the price and ask before renting, to wait out a slow setup instead of starting over, never to touch the provider key, and to offer stop or destroy when done.
• The tools are annotated: gpu_destroy is marked destructive and the read-only tools read-only, so your client can let gpu_status run freely and ask for every gpu_setup. In Claude Code: "allow": ["mcp__gpu__gpu_status", "mcp__gpu__gpu_offers", "mcp__gpu__gpu_ls"].
• gpu.yaml holds the hard limits. The agent can't raise them without editing a file you can see in the diff.
• The watchdog stops the machine whatever the agent does, and the sweep catches anything left behind.

11Integrating an application

No SDK. An app in any language runs gpu --json setup, reads the JSON, and points its OpenAI client at the result. When a request fails because the machine is gone, it runs setup again and retries once.

import json, subprocess
from openai import OpenAI, APIConnectionError

def ensure_gpu(cwd="/srv/my-app"):
    out = subprocess.run(["gpu", "--json", "setup"], cwd=cwd, check=True,
                         capture_output=True, text=True).stdout
    return json.loads(out)

conn = ensure_gpu()
def ask(prompt):
    global conn
    for attempt in (1, 2):
        client = OpenAI(base_url=conn["base_url"], api_key=conn["api_key"], max_retries=0, timeout=120)
        try:
            return client.chat.completions.create(
                model=conn["model"], messages=[{"role": "user", "content": prompt}]
            ).choices[0].message.content
        except APIConnectionError:
            if attempt == 2:
                raise
            conn = ensure_gpu()   # stopped while idle: restart it, then retry

Retry only on connection errors. A 400 or 401 comes from a live server. For TypeScript, examples/typescript-client/src/gpu.ts is one file with no dependencies that does all of this.

Short jobs: wrap them

$ gpu run python score.py --input data.jsonl   # heartbeats keep the machine up while it runs
$ gpu lock --for 3h && ./batch.sh; gpu unlock     # a batch with quiet stretches longer than idle_timeout

gpu run sets OPENAI_BASE_URL, OPENAI_API_KEY, JAMBU_MODEL_ID, JAMBU_GPU_ENDPOINT and the instance identity, so any OpenAI SDK picks the endpoint up by itself.

12Serving models

gpu init --model <repo> reads the Hugging Face model card and picks the engine and the OpenAI routes. Any OpenAI client works unchanged.

GET /v1/models lists the model, its task and its routes. Changing the model keeps the machine: edit model.id and run gpu setup, which redeploys on the same GPU. Use models as their licenses and your purpose allow; open models with their own policies are for legitimate work you own, such as research, evaluation, red-teaming your own systems and creative work.

The custom server contract

1. Listen on 0.0.0.0:$PORT.
2. Answer GET /health with 503 until the model is loaded, then 200.
3. Require Authorization: Bearer $JAMBU_API_KEY on every route except /health and /metrics.
4. Serve /metrics with jambu_requests_running (gauge) and jambu_requests_total (counter). Without them requests don't count as activity, and the machine stops after idle_timeout even while in use.
5. Use the OpenAI request and response shapes where one exists.
6. Exit non-zero when loading fails, after printing why. Never print secrets.

13Running jobs

A job is the agent's own code on the GPU: training, a kernel benchmark, a batch over a dataset. gpu job run packs it, runs it, streams the log and brings the outputs back.

run: python train.py --epochs 3
setup: pip install -r requirements.txt
env:
  EPOCHS: "3"
timeout: 2h
exclude: [data/raw, "*.ckpt"]
label: lora-qwen-7b

$ gpu init --jobs --min-vram-gb 24
$ gpu job run
$ gpu job run -- python test_kernel.py
$ gpu job status     # state, exit, cost_usd
$ gpu job logs -f
$ gpu job pull       # works mid-run
$ gpu job cancel

Job states: running, succeeded, failed, timeout (exit 124), cancelled (exit 130), lost (the machine restarted under it). The provider credential the watchdog uses is removed from the job's environment.

14Contract: CLI output

Pass --json before the command name. stdout then carries exactly one JSON document; errors go to stderr as one line, error: <message>, with a non-zero exit code.

{
  "instance_id": "27581234",
  "state": "ready",
  "endpoint": "http://203.0.113.7:41022",
  "base_url": "http://203.0.113.7:41022/v1",
  "api_key": "k6V0…  (43 characters, per machine)",
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "hourly_cost_usd": 0.196
}

The key is per machine: it survives stops and restarts and changes only when a new machine is rented. Take base_url, api_key and model from the latest setup instead of caching them across restarts. gpu env without --json prints shell exports.

15Contract: exit codes

16Contract: MCP tools

Each tool runs the CLI with --json in the project directory, so errors carry the same exit_code, plus meaning and next for the agent.

17Contract: the watchdog API

HTTP on port 8777 of the machine. Every route except /health needs the header X-Jambu-Token, compared in constant time. The token lives in .gpu/state.json and is never printed. The CLI is the usual caller; you rarely need these directly.

18Contract: provider adapters

Vast.ai ships built in. Any other provider is one Python class that implements providers/base.py. The core, the watchdog, the CLI and the MCP tools stay the same.

$ gpu providers new runpod --display-name "RunPod"   # writes gpu_providers/runpod.py
$ gpu init --provider runpod --model Qwen/Qwen2.5-0.5B-Instruct
$ gpu login --provider runpod
$ gpu --json validate && gpu --json offers           # real GPUs, real prices, nothing rented

An adapter can live in the project (provider.plugin: ./gpu_providers/acme.py:AcmeProvider), in an installed package (entry point group gpu4agents.providers), or in this repository. Errors are the normalized ones from core/errors.py, so exit codes are the same on every provider.

19Configuration

gpu.yaml is the desired state. gpu reads only its gpu_runtime: key. Unknown keys are a hard error, so a typo in idle_timeout can't silently switch off the protection.

gpu_runtime:
  version: 1
  model: {id: stabilityai/sd-turbo}
  compute:
    gpu: {min_vram_gb: 12}
  budget:
    max_hourly_cost_usd: 1.00
    max_session_cost_usd: 5.00
  lifecycle:
    idle_timeout: 15m
    max_lifetime: 6h
    stopped_ttl: 24h

At least one of idle_timeout and max_lifetime must be set, and idle_timeout may not exceed max_lifetime. Override the price cap once with gpu setup --allow-cost-override. Every field: CONFIG.md.

20Security

• The model endpoint needs a key. Unless runtime.api_key is set, setup generates one per machine (43 random URL-safe characters). /health and /metrics stay open; they reveal no data.
• The endpoint is plain HTTP on a public IP and port, so the key and prompts travel unencrypted. For sensitive data, use a private network or a tunnel, or put a TLS proxy in front.
• The watchdog holds a provider key on the machine (/etc/jambu/watchdog.json, mode 600) so it can stop itself. Use a dedicated, restricted key. Opting out (lifecycle.watchdog.enabled: false) leaves shutdown to gpu guard, and validate warns.
• The key never goes through the chat. gpu login takes it in a hidden prompt in your terminal, and the skill tells the agent never to ask for it, read it or print it.
• A shell alias can shadow gpu. With Oh My Zsh's gpu = git push upstream, an agent typing gpu setup in a repository would push. The installer removes the alias for new shells, and the skill has the agent check type gpu first and use command gpu. MCP tools don't go through a shell.
• Account hygiene. gpu ls and gpu sweep only see machines labeled jambu-.

21Full reference

The pages agents read are served from this site as Markdown. The long-form references live in the repository.