NVIDIA open-sourced an agent stack.Here's what's free — and the one layer I had to build myself.

This page covers two of those in depth — the NeMo Agent Toolkit and OpenShell — because those are the two I built against. It is a working write-up, not a pitch: what each one does, where the seams are, what it does not do, the layer I built into that gap, how it was tested, and the Google Cloud specifics that cost me time so they do not have to cost you any.

Two commits pin everything here. NVIDIA/NeMo-Agent-Toolkit at 013255a4 and NVIDIA/OpenShell at ddc8bba9, both read on 11 September 2026. Line numbers move when upstream edits a file, so re-pin before quoting any of this out loud. I do.

1What's actually free, and where the free tier stops

The five things at the top of this page are genuinely free in the sense that matters: you can start today, with no sales conversation and, for four of the five, no account. What differs is how far you get before something stops you.

The API key trap, stated plainly

An nvapi- key minted from build.nvidia.com, the one labeled for API testing use, authenticates against the models list and is forbidden for inference. GET /v1/models returns HTTP 200 with 81 models on it. POST /v1/chat/completions returns 403 on every live model I tried. The NVCF and NGC user endpoints return 403 as well, which is the useful signal: the key carries no org entitlement at all, rather than a particular model being gated.

This costs a day if you assume the 200 means what a 200 usually means. It does not. A models list that answers is not an inference tier that answers.

Second trap, in the same family. A large block of the published catalog is simply gone. Llama 3.1 8B, Llama 3.3 70B and Nemotron Super 49B all return 410 Gone with an end-of-life date of 2026-08-26. Any tutorial, blog post or sample config naming those models is dead on arrival, and there are a lot of them. Treat the live models list as the only reliable source of a callable model id, and do not trust a model name you read in prose — including this prose, by the time you read it.

What the toolkit gives you to build on

The extension surface is real and it is documented. There are exactly 28 registration decorators in packages/nvidia_nat_core/src/nat/cli/register_workflow.py — cite the file rather than the tree, because the same pattern across all of packages/ returns 39 once you catch registry methods, two test helpers and some adapter registrations. Saying "28" while pointing at the tree produces 39 in front of the person checking.

The seam I used is @register_middleware. Everything the plugin needs is exported from nat/plugin_api/__init__.py — Builder, DynamicFunctionMiddleware, DynamicMiddlewareConfig, InvocationContext, InvocationAction, FunctionMiddlewareContext, register_middleware, register_function and FunctionBaseConfig. That file describes its own __all__ as a public contract to keep exact, and a toolkit test pins it. A plugin that stays inside that surface is one NVIDIA has committed to not breaking, which is worth more than any amount of cleverness reaching past it.

Two things to get right that cost me a test each. The entry point group is nat.plugins, not nat.components — nat.components still loads, and is documented as compatibility-only for new third-party packages. And the registry accessor is GlobalTypeRegistry.get().get_middleware(config_type): singular, and it takes the config class, not a name. There is no get_registered_middlewares(); guessing that it existed cost one failing test.

The one that would have shipped silently

This is the finding I would most want someone else to have before they start.

DynamicMiddlewareConfig advertises auto-discovery flags that intercept whole classes of component. They do not reach the function declared under workflow:. The discovery routine returns early unless a flag or an explicit list is set, and then iterates the builder's function mapping — which holds the functions: section only. The root workflow function is not in that mapping.

The failure mode is the dangerous one. With register_workflow_functions: true and no explicit attachment:

• nat info components lists the plugin
• nat run exits 0 and prints the correct answer
• no record is produced at all

Every check short of "is there evidence afterwards?" passes. A governance component that silently observes nothing is worse than no governance component, because it produces confidence instead of coverage. The fix is two-part: attach explicitly with middleware: ["<name>"] in the config, and have the middleware log a warning naming this behavior whenever the flag is set. The end-to-end test asserts on the journal file rather than the exit code, because the exit code was never the thing that was wrong.

That is a candidate upstream issue — either the flag should cover the workflow entry point, or the documentation should say that it does not.

2Three questions, and who answers which

The most important thing I found in OpenShell is not in any summary of it, and it reframed this whole project: OpenShell ships a formal policy prover. crates/openshell-prover encodes sandbox policies, binary capabilities and credential scopes as Z3 SMT constraints and checks reachability queries to detect exfiltration paths and write-bypass violations. It produces four categorical findings — link_local_reach, l7_bypass_credentialed, credential_reach_expansion, capability_expansion — with no severity grade, and any finding blocks auto-approval. The default approval mode at both scopes is manual, so every policy proposal lands pending a human.

So "we verify policy" is a claim NVIDIA can answer with their own prover, and a principal engineer will. The distinction that survives that conversation is about object and time — three genuinely different questions, asked at three different moments.

What "bound" means, and why a permit is not a token

A permit carries binding_request_hash: the digest of the canonical form of the actual request. A permit for "create an issue" would authorize any issue. This one authorizes the bytes it was issued against and nothing else.

That is the one check in the gate that cannot be satisfied by metadata agreeing with itself. The negative test changes one word of one argument — same tool, same server, same unexpired permit, same valid signature — and the call is refused.

Time-to-live is in seconds, not minutes. The rule field is constrained gt=0, le=300 and defaults to 30. A permit that lives for minutes is a window, and an attacker who steals one wants a window.

Attenuation is about fifteen lines, and nothing standard enforces it

When I surveyed the identity and delegation standards in April 2026, I could not find one that enforces monotonic privilege reduction across delegation hops. Each hop's scope must be a subset of the hop before it, and the tool must appear in the final hop's scope. That is fifteen lines of code.

Two design choices in it matter more than the code. It raises rather than returning a flag, so a caller cannot forget to check it. And it runs at issuance rather than at the gate, so a widening chain never becomes a permit that somebody has to notice was wrong later. There is no permit to steal, replay or misread, because one was never made.

Fail-closed, proven by a spy rather than by reading the code

The gate has nine refusal paths and no path that allows without a complete, verified, unexpired, correctly-bound permit. There is deliberately no configuration flag that creates one.

The assertion repeated in every negative test is spy.calls == []. That is the whole discipline in one line: a gate that returns "denied" while the request still reaches the server is not a gate, and the return value cannot tell you which one you have. Only a spy on the forward function can.

Two of the nine checks earn their place beyond the obvious list:

• A valid signature by the wrong identity is refused. Agent planner-1 signs a permit whose subject is executor-7. The signature verifies perfectly against planner-1's registered public key. A naive check passes it.
• The registry bounds the policy. A tool outside an agent's registered tool list is refused before policy is consulted, so policy can narrow what the registry granted and can never widen it.

Refusals come back as ordinary tool errors carrying the reason verbatim, not as crashes and not as a bare "denied". An agent told only that it was denied can do nothing but retry, which is the worst available response to a policy decision.

Denials are written down, and that is the point

Every decision is journaled, allow and deny alike. Writing the denials is not thoroughness — it is the load-bearing part. A record containing only successes cannot distinguish "nothing was refused" from "nothing was checked."

3The record you can't quietly change

Start with what OpenShell does here, because it is more than people assume and I do not want to be caught overstating the gap.

OpenShell signs policy. The signature is first-class metadata in tree — policy-signature and policy-signature-kid annotations, with an example governance interceptor that vends signed provider profiles and denies unsigned, stale or modified policies for every caller. Their canonicalization is worth reading before you write your own: they define a hash algorithm with a versioned prefix, a hand-written canonical JSON writer and framed hashing. They hit the same problem RFC 8785 solves, and solved it their own way for protojson.

OpenShell also emits audit, properly: OCSF v1.8.0, with event types, builders and JSONL formatters in tree, covering network and HTTP activity, SSH activity, process activity, configuration state change and detection finding. The string ocsf appears about 1,480 times in the repository. Their own architecture document says where it goes: OCSF JSONL output may be shipped to external systems.

Now the gap, and I state it as four searches so it can be checked rather than believed. A repository-wide search of every .rs and .md file for merkle, hash chain / hash-chain, cose_sign, and non-repudiat returns zero hits. Not one. The 139 signature and signing_key hits under crates/ are all JWT, mTLS, PKI and TLS — real cryptography doing a different job.

The same shape holds in NVIDIA's Secure Agent Workspace reference design, which asks for signed policies, OCSF audit emission from all three security layers, and a per-engagement delegation record. It signs policies and it collects audit. It nowhere claims a signed, hash-chained or non-repudiable record of the actions themselves. Two artifacts agreeing is a house position, not an omission.

What a closure record is, and the one mistake that would void it

The allowance is journaled before the call goes out and the closure after it comes back, as two entries. A single entry written afterwards would lose every call that crashed mid-flight, and an allowance with no closure following it is exactly the shape a reader needs to see when that happens. A dispatch that raises still closes, with outcome="error", before the exception propagates.

Three tampering moves, and the two that a hash chain does not catch

It is easy to say "hash-chained, tamper-evident" and stop. The useful part is that each layer alone is weaker than it looks.

A rewritten log has no internal inconsistency at all. A truncated one is a valid chain that is simply shorter. Only a commitment made at a moment the attacker cannot revisit, signed with a key they do not hold, disagrees. That is the checkpoint: tree size, head, and a Merkle root over every line, signed by a registered agent key in Cloud KMS — and a checkpoint claiming to cover twelve entries when nine exist is itself proof that three were removed.

Which is why the verifier reports an unanchored journal as UNANCHORED, not PASS, and why journal verify exits non-zero on one unless you explicitly pass --allow-unanchored. A tool that says PASS for a record nothing external vouches for is teaching its operator the wrong lesson every time they run it.

GENESIS is a fixed sentinel rather than null for the same family of reasons: the first entry commits to being first, so entry 5 cannot present itself as the head after 1 through 4 are deleted.

Two standards used verbatim, and why that was the whole point

RFC 8785 (JCS) for canonicalization and RFC 6962 for the Merkle tree. Neither was invented here and that is deliberate: a construction we invented would have to argue it got the hard parts right, and a reviewer would be correct not to believe it.

The reason JCS rather than json.dumps(sort_keys=True) is not pedantry, and the disagreement is reachable. Numbers: Python renders 1.0 as 1.0 and 1e-7 as 1e-07; ECMAScript's Number::toString, which JCS adopts by reference, renders them 1 and 1e-7. Key order: Python sorts by Unicode code point, JCS sorts by UTF-16 code unit. U+1F600 is above U+FB33 by code point and below it by code unit, because its first UTF-16 code unit is the surrogate D83D. Two implementations that both look sorted produce different bytes, and therefore different hashes, for the same object.

There is a test whose only job is to assert that the naive implementation actually disagrees on that input. Without it, the test that checks correct UTF-16 ordering would pass for both the right and the wrong implementation and would prove nothing.

Integers past Number.MAX_SAFE_INTEGER are refused, not rounded. A hash over a silently rounded value is a hash over something the caller did not write, which for an evidence system is the worst failure available: silent, and confidently wrong.

RFC 6962 earns its place through its domain separators — the 0x00 and 0x01 prefix bytes that prevent a leaf being presented as an interior node. Verified against the Certificate Transparency project's own root vectors for tree sizes 0, 1, 2, 3 and 8.

4How it was tested, and the bugs that only live tests find

Three rules. All three came from being wrong first, which is the only way I know of to arrive at a testing rule you actually keep.

Assert on the artifact, never on the exit code of the thing that was supposed to produce it

The Phase 1 middleware was configured, listed by nat info components, and recorded absolutely nothing while nat run exited zero with the correct output. Every available signal said the run succeeded. The only check that catches that is "is there evidence afterwards?"

A test double must reproduce the external system's ordering at its precision

Forty-three assertions passed against an in-memory reader while the two worst bugs in the anchoring worker sat in the gap between the datastore's real ordering and the ordering my pure code computed. Rewriting the double to order at full precision turned both bugs into failing assertions immediately. Details in §5 — that one is Google Cloud specific and it is the most expensive thing on this page.

Where a contract is owned by someone else, at least one test must talk to them

Thirty-five tests were green while the Cloud KMS integration could not create a single key. A mock would not have caught it: a mocked create_key would have encoded the same false assumption the implementation did, and the two would have agreed with each other indefinitely. The only thing that disagreed was Google.

The Merkle bug that hides behind powers of two

The inclusion-proof verifier was written wrong the first time, and the shape of the wrongness recurs often enough to be worth naming.

An audit path is emitted leaf-first. A verifier that walks the tree root-first consumes the siblings in the wrong order — and still verifies every proof for every tree whose size is a power of two. A test suite using 2, 4 and 8 leaves passes it completely, and those are exactly the sizes a person reaches for when writing a tree test.

Caught by parameterizing over 1, 2, 3, 4, 5, 6, 7, 8, 13 and 37. Fixed by implementing RFC 6962 §2.1.1 step for step rather than reasoning about tree shape independently and arriving somewhere that looked equivalent.

A verifier must not crash on hostile input

This is the bug I am most glad a test found.

The checkpoint checker called the signing code to verify a signature. The signing code deliberately raises when the key id names no registered agent — which is correct for its own caller, because "did X sign this?" has no answer when there is no such X.

But the checkpoint checker is asking a different question, and that question does have an answer. A checkpoint signed by an identity the registry does not vouch for anchors nothing. No. Letting the exception propagate meant anyone could crash the verifier by handing it a checkpoint signed with a key they made themselves — a denial of service aimed squarely at the audit path, which is the one thing an attacker most wants unavailable. I had built them a switch.

check() now returns false and never raises.

It was found only by a test written as prose about what must not happen. The verification script I wrote by hand covered the three tampering strategies and the unsigned-checkpoint case. It did not think to sign a valid checkpoint with a stranger's key, because a script written to confirm that the good path works does not generate that case. Nine of the nineteen tests in that phase are named test_NEGATIVE_..., and that ratio is not an accident.

One more, which is about transcription rather than logic

Every file that lands in this project gets its content hash compared against the hash of the bytes that were actually tested. That check has caught seven transcription faults so far — dropped comment blocks, a missing section divider, and three escaping errors in the test file that is itself about escaping.

Two of those three would have passed the entire test suite. In Python an escape sequence and the character it denotes are the same string, so every assertion holds. What would have shipped is a file contradicting its own docstring and carrying the exact fragility the docstring exists to warn about — invisible to the tests, and visible only by comparing hashes. It costs one field in a tool result.

5The Google Cloud specifics

This section is the one I would have wanted before I started. Everything in it cost me time.

Cloud KMS: an asymmetric key has no primary

The key-creation path read created["primary"]["name"] and fell back to listing key versions when it was missing. Two things were wrong, and only a real API call could show either.

First, primary is never present on an ASYMMETRIC_SIGN key. Cloud KMS populates it for symmetric ENCRYPT_DECRYPT keys only. So the path written as a fallback is in fact the only path these keys ever take. Second, that path was broken: it passed the bare resource name where a URL was required. The structural tests passed. The software-backend tests passed. Thirty-five green, zero keys.

The correct move is to GET {cryptoKey.name}/cryptoKeyVersions and take the version from the list. And make the empty-version case raise rather than return a key URI naming nothing — a registry entry pointing at a key that cannot sign is worse than a failed registration.

How to prove "the private key cannot be exported" properly

The weak version of this test is to attempt an export and collect a 403. It is weak because a permission denial says this identity cannot do it today, and the fair reply is "so grant yourself the permission."

The strong version is that the operation is absent from the API. The complete method set on cryptoKeyVersions is eighteen methods: asymmetricDecrypt, asymmetricSign, create, decapsulate, delete, destroy, exportTrustedKeyWrappedCryptoKeyVersion, get, getPublicKey, importTrustedKeyWrappedCryptoKeyVersion, import, list, macSign, macVerify, patch, rawDecrypt, rawEncrypt, restore. There is no getPrivateKey and no exportPrivateKey; those strings occur zero times in the whole 381,189-byte discovery document.

Then do it holding roles/cloudkms.admin on the key ring containing the key. For that key you are as privileged as the API allows, and the private half still cannot be retrieved — not because you are refused, but because there is nothing to ask:

GET .../cryptoKeyVersions/1/privateKey   -> HTTP 404  (route does not exist)
GET .../cryptoKeyVersions/1:export       -> HTTP 400  (resource name does not match pattern)
GET .../cryptoKeyVersions/1/publicKey    -> HTTP 200  PEM

A 404 from the frontend and a pattern-match rejection. Not an authorization decision at all. An admin who cannot export is a much better witness than a stranger who was refused.

State the caveat before someone else finds it. exportTrustedKeyWrappedCryptoKeyVersion does exist. It returns key material wrapped to a trusted wrapping key, for moving keys between KMS instances, and it is not available for a key created the way these are. So the defensible claim is "no method returns private key material in the clear" — not "the key can never leave." There is a test asserting the caveat, so if that method is ever renamed or removed, the documentation is forced to be corrected.

That negative test runs against the live discovery document, so it keeps checking: it fails the day Google adds an export method. A test that only ever confirms today's answer stops being evidence the moment the world moves.

Cloud KMS billing shape, and the cleanup that is also a test

A cryptoKey can never be deleted. Only its versions can be destroyed. A live test that does not clean up therefore leaves a billable key version inside a permanent key on every single run, forever.

Have the fixture destroy every version it created. It costs nothing and it exercises the same code path that key revocation depends on, so the cleanup is itself coverage. Report teardown failures and swallow them — losing a real result to a janitorial error is the worse trade. And note that destruction is not immediate: the version moves to DESTROY_SCHEDULED with a destroy time roughly 30 days out, so assert that state rather than a deletion that never happens synchronously.

The same shape drives a design decision. One signature per decision is one KMS round trip on the hot path of every tool call — latency and cost both. A checkpoint covers every entry up to its tree size with a single signature, which is what Certificate Transparency does and for the same reason. The cost of that choice is real and gets reported rather than hidden: entries appended after the last checkpoint are chained but not anchored, and the verifier reports the anchored prefix and the unanchored tail separately instead of averaging them into one comfortable verdict.

Firestore: millisecond truncation is permanent, silent evidence loss

This is the most expensive bug on this page and it is three words long.

The reader used toDate().toISOString(), which truncates to milliseconds. Server timestamps carry microseconds. The ordering code then compared timestamps as strings, so the anchoring worker's order stopped agreeing with Firestore's paging order — a row correctly returned after the cursor was then discarded for sorting before it. Three documents in, one vanished, and page two anchored nothing.

Permanent, silent evidence loss, in an evidence system, from a formatter. I reproduced it before fixing it, which is the only way to know you fixed the thing you thought you did.

Two more Firestore facts worth having:

• Firestore orders by (seconds, nanoseconds, document id). If your test double orders by anything less precise, it will agree with your code and disagree with production.
• Firestore sorts null before every timestamp. A document whose timestamp field has not been populated yet does not sort at the end where you probably assumed it would.

OpenShell on Google Cloud: it is an install, not a build — and where it cannot run

OpenShell's enforcement is Landlock plus seccomp plus namespaces, driven through a compute driver (Docker, Podman, MicroVM or Kubernetes). That requirement is not negotiable and it rules out more environments than you would expect.

It will not run on a gVisor-backed sandbox: no /sys/kernel/security, an unreadable /proc/kallsyms, and typically no container runtime on the path. That is structural, not a missing package, and it is not worth retrying there. On a normal GCE instance it is an install from a released Debian or RPM package, not a Rust build — which is a very different afternoon.

The OpenShell policy trap: the rule is a pair

A network_policies block that lists endpoints but has no binaries key allows nothing — not even the hosts it explicitly names. The rule is a (binary, endpoint) pair, not an endpoint.

I had a policy naming a Google endpoint and got CONNECT tunnel failed, response 403 for that exact host. It looks precisely like a broken gateway. Adding binaries with the absolute path made the same endpoints work immediately while un-named hosts stayed 403.

And the binary path must be the resolved path. In the community base image python3 is not /usr/bin/python3 — it lives under the uv-managed CPython tree. Find it with readlink -f $(command -v python3) inside a throwaway sandbox before you write the policy.

One more, offered as a candidate upstream issue rather than a finding: openshell policy set on a live sandbox is refused if the file carries filesystem or landlock keys, because the process policy is applied at startup. The quickstart policy carries both, with an inline comment saying they are required for policy set. So the file as shipped cannot be applied by the command the README shows applying it. I would confirm that against a current build before filing it — it may be that only a difference from the startup policy is refused.

The result worth showing

This is my favorite output of the whole build, and it is one screen of transcript.

That is the proof the two planes are genuinely independent rather than two names for one check. The gate is transport-level and model-agnostic, so a NIM endpoint is just another destination behind a permit; OpenShell is deciding something else entirely, at the kernel, about a binary and a host. Neither one substitutes for the other, and you want the run where they disagree.

6What isn't done

Every one of these is written into the source it applies to, not just here. A limit you only mention on your website is a limit you are hoping nobody checks.

The correlation gap — the honest limit of the two-vantage-point claim

Reconciliation compares the in-process view with the on-the-wire view on agent_id and tool. That catches a call reaching the server with nothing in the agent's own accounting for it, and it catches a request that changed between authorization and the wire.

It does not match individual calls to individual records, because nothing yet carries a correlation id between a toolkit function invocation and the outbound request. Until permit_id is threaded through the middleware, this is a count-and-attribution check, not a per-call join. That is the next piece of engineering and it is the one I would want reviewed hardest.

Unverifiable is not a pass

An argument with no canonical form gets an unrepresentable: marker rather than a hash. Reconciliation reports that as a divergence, not a match. It would have been easy to let it pass quietly and easy to never notice that it had.

The registry is not chained

The agent registry — who exists, which key is theirs, which tools they may reach — is the thing the whole gate defers to, and it is currently an ordinary append-only file with the same weakness the journal had before it was chained. The evidence plane should cover the thing that bounds it. It does not yet.

A packaging smell I would rather name than be shown

Nineteen modules make up this work and only three of them touch NVIDIA — the middleware, its config, and the registration hook. The other sixteen are a standalone evidence plane: canonicalization, Merkle trees, checkpoints, permits, the gate, the verifier, plus a second independent implementation of the chain and verifier in TypeScript, written specifically so that one language's bug cannot be the reason a proof checks out.

RFC 6962 code has no business living inside a directory named after somebody else's plugin namespace. The right end state is two packages: a dependency-light evidence core that knows nothing about agent frameworks, and a thin provider plugin that adapts it to the toolkit's middleware seam. That split is recorded as an architecture decision with the module boundary drawn, and it is deliberately deferred — a packaging refactor across a tree with 136 passing tests buys correctness nothing and risks a green tree. It is scheduled, not forgotten.

And the general disclaimer, which is not boilerplate

This is a reference implementation built to make an argument, not a product with an SLA. It has not been through an external security review. The threat model it addresses is an agent, or someone holding an agent's credentials, doing something outside what was authorized — and a record of that surviving contact with whoever would prefer it did not. It is not a defense against a compromised host, and nothing on this page should be read as claiming otherwise.

What's released

The plugin attaches through the documented nat.plugins entry point and the public nat.plugin_api surface only. It does not fork, patch or vendor any part of the toolkit, which is the whole discipline: a governance layer that requires you to modify the thing it governs is not a governance layer, it is a fork with opinions.

A minimal workflow attaches it in four lines:

middleware:
  paracoding_observer:
    _type: paracoding
    journal_path: ./journal.jsonl

workflow:
  _type: your_workflow
  middleware: ["paracoding_observer"]   # attach explicitly — see §1

The plugin and the independent verifier are Apache-2.0 and are being prepared for public release alongside the rest of paracoding-oss. This page goes up first because the measurements are useful to somebody today whether or not they ever run a line of my code — the four searches, the (binary, endpoint) rule, the primary field, the millisecond truncation and the NIM key behavior are all true regardless of what I built.

Written by Scott McDonald. The project is paracoding.ai. The reason is curiosity, tokens to burn, and a drive to push security layers deeper than the ones I keep being handed — which is what this turned out to be. Corrections are welcome and get made in place, with the correction marked.

Independent work. Not affiliated with, sponsored by or endorsed by NVIDIA or Google.