Designing an offline root CA for small operations

Keeping the root CA offline is advice everyone repeats and almost no one operationalizes. The phrase fits in a tweet. The procedure behind it does not. What follows is my attempt to explain what “offline root” actually buys you when you’re a single engineer with no HSM budget, no ceremony room, and a real requirement to issue VPN certificates on Tuesday.

I think the field over-indexes on imitating high-assurance PKI ceremonies badly, when the actual goal is much narrower: identify the one key whose compromise is catastrophic, reduce how often it’s exposed, and build a recovery path you could actually execute in a bad week. Everything else is theater.

What the root is actually protecting

A leaf certificate compromise burns one identity. An intermediate compromise burns everything that intermediate was ever allowed to sign. A root compromise burns the concept of trust itself for that hierarchy: there’s no longer a principled way to tell a legitimate certificate from an attacker-issued one.

That asymmetry is the whole design. The root’s job is narrow: mint and, rarely, replace the intermediate. It should never touch routine issuance. In my world view, the frequency with which a key is present is itself an attack surface, independent of how well the key is protected in isolation.

A hierarchy that actually matches the threat model

For a small VPN PKI, the shape is simple:

Offline root CA → signs a constrained intermediate CA → the intermediate signs client and service certificates, enforces policy, handles revocation, generates CRLs.

The root sits outside the running platform entirely. The intermediate absorbs all the routine grind, and its constraints (path length, key usage, lifetime, SAN restrictions) are where the actual engineering happens. Bolting on a second certificate in the chain is trivial. Making that certificate genuinely less powerful than the one above it is the real work.

The root should never be reachable from the app

An admin web UI is a large, hungry attack surface (HTTP parsing, sessions, templates, third-party deps, all of it). Putting root key material anywhere behind that surface is a category error, and it’s the same error a web-app security checklist would flag on the client side: never trust a boundary you haven’t drawn on purpose.

The pattern I like: browser talks HTTPS to a control plane, the control plane talks to a certificate daemon over an AF_UNIX socket, and the daemon is the only thing that ever touches CA state. The root itself never appears in this path at all (it’s invoked only during intermediate lifecycle events, off to the side).

This doesn’t neutralize a compromised web app. It contains it. The attacker inherits whatever narrow command surface the daemon exposes, not an OpenSSL shell and a filesystem.

Why the Unix socket carries real weight

A local socket gets you a lot for free: no network listener, filesystem-level access control, a hard process boundary, a protocol surface small enough to actually reason about. The daemon should expose exactly the operations you need (sign a CSR, build a client bundle, revoke, regenerate a CRL, health check) and nothing resembling “run arbitrary OpenSSL command.”

That said, the socket is a boundary, not a magic trick. A world-writable socket, a shared service account, or a loosely validated request schema erases most of the benefit. The boundary is only as good as the daemon identity, socket permissions, request validation, and audit trail around it, and a hardening checklist would ask the same three questions it asks of any privileged local service: who can reach it, what can it be told to do, and is every call to it logged somewhere the operator actually reads.

The ceremony is a procedure, not a performance

A small team still needs a repeatable root-key procedure, written down before the root is ever generated: who’s present, what machine, what media, how integrity of the tooling is verified, how entropy is sourced, how the key gets encrypted, where backups live, which fingerprints get recorded, which intermediate constraints get applied.

This is worth treating as a compliance control narrative even if no auditor will ever see it: what the control is, who operates it, how often it runs, and what evidence proves it happened. Writing that narrative down before the root exists is what turns “we did a ceremony once” into something you could actually reconstruct or hand to a new hire. What I find most notable about ceremonies, other than how often they’re skipped, is that their value comes almost entirely from being rehearsed and reviewable, not from being elaborate. A short procedure your team can actually execute correctly beats an impressive one that’s never been run outside a doc.

Offline is not a synonym for air-gapped

A dedicated offline laptop, a live OS booted for the occasion, a machine with networking physically disabled, a controlled VM: any of these can work, and the right pick depends on the threat model, not on which sounds most serious.

The property that actually matters: no network access during root operations, tooling verified in advance, temp files identified and wiped, and the environment reproducible when the intermediate eventually needs renewal. A laptop that hasn’t been patched in three years is not secure just because the Wi-Fi is off. Software integrity doesn’t stop mattering once you unplug the ethernet cable, and a stale offline image carries the same kind of risk a dependency audit exists to catch on the online side: nobody looked at what’s actually installed in a long time.

Storage and recovery are the same problem, not two problems

One USB stick in a drawer is not a recovery plan, it’s a single point of failure with extra steps. You want multiple encrypted copies, on separate media, in separate physical locations, with a clear answer to: what happens when one device dies, who holds the decryption material, can any single person reconstruct the whole thing alone, has restoration actually been tested.

This is the part people get backwards. They optimize confidentiality to the point of unrecoverability, and then the intermediate expires or gets compromised and nobody can get to the root that’s supposed to save them. Availability and confidentiality have to be designed together or you’ve just built a very secure way to lose your own PKI. Treat “has anyone actually restored from this backup” the way a disaster-recovery control review would: an unrehearsed recovery path is a documented gap, not a completed control, no matter how good the encryption is.

Constrain the intermediate on purpose, not by default

The intermediate should be meaningfully narrower than the root: bounded certificate types, naming rules, validity windows, SAN policy, key usage, whether sub-CAs are even allowed. Its lifetime should be shorter than the root’s but long enough that renewal is a planned event, not an incident.

Signing an intermediate with broad, unconstrained capability because that’s the default CLI invocation is the single most common shortcut I’d flag here. It’s the PKI version of a cloud role with a wildcard action in its policy: convenient the day you write it, and the first thing an attack-path analysis would flag the day someone actually maps what that credential can reach. It defeats most of the point of having a hierarchy at all.

The online intermediate needs its own hardening

Because the intermediate is always up, it needs runtime discipline the root doesn’t: dedicated service account, no network-exposed CA API, Unix-socket-only access, strict schemas, policy enforced inside the daemon rather than trusted from the caller, tight key permissions, structured audit logs, rate limits, working revocation and CRL generation, backups kept separate from the web app. This is the same checklist a Linux hardening pass runs against any privileged daemon: non-root user, minimal capabilities, auditd watching the key files, nothing listening that doesn’t need to.

The control plane should be able to request a certificate. It should not get to decide the final policy. If the control plane asks for a client cert with some subject, the daemon independently validates the subject, the profile, the lifetime, the key type, before it signs anything. That independent check is what gives the boundary meaning even when the caller itself is compromised, and it’s the same “don’t trust the caller’s claim about its own privilege” logic that shows up anywhere one component is a confused deputy for another.

Revocation is issuance’s other half

A system that can issue but can’t reliably revoke is only half built. You need real answers to: how a cert gets selected for revocation, how fast the CRL regenerates, how the VPN service actually receives the update, what happens when publication fails, whether CRL expiry is monitored, whether an operator under pressure can find the right serial number.

A beautifully protected root does nothing for you if revocation is a process nobody can execute at 2am. Issuance, revocation, replacement, and distribution are one lifecycle, not four separate features you can ship independently, and CRL expiry is exactly the kind of thing that belongs on a monitoring baseline with an alert attached, not something you discover is broken during the incident that needed it.

Intermediate compromise and root compromise are different emergencies

Intermediate compromise is bad but bounded, and it maps cleanly onto a standard incident lifecycle: identify the compromise, contain by pulling trust and isolating the daemon, preserve logs and the daemon’s audit trail as evidence, eradicate by cutting a new intermediate off the offline root, recover by reissuing and pushing new CRLs, then run the lessons-learned pass. Painful, survivable, and the root never left its box.

Root compromise is a different category of event. Every relying system potentially needs a new trust anchor. You may no longer be able to distinguish legitimate certificates from attacker-issued ones anywhere in the hierarchy, which is closer to a full trust-anchor rebuild than an incident response playbook in the usual sense. This asymmetry, more than anything else, is the actual argument for keeping the root out of routine operation. It’s not about the root being “more secure,” it’s about what happens to your entire trust graph if it isn’t. Deciding in advance which of these two playbooks you’re in, and who has the authority to declare it, is worth writing down before you need it, the same way a SOC escalation matrix assigns severity and response time before the alert ever fires.

Cargo-culting is its own risk

A handful of patterns get copied from enterprise PKI without anyone asking whether they help a two-person operation: elaborate ceremonies that are never rehearsed, so nobody can restore the key when it matters; manual steps performed once every few years, forgotten by the time they’re needed; offline machines running stale, unverified tooling; a perfectly encrypted root protected by a password exactly one person remembers; every backup component stored in the same drawer; and treating an HSM as if it were the architecture rather than one control inside a much larger operating model.

An HSM protects a key. It doesn’t define policy, authorization, revocation, auditability, recovery, or trust-anchor distribution. Buying one doesn’t buy you the rest of the design, in the same way that buying an EDR agent doesn’t buy you a detection strategy. The control is only as good as the operating model wrapped around it.

What a defensible small-scale setup actually looks like

The root stays out of routine systems entirely. Issuance never requires it. A web app compromise never directly reaches CA key material. The intermediate enforces policy independently of whatever the caller claims. Revocation and CRLs are operational, not aspirational. Both root and intermediate have tested recovery paths. Losing one storage medium doesn’t lose the PKI. One compromised runtime credential doesn’t cascade into every trust layer. And the operators can actually explain and reproduce the procedure, including replacing the intermediate, without inventing the process live during an incident.

TL;DR

“Offline root CA” isn’t a checkbox, it’s an operational boundary, and the value comes from three things: reducing how often the most powerful key is present, minimizing which systems can reach it, and preserving a recovery path that still works when someone finally needs it in three years. For a small team, the right design isn’t the most ceremonial one you can build. It’s the smallest one that creates real separation, that you can actually test, and that will still make sense to whoever’s on call when the intermediate finally needs replacing.