I wanted the problems

There is a whole category of tool now — Coolify, Dokploy, Kamal, Dokku — whose entire pitch is that you shouldn’t have to do what I spent last week doing. Point one at a fresh VPS, connect a repo, and you get containers, TLS, a tunnel, a dashboard and git-push deploys in about the time it takes to read their landing page.

I didn’t use any of them. Not because they’re bad — they’re genuinely good, most people should use one, and I’ll probably put one on this box eventually.

I skipped them because I wanted to hit the problems myself.

That sounds like a slightly precious thing to say, so let me be specific about what it bought and what it cost, because I don’t think “do it the hard way” is good advice on its own. It was good advice this time, for a reason I could name in advance.

The difficulty doesn’t disappear, it relocates

A PaaS layer doesn’t remove the tunnel, the container networking, the certificate story or the firewall interaction. All of that still happens. It happens somewhere you’re not looking, configured by someone who already understood it.

Which is a real service — right up until the first time something breaks in the part it handled for you. Then you’re debugging a system whose failure modes you’ve never seen, through an abstraction that was designed to stop you needing to see them.

I’ve been on the wrong end of that trade before, and the tell is always the same: everything is green and nothing works, and you have no model to reason with. So this time I wanted to build the model first, on a low-stakes box, while nobody was depending on it.

The deal I made with myself was roughly: spend the afternoon now, on my own schedule, or spend it later at the worst possible moment.

What I was actually deploying

A Go API and two static frontends — three hostnames, one Cloudflare tunnel, Postgres hosted elsewhere. The specifics don’t matter much except for one thing: three services, not one. A single-container deployment would have hidden at least two of the problems below. The moment there’s more than one thing behind the tunnel, the interesting questions start.

By the end, the box had no inbound port open except SSH. Not 443. Not 80. Nothing. cloudflared runs as a container, dials out to Cloudflare, and holds that connection open; requests come back down it. Traffic only ever flows one way at connection time, which is the same trick Tailscale uses and the reason I couldn’t block outbound when I hardened this box.

That single property — nothing connects in — turns out to invalidate a surprising number of things I thought I knew.

The firewall stopped being the thing protecting me

I have ufw default deny incoming, enabled and verified. It is not what keeps my API off the internet.

Docker writes iptables rules directly, underneath where ufw operates. Publish a port from a container and it’s reachable from the public internet regardless of what ufw says. This is well documented and widely known as a gotcha — but knowing it as trivia is different from having it be load-bearing in your design.

What actually keeps my API private is a thing that isn’t there:

api:
  image: ghcr.io/me/api:latest
  # no ports: — cloudflared reaches this over the compose network

No ports: line anywhere in the file. That absence does the work. If I added ports: "9000:9000" tomorrow for a debugging session and forgot to remove it, ufw would not save me, and ufw status would keep telling me exactly what I wanted to hear.

So the question I review changed shape. It used to be is the firewall right? Now it’s does anything publish a port? Those feel like the same question and they are not, and I would not have noticed the difference if a tool had written the compose file for me.

localhost became the wrong answer

Every tunnel tutorial routes to http://localhost:8080, because most people run the tunnel agent on the host, where that’s true.

Mine runs in a container. Inside it, localhost is itself — nothing listening. The services are sibling containers on a shared Docker network, addressed by service name:

The part that made me stop and think: two of those containers both listen on port 80 and never collide. Separate networks, separate namespaces. Port conflicts are a property of publishing to a host, and I’m not publishing anything.

I’d spent years treating “what port is it on?” as a real question. Behind a tunnel it mostly isn’t one, and that reframing is the kind of thing you only get by having to answer it yourself.

The header I’m told never to trust became the only truth

Standard advice, correct nearly everywhere: never trust X-Forwarded-For. Anyone can send it. Build a rate limiter on it and I’ll change it every request and walk through.

Behind a tunnel, CF-Connecting-IP is trustworthy — and specifically because of the same property that made everything else strange. There is no route to my origin except through Cloudflare. Nobody can reach the container to forge a header, because the container has no address they can reach.

The thing doing the work isn’t Cloudflare being honest. It’s the absence of an alternative path.

That distinction matters enough to write down at the call site, because it means the trust is conditional on a topology decision made in a completely different file:

// Trustworthy ONLY because no inbound port is open — there is no path to
// this process except through Cloudflare. If anything ever publishes a
// port on this host, this line becomes a spoofing vector.
ip := r.Header.Get("CF-Connecting-IP")

A comment isn’t enforcement. But the realistic failure is someone adding a ports: line two years from now for an unrelated reason, and that comment is the only thing in the repo that would tell them what they just broke.

Everything healthy, everything 502

This is the one I’d have paid for. It’s also the moment the whole approach justified itself.

I brought the stack up. Four containers, all Up. The Cloudflare dashboard said Healthy, one active replica. All three routes present and correct. Then:

curl https://api.example.com/health   # 502
curl https://app.example.com          # 502
curl https://forms.example.com        # 502

Every hostname dead. Every component reporting fine.

The cause: earlier, to get the tunnel to register at all, I’d run cloudflared as a throwaway container just to prove the token worked. That container was still connected. Cloudflare saw two replicas of one tunnel and load-balanced between them — which is a real feature, meant for running a tunnel across several machines.

But my two replicas weren’t equivalent. One was inside the compose project and could resolve web, forms and api. The other sat on the default bridge network, where those names mean nothing at all. Requests landing on it had nowhere to go.

A correct token is not a correct network. The stray container was authenticated, healthy, and correctly registered with the right tunnel. It simply couldn’t see anything.

What makes this the centre of the post isn’t the mistake — it’s that no dashboard I had was capable of showing it. Cloudflare knew there were two replicas and considered that healthy, because from its side it was. Docker knew both containers were running. The routes were right. The token was right. Every green light was accurate, and the site was completely down.

The fix was docker ps, notice two cloudflared containers, stop the one that wasn’t mine.

And this is exactly the shape of failure a PaaS leaves you least equipped for. Not because it would have caused it — it would have prevented it, by managing the agent for me. But the class of bug is the same one you eventually hit anyway: every observable signal accurate, the conclusion still wrong. Debugging that requires a model of what the pieces are for, and there is no way to acquire that model from a UI that has been hiding them.

The rule I kept: one tunnel token, one process. A second app gets a second tunnel — they’re free — rather than a second replica of the first. Reuse the token across networks and you’ve built a load balancer between machines that can’t serve the same routes.

I couldn’t get a certificate

I put Caddy in front of the static files for the reason everyone does: automatic HTTPS.

It cannot get a certificate here. ACME validation requires Let’s Encrypt to reach in — HTTP-01 wants port 80, TLS-ALPN-01 wants 443 — and I have deliberately made that impossible. Caddy will try, fail, retry forever, and fill the logs with certificate errors while the site works perfectly.

So the one feature I chose it for had to be switched off:

{
    auto_https off
}

TLS terminates at Cloudflare’s edge. Caddy is, in this deployment, a static file server with a good config format.

There’s a DNS-01 escape hatch — prove ownership with a TXT record rather than an inbound connection — but it wants a custom Caddy build and a DNS API token on the box, in order to encrypt a hop between two containers on one machine that never touches a network interface. I’ll take the plain HTTP.

What the hard way actually cost

I don’t want to pretend every hour was educational, because it wasn’t, and that’s the honest half of this.

Some of it was pure friction. I picked the wrong domain from a dropdown and didn’t notice for ten minutes. I pasted a database URL with the wrong variable name and quotes Neon had helpfully added. I hit a Permission denied (publickey) that turned out to be an SSH config alias, not anything to do with deployment at all. None of that taught me anything. It was just tax.

And the real cost is that the thing works now and I still can’t hand it to anyone. A Coolify deployment comes with a UI someone else can operate. Mine comes with a runbook I wrote, which is worth exactly as much as my memory of writing it.

The split, roughly: maybe half the afternoon went into the five things above, which I’d have paid double for. The other half went into typos and dropdowns, which I’d have happily skipped and which no amount of framing makes valuable.

What I’d still hand over

I might put Coolify on this box eventually, and the difference is that I’d now know what it was taking over — that it manages a tunnel agent whose network membership is the thing that matters, that its firewall rules are the second line of defence and not the first, that its certificate handling is doing something different from what it does on a normal server.

Two things are still missing, and they’re the ones I’d want a tool for least, because they need judgement:

No rate limiting. Four unauthenticated routes, and the one that matters isn’t the one I first assumed. I’d flagged the admin endpoint as the highest-value target; it isn’t — its token is 256 bits and the middleware rejects a bad one before doing any real work. The expensive route is login, because bcrypt burns a quarter-second of CPU by design, on an unauthenticated path, on one small VPS. There, cost is the attack surface, not secrecy.

The tunnel’s routing lives in a dashboard, not in git. I chose the token-based setup for a simpler compose file, and the trade is that those three hostname mappings exist only in Cloudflare’s UI. I keep a table of them in my runbook, which is a mitigation and not a fix. At three hostnames it’s fine. It won’t stay fine.

What I actually got

Not a deployed app. I could have had that before lunch.

What I got is that when the site went down and every dashboard I owned insisted it was fine, I had somewhere to start. I knew a tunnel agent’s usefulness is a property of the network it’s on, not of the token it holds, because I’d wired that up myself an hour earlier and could picture it.

That’s the whole return, and it only shows up once, at the worst moment, in a form you can’t schedule. Which is why I don’t think “always do it the hard way” is good advice — most of the time you should use the tool. But at least once, on a box nobody depends on, with an afternoon you can afford to lose:

Go and get the problems while they’re cheap.