I run my own VPS with dozens of containers and I like it that way. I like knowing where things live, I like docker compose up, I like not discovering a surprise invoice. So when the conversation turned into "we need to run our own model", my instinct was the usual one: rent a box with a GPU and put it there.
That's what we did. And it was expensive in the dumbest possible way.
The problem wasn't the GPU price
Our load was a classic Brazilian B2B shape: a morning spike, a late-afternoon spike, silence otherwise. Documents coming in for extraction, a new batch of embeddings, the odd classification. Adding it up honestly, about three hours of real work per day.
The machine stayed on for 24. That's 87% idle and everyone knows how to do that math.
What nobody mentions is the second part: because the machine was expensive, it became precious. Nobody wanted to try a bigger model because "we'd have to swap the instance". Nobody wanted to run a batch reprocess because it would compete with the production queue. An experiment that should have cost forty minutes of curiosity cost a meeting instead.
The money was annoying. The not-touching-anything was what hurt.
What Modal changes
The core idea is simple: you write a Python function, declare which GPU it needs, and it spins up when someone calls it and dies when nobody does. No cluster to maintain, no idle instance.
That's the whole file. No separate Dockerfile, no Kubernetes manifest, no registry to push to. The image is declared next to the function, and deploying is one command.
Three things actually convinced me:
Scale to zero is real. No requests, no GPU charge. For bursty load that isn't an optimization, it's a change of category.
Parallelism is free. extract.map(list_of_pdfs) spins up dozens of containers at once. Reprocessing six thousand documents went from a weekend project to a lunch break.
The environment is code. The .pip_install sits right next to the function. The "works on my machine, breaks on the server" dance mostly disappeared, because there is no "server" with its own state to drift.
What hurts
Cold starts are real. A fresh container with a large model to load takes anywhere from a few seconds to considerably more, depending on weight size. For an async job that's irrelevant. For a synchronous endpoint with a human waiting, it's the whole problem.
You can tame it three ways, and we use all three:
- Memory snapshots, which freeze state after the model has loaded and restore far faster than loading again.
- Keeping a container warm through the known peak window, accepting idle cost from 8 to 11am because latency matters there.
- Pushing whatever can be async to async. A lot of what looked synchronous wasn't; it was just badly designed UI. An incoming document can become a notification instead of a spinner.
Second honest point: it's lock-in. The decorator is theirs. Migrating later means rewriting the orchestration layer — not the pipeline itself, which is ordinary Python, but the shell around it. I accepted that knowingly, because the alternative was maintaining Kubernetes with a GPU node pool, and I've done that enough times to know the price.
Third: per-second GPU cost isn't cheap compared to renting raw. If your load is constant, 24/7, Modal will likely cost more than a dedicated box. The savings come from the usage pattern, not the price sheet. Run the numbers against your real traffic profile before believing any blog post, including this one.
How it split
I didn't migrate everything, and I don't think anyone should.
Still on the VPS: Postgres, web apps, business jobs, everything light that stays on. That's cheap and predictable where it is, and moving it would trade calm for nothing.
Moved to Modal: anything needing a GPU, anything in large batches, and experiments. Especially experiments.
That last part was the surprise. When trying a different model costs "write another function and call it", you actually try. We swapped the extraction model twice in two months, not because the previous one was broken, but because comparison got cheap enough to do properly, with real data, instead of debating it in a meeting.
If I were starting over
I'd start with the most boring job you have — the reprocess nobody runs because it disrupts production. It's the smallest possible surface, no user is waiting, and it shows you the scaling behavior on the very first run.
And I wouldn't migrate a synchronous endpoint before measuring cold start with your model. The only number that matters is yours; any blog benchmark, this post included, is measuring something else.