Skip to content
Calango Lab — homeCalango Lab
08 What holds the rest up

The system is up. And when it goes down?

Servers, deployment, backups and monitoring: the part of the work nobody notices while it holds, and the only part that matters the day it stops.

checklist.md
Do you have backups?almost always yes
Has anyone ever restored one?almost always never
How long does coming back take?nobody knows the number
Who gets told when it goes down?the customer, by phone
Who has server access today?sometimes a former supplier

Five questions that tell you whether yours is standing

Almost every company says “yes” to the first and stalls on the second. That’s exactly where the difference lives: a backup nobody ever restored isn’t a backup, it’s hope — and the day you find out is always the worst possible day.

The other three are the same kind of question. They don’t ask what technology you run, they ask how long your company stays down, who finds out, and who can still get into the server. The last one is usually the most uncomfortable: in a good share of the cases that reach us, the person holding the key stopped working for the company years ago.

If you answered all five without hesitating, you probably don’t need this service — and it’s better to know that now than after a proposal.

Software nobody watches isn’t under control. It’s running on luck.

Resilience

Failing is inevitable. Being offline is a choice.

Disks fail, providers have incidents, an update goes out wrong. No configuration prevents that. What gets decided at design time is what happens next — whether the site goes down with it, or the traffic routes around it without anyone outside noticing.

Edge
ingresstls · cache · rate limit
Application
instance 01são paulodown
instance 02são paulodown
instance 03porto alegredown
Data
databaseprimary
replicareads · mirror

What the diagram is saying

The health check is what separates one outcome from the other. Without it, the edge keeps sending people to a dead instance and the visitor gets an error. With it, the instance drops out of rotation within seconds and traffic redistributes — the same detour you can watch happening up there, every so often.

And what it’s not promising

Notice that a few requests are lost at the moment of failure, in amber. That’s honest: failover isn’t magic, and anyone promising zero loss is selling something else. What you gain is the difference between a handful of failed requests for a few seconds and the whole store being offline for three hours.

The scope

What we take responsibility for

Four fronts usually sold by four different companies — and it’s in the seams between them that problems drop off everybody’s radar.

Servers, network and database

Sizing, configuring and hardening whatever holds the application up: a patched system, a firewall closed by default, key-based access instead of passwords, certificates that renew themselves, and a clean line between what is public and what never should be. It’s the work that shows up on no screen and accounts for most of the incidents that never happen.

Deployment pipeline

Build and release with no manual step, a staging environment that matches production, and a way back when something slips through. A deploy that’s frightening is a deploy that gets postponed, and software that’s not released isn’t fixed — the fix sits there ageing, waiting for a window that never comes.

Backups and restores

A copy routine, a copy outside the same provider, and — the part usually missing — a restore genuinely tested, with the time to come back measured and written into the proposal. Until someone has restored it, a backup is an assumption about a file nobody has opened.

Monitoring and cost

Logs, metrics and alerts, so a problem shows up on a dashboard before it shows up as a customer complaint. Plus a review of what the cloud is charging: a good share of the bill is usually machines oversized out of caution, and cutting that often pays for the work itself within the first six months.

How we come in

What hurts first, what is elegant later

Nobody needs elegant architecture before they have a backup that comes back. The order below follows the risk, not the portfolio — and it’s the same whether we built the infrastructure or inherited it from another supplier.

  1. Assessment

    Map what exists: where it runs, who has access, what has already gone down and what nobody can explain. Without that, taking over a server is driving with a fogged windshield.

  2. Priorities

    What first? Almost never the elegant architecture. It’s a backup that restores, access under control and an alert that reaches a person — in that order, because that’s the order of what hurts.

  3. Execution

    Changes in stages, each one reversible. Infrastructure is where one big change at once is how you bring down what was standing.

  4. Operation

    The steady state: watching, tuning, testing restores again and reviewing cost. This is where the work stops being a project and becomes a contract.

Questions

What people usually ask

Is this not what maintenance already covers?

They are different layers and they usually travel together. Maintenance looks after the software — fixing, evolving, handling requests. This front looks after the ground it runs on: server, releases, backups and alerts. You can hire just one; what you can’t do is assume hiring the first solves the second, which is the most expensive misunderstanding in this area.

We already have hosting. Do we need to switch providers?

Most of the time no, and when the answer is yes you will hear the reason as a number, not an opinion. Much of the gain is in configuring properly what already exists: a backup that restores, a deploy with no manual step, and an alert that reaches a person. Migration is an expensive decision and only enters a proposal when it pays for itself.

Do I need three servers like in the diagram?

Probably not. The diagram shows the principle, not the recipe: for most sites and stores, one well-configured machine with a tested backup and a written recovery plan is enough, and redundancy would be money sitting idle. It starts paying for itself when an hour offline costs more than the second server — and you’re the one who has that number. The math goes into the proposal.

Are you on call 24 hours a day?

No, and promising that would be a lie from a team our size. What exists is automated alerting, a response time written into the contract, and systems built to degrade rather than fall over — because most incidents are avoided in the configuration, not on the night shift. If your operation genuinely needs someone awake at three in the morning, that’s a larger setup, and we say so in the first conversation instead of finding out together during the first incident.

Do you take over infrastructure for systems you didn’t build?

Yes, and it’s a good share of the work. It starts with an assessment of what is there today, because taking over a server without knowing what runs on it is accepting the responsibility without the information. When there’s no documentation and no access, the road goes through legacy rescue first — it’s the same investigative work, and it comes first.

What happens if we want to leave later?

You take everything: access, documentation of what was built, the pipeline scripts and the restore procedure. Nothing here depends on a tool of ours that only we can operate. Infrastructure only one supplier understands is exactly the problem most of our clients arrive wanting to solve.