Deploy order is part of the design

The change was four lines. A contractor should only be able to read a job once their subscription is active, so the rule that had said request.auth != null now also had to check that a mirror document existed marking them as paid. A function already wrote that mirror on every subscription event. Deploy the rule, done.

It is not done, because those four lines are not one change. They are three changes that land at different times, and the order they land in determines whether the system is briefly wrong, briefly insecure, or neither.

Three artefacts, three deploys, three clocks

A typical Firebase and Cloudflare stack has at least three independently deployable things, and almost nothing forces them to be consistent with each other.

• Security rules, deployed by their own command, taking effect across the project within a minute or so of publishing.
• Server-side code — functions, workers, API routes — deployed by a different command, on its own build, with its own propagation.
• The client bundle, which is deployed a fourth way and then cached on devices you do not control, so old versions of it keep running for as long as people keep tabs open.
• The data itself, which does not deploy at all. It changes when something writes to it, and nothing writes to a million existing documents on your behalf.

A code review reads all of this as one state. Production experiences it as a sequence. The question a reviewer should be asking is not only is this correct, but is every intermediate state survivable.

The two orders, and what each one costs

For the subscription change, there are two ways to sequence it.

Rules first. The moment the rule is live, every request is checked against a mirror document that mostly does not exist yet, because the function that writes it has not shipped and no backfill has run. Every existing paying contractor is locked out of their own jobs. Nothing is insecure; everything is broken. Support finds out before monitoring does.

Rules last. Ship the function, run the backfill, confirm the mirror is complete, then ship the rule. During the gap the old rule is still in force, which means access is looser than you intend for a few minutes. Nothing new is exposed that was not already exposed yesterday.

The second is right, and the reason is worth stating precisely rather than as a rule of thumb. The intermediate state of a tightening change should be the state you are already in, not the state you are heading to. You are moving from a weaker rule to a stronger one; the correct interim is the weaker rule, which is a risk you have already been carrying, as opposed to the stronger rule evaluated against data that has not caught up, which is an outage you are choosing to create.

Loosening changes invert it. If a rule is being relaxed and code depends on the new access, the rule goes first, because the interim is again the state you are already in: restricted. Code that tries to do the new thing fails, which is annoying, rather than users seeing something they should not, which is not recoverable by rolling back.

Stated once: the safe order is whichever one makes the gap look like the past.

The backfill is the part that actually takes the time

The rule is a minute. The function is a few minutes. The backfill is where the work is, and it is the step that gets underestimated because it feels like a chore rather than a change.

Things I now treat as non-negotiable for a backfill:

• It must be re-runnable. It will be interrupted. Write it so that running it twice produces the same result as running it once, and so that a second run skips what is already done rather than rewriting it.
• It must be countable before it runs. Know how many documents it will touch, and therefore what it costs and how long it takes, before starting. A backfill whose size is a surprise is a bill that is a surprise.
• It must be verifiable when it finishes. Not "it completed without error" — an assertion that the number of subjects equals the number of mirrors. Completion is not correctness.
• It should write nothing the live path would not write. If the backfill and the function produce subtly different documents, you have two schemas and will discover which one the rule actually accepts at the worst moment.

That last point is the one that has bitten me hardest. A backfill written as a one-off script is a second implementation of a write path, and second implementations drift. The better shape is for the backfill to call the same function the live path calls, so there is one definition of what the document looks like.

The client is already in the field

Everything above treats the browser as if it updates when you deploy. It does not. Someone has had a tab open since yesterday, someone else has a service worker serving a build from last week, and a mobile app shipped through a store has a version distribution rather than a version.

The consequence is that every rule change is also a compatibility question with clients you have already shipped. If the new rule requires a field the old client does not send, the old client starts failing writes, and the user has no idea why. The failure is silent, remote, and correlated with exactly the people least likely to report it.

The practical pattern is the same one used for database migrations: make the field optional first, ship the client that writes it, wait, then make it required. Two deploys and a wait, where one deploy looked sufficient. The wait is not caution, it is the actual mechanism.

For apps distributed through stores the wait is measured in weeks, not minutes, and some proportion of users will never update. That is a reason to be extremely reluctant about rules that require anything new from a native client, and a reason to build a version floor into the system early, while doing so is still cheap.

What I write down before shipping one of these

Not a ceremony — six lines in the pull request, which take longer to think about than to type.

1. The artefacts this change touches: rules, functions, client, data.
2. The order they ship in, and one sentence on why that order and not the reverse.
3. The intermediate state after each step, described as what a user experiences, not as what the system contains.
4. The backfill: how many documents, what it costs, how it is verified.
5. The assertion that proves the whole thing landed — a specific query or request with an expected answer, not "check the site loads".
6. The rollback, which is usually re-deploying the previous rule, and a note on what is not recoverable that way.

That last line is the one that has changed how I sequence things. Rules roll back cleanly: they are a single artefact with a previous version. Data does not. A migration that rewrites documents in place has no previous version unless you made one, which is why the safe shape is to write a new field alongside the old one and stop reading the old one later, rather than to transform in place and hope.

Deploy order in the small

The same reasoning applies well below the level of a migration, and this is where it earns its keep day to day.

Adding a field the rules validate: ship the rule that allows the field before the client that sends it, never after. Removing a field: stop sending it, wait for the clients to turn over, then stop allowing it. Renaming anything at all: do not — add the new name, write both, read the new, drop the old months later. A rename is a delete and a create wearing one commit, and the documents that already exist do not know that.

Splitting a monolithic deploy into several is usually the right instinct, with one exception worth naming: on platforms that bill per build, ten small deploys cost ten builds. The resolution is to batch the artefacts that are safe to land together and separate only the ones whose order matters. Order is a correctness property; deploy count is a cost property. Optimise them separately.

The general version

Configuration, schema and code are one system, and only one of them has a version control history that anybody looks at. Rules live in a repository but take effect the moment they are published. Data has no version at all. The client has many versions at once.

So the diff is not the change. The change is the diff plus an ordering plus a wait, and the ordering is a design decision with a right answer that is derivable rather than a matter of taste: make the gap look like the past. Write the order down in the pull request, because the person deploying at eleven at night is going to be you, and you will not re-derive it then.