Long before machines, the people whose work could alter a life were held to something quiet and demanding: a standard of care. The clinician, the counsel, the trustee — each owed the person in front of them attention, competence, and honesty, whether or not it was convenient. The standard existed for the one with the least power in the exchange. It was never a courtesy. It was the price of being trusted to decide at all.
That obligation was attached to the decision, not to the hands that happened to make it. And so when the deciding moves — from a person to a model that screens, scores, ranks, flags, or forecasts — the obligation does not stay behind on the desk it left. It travels with the decision. It lands wherever the decision now lives.
The trouble is that, in the move, the duty often lands on no one. The person who built the model says they only built a tool. The person who deployed it says they only followed the output. The output says nothing at all. And the human on the other side — turned down, flagged, ranked lower, quietly filtered out — is left with a decision that shaped their life and a door with no one behind it.
A machine can make the decision. It cannot assume the duty that comes with it. Care, when a machine is deciding, has to be designed — it does not happen on its own.
i.The duty did not get smaller. It got harder to see.
It is tempting to believe that automating a decision dilutes the responsibility for it — that a thousand decisions a second cannot each carry the weight one careful decision did. The opposite is true. Scale does not soften a duty; it multiplies the number of people to whom it is owed. A standard of care that applied to one applicant in a room now applies to every applicant the system never meets. The obligation is exactly as large as before, paid out across far more lives.
What changes is only its visibility. A human decision could be questioned in the moment, to a face. An automated one is made in advance, at a distance, behind a system the affected person cannot see. The care did not leave. It went quiet. And a duty that has gone quiet is the easiest kind to neglect without noticing.
ii.What the duty actually asks — four things.
If the standard of care is to mean anything once a machine holds the decision, it has to be said plainly, in terms a system can be held to. Four obligations carry over intact from the older, human standard:
- iCompetence.first, do no harm
The decision must meet a real standard — sound, current, fit for the person it concerns — not merely whatever the model returned. A system that decides worse than a careful person is not a tool. It is a hazard that was chosen. - iiCandor.informed consent
The person has a right to know that a machine decided, and to understand the grounds in terms they can act on. Care that hides itself from the cared-for is not care. "Proprietary" is not an answer one is owed. - iiiRecourse.the second opinion
Every automated decision needs a door back to a human with the authority to look again, and to overturn. A decision no one can revisit has not been governed. It has only been imposed. - ivNon-abandonment.continuity of care
One does not deploy and walk away. The duty continues for as long as the system decides — watching for the person it begins to harm, and reaching them before the harm compounds.
iii."We will not be liable" and "they will be alright" are different sentences.
Most of what is sold as governance answers the first sentence. It assembles the record that proves, after something goes wrong, that the institution tried. This is insurance wearing the clothes of care. It protects the powerful party from the consequences of the decision; it does nothing for the person the decision fell on.
A standard of care answers the second sentence. It asks whether the person on the other side — who did not choose the system, cannot see it, and often cannot pay to be protected from it — will be treated as someone to whom something is owed. The remarkable thing is that the second sentence, honestly answered, tends to satisfy the first as well. Protection of the institution turns out to be a consequence of caring for the person, not a substitute for it.
iv.The one with the least power is still the point.
The oldest standards of care were built around the most vulnerable person in the exchange — not the institution, not the professional, but the one with the most to lose and the least say. That is the inheritance worth keeping. In an automated decision, that person is almost always invisible: the applicant behind the filter, the borrower behind the score, the patient behind the flag. They cannot see the system. They did not agree to it. They frequently cannot afford to contest it.
To build for them — to design the care a machine cannot supply on its own — is not a constraint on what these systems can do. It is the condition under which they earn the right to decide about a life at all. That is the work this standard is for: to make the quiet duty visible again, and answerable to the person it was always owed to.