There is a version of this story where the problem is bias. Someone runs the numbers, finds a disparity, and recommends a fix. That version is cleaner. It has a villain, a corrective, and a resolution. This is not that version.
This is the version where the problem is architecture.
Hiring systems, tenant-screening services, credit models, and insurance underwriting tools are not one technology. Some use fixed rules. Some parse documents or assign conventional statistical scores. Some use machine learning. A newer layer may be marketed as artificial intelligence. But across these differences, the architectural move is similar: selected facts about a person are translated into fields, variables, scores, or classifications, and that representation begins to govern what happens next.
The gate is a representation
In employment, software may determine whether an applicant appears to meet stated qualifications, score a resume, or administer a game, test, or personality assessment. The Justice Department has warned that such tools can screen out qualified applicants with disabilities, including when an employer uses technology supplied by another company. In one concrete enforcement action, the Equal Employment Opportunity Commission alleged that iTutorGroup programmed application software to reject applicants at specified age thresholds; the company settled the case after more than 200 applicants were allegedly rejected.
In housing, a tenant-screening report may combine credit history, rental records, eviction filings, criminal records, and other data, then produce a score or recommendation. The applicant may not see the score itself. The Department of Housing and Urban Development has warned that overbroad or poorly tailored screening criteria can exclude applicants in discriminatory ways and that advanced screening systems may obscure the precise basis for a denial.
In credit, the original problem is not that carrying no debt is automatically treated as proof of risk. It is that a person with no record, too little information, or only stale information may be classified as unscored. The Consumer Financial Protection Bureau distinguishes among credit-invisible and insufficient or stale unscored files. Absence does not establish instability, but a model built to evaluate recorded history may be unable to treat absence as affirmative evidence of stability.
Insurance is different again. It is regulated primarily through state systems, and algorithms may support underwriting, pricing, claims, marketing, or fraud detection rather than a single application gate. The National Association of Insurance Commissioners’ model bulletin on insurers’ use of AI identifies risks that include inaccuracy, unfair discrimination, and limited transparency, while emphasizing that insurers remain responsible for compliance with applicable law. The bulletin is a model for state regulators, not a uniform federal rule.
These sectors should not be collapsed into one system. They use different data, pursue different objectives, and provide different legal rights. But they raise the same architectural question: which parts of a person become legible to the gate, and what happens to everything that does not?
Machine-legibility
A resume must contain terms and fields a parser can recognize. An assessment must measure the intended skill without confusing it with a disability or an applicant’s ability to use the assessment interface. A credit or rental file must contain records the model knows how to interpret. These requirements are not necessarily malicious. They are nevertheless choices about what will count as evidence.
They can also reward prior conformity. A system benchmarked against current successful workers may reproduce features of a workforce formed under earlier patterns of exclusion. The Justice Department specifically notes that comparing applicants with current employees may disadvantage people historically excluded from those jobs. The apparent neutrality of the comparison does not make the reference population neutral.
Discontinuity presents a related problem. A gap may represent illness, care work, migration, informal employment, study, incarceration, recovery, or simple unemployment. A thin domestic credit file may belong to someone with a long financial history in another country. Negative credit history may reflect a medical or family emergency unlikely to recur. HUD’s housing guidance recognizes several of these circumstances and recommends opportunities for applicants to present relevant context.
Without such a process, different lives collapse into the same field: missing history, interrupted history, insufficient evidence. The system does not distinguish among them because the distinction is not present in the representation it was given.
That is what the mechanism does. It converts selected dimensions of human complexity into a score or classification, then allows the classification to stand in for the person. The score may be useful. It may even be predictive for a defined purpose. But usefulness is not completeness, and prediction is not judgment without values.
The asymmetry of error
No screening model merely reduces error. It distributes error. A threshold trades one kind of mistake against another: advancing an applicant who will later be rejected, or excluding an applicant who would have succeeded. The balance is set through objectives, variables, validation standards, and thresholds. Those are institutional decisions even when software executes them.
Where an institution experiences an erroneous approval as a visible cost while the cost of an erroneous denial falls largely on the applicant, the system has a reason to become conservative. The rejected applicant absorbs the lost job, apartment, credit opportunity, time, application fee, or search cost. Unless law, audit, or process forces that error back into institutional view, it may never appear in the institution’s performance measures.
The institutional attraction is straightforward: speed, scale, consistency, and reduced labor at the first stage of review. Those gains are real. The question is what they purchase. Human review after an automated screen is not a complete safeguard if the human reviewer never sees the people the screen removed.
Rights are uneven, not absent
The original temptation is to say that an excluded person receives no notice, explanation, or appeal. That is too categorical. The protections differ by sector.
If a landlord takes negative action based on a covered tenant report, the Fair Credit Reporting Act generally provides a right to an adverse-action notice, a free copy of the report upon timely request, and a process for disputing inaccurate information. The Federal Trade Commission explains those rights, while also noting that an applicant may not see a proprietary score in the report received.
Creditors face a stronger explanation requirement. Under the Equal Credit Opportunity Act and Regulation B, a creditor must provide the specific principal reasons for adverse action. The CFPB has made clear that complexity is not an excuse for failing to provide accurate reasons. A creditor cannot satisfy the rule merely by saying that an applicant failed to reach an internal score.
Employment protections focus on discrimination, job-related selection criteria, and reasonable accommodation, among other duties. Insurance protections vary with state law and the particular line of insurance. A formal right may therefore exist even when the practical path to identifying and correcting the problem remains slow, fragmented, or difficult to use.
This distinction matters. Opacity is not always the literal absence of a notice. It may be the gap between receiving a notice and understanding the decisive mechanism; between obtaining a report and seeing the score built from it; between disputing an incorrect record and contesting a judgment made from accurate but poorly chosen data.
The accountability problem
An automated decision may involve an institution, a software vendor, a data broker, a model developer, and a frontline employee. Each layer can make the decision harder to locate in practice. But divided production does not necessarily erase legal responsibility. Employers can remain responsible when they use another company’s discriminatory hiring technology. HUD states that housing providers retain responsibility for giving applicants a meaningful opportunity to contest screening information. Creditors remain responsible for explaining their own adverse decisions.
The stronger claim is therefore not that automation makes liability disappear. It is that automation can separate the practical experience of judgment from an identifiable human moment. The applicant encounters a result assembled across systems, while the people inside the institution encounter only their own segment of the process.
When a system automates the gate, it does not remove human judgment from the outcome. It removes human judgment from the moment of accountability.
The judgment remains in the choice of variables, the construction of the reference population, the treatment of missing data, the relative cost assigned to different errors, the threshold for passage, the accommodation procedure, and the design of the appeal. Automation embeds these choices. It does not transcend them.
Diffuse decision-making can produce diffuse responsibility even where legal duty remains intact. The institution gains distance and scale. The vendor gains a contract. The individual employee gains a process to follow. What can remain unmeasured is the population that never clears the screen and therefore never becomes visible as a mistaken exclusion.
Machine-legibility is not a neutral standard merely because a machine applies it. Passing a screen does not prove that the system captured the whole of a person’s qualification, reliability, or risk. It proves first that the person was readable under the system’s chosen terms.
That is the architecture. The question worth asking is not only how applicants can optimize within it. The question is what the institution measures, which errors it can see, and what it costs to keep calling the result objective.
Discussion