Whose Interests Are Represented in AI Design?

A fair system no applicant helped shape

Imagine an AI hiring tool that passes every technical check. Its accuracy is validated, its error rates are similar across groups, and the recruiter interface meets accessibility standards.

Yet no applicant was ever asked what a fair assessment looks like. The video interview cannot be completed with a screen reader. The scoring criteria were set by the hiring team alone. And a rejected candidate can only “appeal” by resubmitting to the same system.

Is this system fair — and to whom?

A hypothetical illustration, not a description of a particular product or employer.

The short answer

AI represents the interests of whoever can turn their needs into requirements, data, metrics, procurement conditions, legal duties or deployment authority. In most organisations that favours funders, product owners, purchasers, managers and measurable business outcomes. The harder interests to represent belong to rejected applicants, monitored workers, scored citizens, bystanders, under-documented communities and people who never chose the system.

Being included is not the same as having influence. A person can be represented in training data, invited to a consultation and given an accessible interface — and still have no say over whether the system should exist at all.

The evidence is stark. A review of 80 research projects describing themselves as “participatory AI” found that all of them involved stakeholders in user-interface design, but only 8 let them influence the model, features, objectives or thresholds; only 3 involved communities throughout the lifecycle; and none allowed participants to rule out AI as the solution.¹ Inclusion usually fails not because nobody was consulted, but because the consequential decisions were already closed.

Beyond “the user”: a stakeholder map

User-centred design is not enough when the operator is the user but someone else bears the decision. A recruiter uses the hiring tool; an applicant bears the rejection. A manager uses workforce analytics; a worker bears the monitoring.

These positions overlap but should not be collapsed. “Data subject” is a legal relationship, “community” describes collective effects, and “decision subject” identifies who bears an outcome.

How interests enter — or fail to enter — a system

business objective → product requirements → training data → labels and proxies → benchmarks → procurement → law → UX and testing → community and worker voice

Each step converts some interests into the system and leaves others out. The healthcare algorithm studied by Obermeyer and colleagues shows why “representative data” is not enough: it predicted healthcare cost as a stand-in for health need. Because less was spent on Black patients with the same illness, equally scored patients were not equally sick; fixing the target would have raised the share of Black patients receiving extra care from 17.7% to 46.5%.² The failure was not sampling. It was problem formulation and label choice.

A rigorous assessment asks who is missing from the data and why; whether the target reproduces earlier exclusion; who defined “good performance”, “risk” or “need”; whether cost, postcode, device or school encode structural disadvantage; whether average accuracy hides subgroup failure; which languages and dialects are supported; and whether several conditions combine into failures invisible in single-variable averages.

Accessibility is system architecture, not interface polish

WCAG 2.2 requires content to be perceivable, operable, understandable and robust — and W3C itself notes that even full conformance does not meet every cognitive, language or combined need.³ The European Accessibility Act applies to specified products placed on the market, and services provided to consumers, after 28 June 2025.⁴

For AI, accessibility must cover the whole service: alternative inputs, model performance across speech and language variation, comprehensible outputs and explanations, enough time and non-digital alternatives in the decision process, a reachable human who can change the outcome, and complaint routes that do not penalise anyone for using them.

There is a paradox here. AI can provide captions or voice access while working worse for the people who depend on them: an audit of five commercial speech-recognition systems found an average word error rate of 0.35 for Black speakers versus 0.19 for White speakers, traced largely to the acoustic models.⁵

Kinds of harm across the lifecycle

Fairness therefore has at least four objects: the prediction, the decision rule, the process and the resulting distribution. A model can satisfy one parity measure yet pursue an unjust objective, be accurate yet inaccessible, or be statistically balanced while affected people receive no notice or appeal.

What real cases show

Hiring — iTutorGroup (United States)

The EEOC alleged software automatically rejected women aged 55+ and men aged 60+, rejecting more than 200 qualified applicants; the settlement provided $365,000.⁶ Applicants were decision subjects who never used the system.

Healthcare risk algorithm

Cost was a biased proxy for illness despite good predictive performance.² Audit objectives and labels, not only demographic fields.

Facial recognition in retail — Rite Aid (United States)

The FTC alleged surveillance without reasonable safeguards caused false matches and customer harm; the order restricted use of the technology.⁷ Bystanders need rights even though they never chose the system.

Speech recognition

Five systems showed materially higher error rates for Black speakers in the tested corpus.⁵ Accessibility claims need contextual subgroup testing.

Housing advertising — Facebook (United States)

HUD charged in 2019 that the platform's ad targeting enabled housing discrimination.⁸ Optimisation can decide who even sees an opportunity.

Exam grading — UK, 2020

Statistical models were used to award grades, which were withdrawn after widespread concern; the statistics regulator's review highlighted public confidence, transparency and individual-level impact.⁹ Legitimacy needs procedural justice and appeal, not only a sound calculation.

Credit — Apple Card investigation (New York)

The regulator analysed underwriting data for about 400,000 applicants and found no fair-lending violation, while noting transparency and customer-service deficiencies and wider structural credit inequality.¹⁰ Not every disparity allegation holds up; inclusion governance needs independent evidence.

Across these cases, consent is often beside the point. Nobody can meaningfully “decline” an employer's screening tool, a public-benefit model or shop surveillance without giving up the opportunity itself. Legitimacy must come from necessity, proportionality, participation, safeguards and remedy — not from a fictional opt-in.

What European law provides — and what it doesn't

• EU AI Act. Article 27 requires certain deployers to identify affected persons and groups, risks, oversight and complaint arrangements before deploying covered high-risk systems; Article 86 gives affected persons a right to an explanation of certain high-risk AI decisions.¹¹ Following the AI Omnibus, Annex III high-risk rules apply from 2 December 2027.¹²
• GDPR. In SCHUFA, a score relied on strongly by a lender counted as automated decision-making.¹³ In Dun & Bradstreet Austria, the Court held that people may require an intelligible explanation of the procedure and principles actually applied — and that claimed trade secrets must be given to a court or regulator to balance, not used as an automatic shield.¹⁴
• Equality, employment and consumer law continue to apply. AI Act compliance is not a defence to discrimination or an inaccessible service.
• Council of Europe Framework Convention brings a lifecycle human-rights, democracy and rule-of-law approach, including equality and remedies.¹⁵

The limit: these frameworks regulate providers and deployers and give affected people selected rights. None is a general right to co-design, to veto, or to stop a deployment. Global frameworks — NIST, OECD, UNESCO and ISO/IEC 42001 — converge on lifecycle risk management and human rights, but none reliably guarantees that affected communities can shape the problem definition.

Meaningful participation: influence, not attendance

Participation is meaningful only when participants:

• enter before AI and the objective are fixed;
• receive understandable information about capabilities, limits and trade-offs;
• bring lived experience, operational knowledge and accessibility expertise;
• are resourced — paid, given time, translation and assistive access;
• can disagree without risk to their job, service or reputation;
• can propose non-AI alternatives and change data, thresholds, workflow or purpose;
• know who decides and what authority they hold;
• receive a documented “you said / we changed / we did not change, because” response;
• stay involved through pilots, monitoring and redesign.

Can participation become ethics-washing?

Yes. An organisation can consult affected communities, document their concerns and publicise its participatory approach without changing a single consequential decision. Participation that merely legitimises a predetermined deployment may be worse than honest non-participation, because it launders institutional choices through community presence. The difference between consultation, influence and decision authority is the difference that matters.

One question should be mandatory in every assessment: what decision was changed because affected people participated — and what happened when their recommendations were rejected?

Detecting hidden non-user harm

For every claimed benefit, trace the chain:

benefit → beneficiary → enabling data and labour → decision subject → excluded population → downstream burden → remedy owner

“Faster hiring” benefits recruiters and successful applicants, but may shift unexplained rejection, inaccessible assessment and appeal effort onto candidates who never used the system. If an organisation cannot evidence each link, the benefit case is incomplete.

Decision rules

Redesign

A material stakeholder group, accessibility route, label rationale or non-user impact has been left out.

Consult further

Evidence is thin, culturally narrow, self-selected or based on stand-ins for affected people.

Independent review

Incentives conflict with affected interests, stakes are high, trade secrets block scrutiny, or fairness claims are contested.

Restricted pilot

Remaining uncertainty is manageable, reversible, transparently monitored and excludes irreversible decisions.

Suspend

Material harm, inaccessible remedy, unexplained disparity or loss of effective oversight appears.

Withdraw

The purpose is disproportionate, the system cannot be made accessible, harms cannot be mitigated, or a less intrusive alternative achieves the aim.

Real tensions

Collecting demographic evidence can conflict with privacy. Broad participation can slow decisions. Communities contain disagreement, and a few vocal participants can capture a process. Equalising one metric can worsen another. Repeated consultation can exhaust people without changing power. The right response is neither “all algorithms discriminate” nor “the metric passed” — it is contextual evidence, independent challenge and continuing contestability.

Conclusion

The decisive question is not whether diverse people were included. It is:

Which affected interests gained power over which decisions, at what stage, with what evidence — and what happens when the system harms someone who never chose to use it?

Accessibility, dataset diversity and fairness metrics are necessary. But without participation, power, contestability and remedy, they can make an exclusionary system run more smoothly rather than make it just.

Method and limits

This is a research-led synthesis of empirical research, legislation, case law, regulatory findings and standards, prepared in October 2026. It is not legal advice; legal status is described as of 3 October 2026. I checked each source below against its official or published record. The stakeholder map, the Inclusive AI Impact Assessment and the decision rules are my own proposals.

Sources

1. Empirical research Delgado, F., Yang, S., Madaio, M. & Yang, Q. (2023). The participatory turn in AI design: Theoretical foundations and the current state of practice. EAAMO '23. doi:10.1145/3617694.3623261. ↩
2. Empirical research Obermeyer, Z. et al. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. doi:10.1126/science.aax2342. ↩
3. Standard W3C. Web Content Accessibility Guidelines (WCAG) 2.2. w3.org. ↩
4. Legislation Directive (EU) 2019/882 (European Accessibility Act). eur-lex.europa.eu. ↩
5. Empirical research Koenecke, A. et al. (2020). Racial disparities in automated speech recognition. PNAS, 117(14). doi:10.1073/pnas.1915768117. ↩
6. Regulatory action U.S. EEOC. iTutorGroup to pay $365,000 to settle EEOC discriminatory hiring suit. eeoc.gov. ↩
7. Regulatory action U.S. FTC. Rite Aid Corporation, FTC v. ftc.gov. ↩
8. Regulatory action U.S. Department of Housing and Urban Development (2019). HUD charges Facebook with housing discrimination over company's targeted advertising practices. archives.hud.gov. ↩
9. Official review Office for Statistics Regulation (2021). Ensuring statistical models command public confidence. osr.statisticsauthority.gov.uk. ↩
10. Regulatory finding New York State Department of Financial Services (2021). DFS issues findings on the Apple Card and its underwriter Goldman Sachs Bank. dfs.ny.gov. ↩
11. Legislation AI Act, Articles 27 and 86. Art. 27 · Art. 86. ↩
12. Official guidance European Commission. AI Act — regulatory framework for AI. digital-strategy.ec.europa.eu. ↩
13. Case law CJEU, Case C-634/21, SCHUFA Holding (Scoring), 7 December 2023. eur-lex.europa.eu. ↩
14. Case law CJEU, Case C-203/22, Dun & Bradstreet Austria. eur-lex.europa.eu. ↩
15. Treaty Council of Europe Framework Convention on Artificial Intelligence (CETS No. 225). coe.int. ↩