From Technical Safety to Societal Impact: Rethinking AI Governance
Contents
Executive Summary
This panel discussion reframes AI safety beyond technical robustness (model alignment, benchmarks, red teaming) to encompass governance, institutional contexts, human rights, and societal impact. Speakers argue that AI systems fail not due to technical flaws alone but because they are embedded in exploitative institutional, economic, and political systems. The conversation emphasizes the need for multidisciplinary approaches, inclusive decision-making, transparency mechanisms, and government-level enforcement to address real-world harms currently affecting vulnerable populations.
Key Takeaways
-
Safety requires accountability beyond intentions. Voluntary corporate commitments, panel discussions, and "should" statements are insufficient. Real safety enforcement requires legal consequences, government regulation, and a 51% political threshold for action.
-
Affected communities must be present in design, not consulted afterward. Genuine inclusion means women, marginalized populations, and those harmed by AI systems are decision-makers during design—not token participants in post-hoc evaluation. History shows that tokenism reproduces extraction.
-
Transparency means stating tradeoffs explicitly. Companies and researchers must publicly document what their systems don't cover, which languages perform poorly, which populations are underrepresented, and why. Vague promises of "safety for everyone" obscure these necessary hard choices.
-
Regulatory frameworks must be multilingual, context-aware, and dynamic. A single English-language model card cannot address safety across 2,000+ human languages and vastly different institutional and cultural contexts. Governance artifacts must adapt as systems change and as evidence from diverse regions emerges.
-
The future depends on insistence, not good faith. As the closing moderator stated: happiness and wellness for all will not arrive automatically. Panelists and audience members must actively push politicians, communities, and organizations to adopt stronger positions on safety, labor protections, and equitable benefit-sharing.
Key Topics Covered
- Beyond technical safety: Limitations of focusing solely on algorithmic robustness, alignment, and model performance
- Governance and policy frameworks: National AI strategies, multidisciplinary regulation, and institutional accountability
- Representation and diversity: Gender disparities in AI leadership and the exclusion of affected communities from design processes
- Data and infrastructure sovereignty: Data policies, data centers, cloud computing, and their societal implications
- Multilingual and multicultural AI: Language representation gaps and context-specific safety evaluation
- Real-world harms: Labor exploitation, extractive practices, groundwater depletion, biased targeting of vulnerable populations
- Transparency mechanisms: Model cards, data cards, system cards, and documentation requirements
- Regulatory enforcement: The need for accountability, legal consequences, and enforcement beyond voluntary compliance
- AI measurement and metrology: New frameworks for studying sociotechnical systems ("social machines")
- International coordination: Role of summits, UN bodies, and cross-border governance
Key Points & Insights
-
Technical safety is necessary but insufficient. Model alignment, robustness, and benchmarking matter, but they do not address whether AI systems produce human value or harm in real deployment contexts. Safety failures often stem from institutional, economic, and political systems, not architectural flaws.
-
Governance must center human protection, not system protection. Mozambique's national AI strategy frames safety as "protection of people," requiring continuous human oversight, institutional accountability, and transparency about how algorithmic decisions affect individuals' lives.
-
Representation in decision-making is both a safety and ethical issue. Wendy Hall emphasized that 50% of the population (women) were excluded from the main summit sessions despite "AI" meaning "all inclusive." Without diversity in designing systems, hidden biases cannot be identified or corrected.
-
Input and output controls are equally critical. Professor Yanidis distinguishes between the AI technology itself (which should innovate freely) and its use, noting that both the data fed into systems (input) and decisions made by systems (output) require multidisciplinary oversight, not just technical review.
-
Current harms are immediate and documented, not hypothetical. Jibu Elias highlighted extractive practices: tribal populations excluded from AI benefits, languages not represented in models, groundwater depletion from data centers, and community leaders being bribed rather than genuinely consulted. Accumulated risks are more pressing than existential risks.
-
Precision in safety conversations matters more than universal rules. Sara Hooker argues that expecting universal AI rules is unrealistic; instead, forums should focus on precise definitions of safety tradeoffs. Critically, companies should transparently report what their models don't cover and what populations they have deprioritized.
-
History suggests regulation requires crisis or 51% political will. Tom Romangh's "51% rule" indicates that meaningful regulation typically emerges only after public outcrage (e.g., deepfake nudes) or political consensus reaches a threshold. Voluntary corporate compliance is insufficient without enforcement mechanisms.
-
Transparency documents (model/data/system cards) must be dynamic, multilingual, and context-specific. Current documentation is predominantly English-language and static, failing to reflect performance variations across languages, regions, and use cases—essential for serving non-English-speaking populations.
-
Development and inclusivity frameworks from other disciplines are underutilized. Niha Kumar cited feminist studies, design thinking, and development studies, which offer decades of research on who benefits, who decides, and how historical extraction repeats. AI governance should integrate these perspectives from the outset.
-
Government is the only institution capable of integrating all stakeholder perspectives. Despite imperfect summits and forums, governments—not tech companies or civil society alone—can balance technical, civil society, and industry concerns and enforce tradeoffs across constituencies.
Notable Quotes or Statements
-
Virginia Dignam (moderator): "AI systems do not fail simply because of flaws in the model architecture or in the data or in the alignment techniques. They fail or they produce harm because they are embedded in institutional, economic and political systems."
-
Wendy Hall: "If it's not diverse, it's not ethical… If you haven't got a diversity of people discussing a problem, how are you going to actually sort out the biases? If you haven't got women at the top level making these decisions trying to set up the guidelines, it's not going to work."
-
Sara Hooker: "Too often when these conversations enter this arena, there's a misconception about the sheer difficulty of how do you actually impose constraints on these systems… The truth is that there will be some tradeoff. Someone will not be represented. That's okay. It's more that they have to be stated out loud."
-
Jibu Elias: "There are real consequences right now on people's rights, freedoms, ability to live with their dignity and people's rights to participate in democratic processes. All of these are undermined… This is the time to get up get your voices up as citizens as consumers as professionals in your own right and try to change the narrative because otherwise it's going to just be a repeat of history."
-
Tom Romangh: "There is also this conversation around moderates… It's not until some kind of action happens, some kind of consequence, some kind of issue happens that people wake up to the folks who've been screaming about it for years. So what I encourage everybody in here is not be a moderate. Pick a side and start encouraging your politicians, your family, your community."
-
Rasmus Anderson (referencing regulatory history): "When you get into a car, there's a risk something will happen, but you still need to get places… With safety, we do have to take some of the same lessons as from nuclear, from flights. It used to be that when you got on an airplane, something like 200 or a thousand more of them crashed than today, and we've reduced that level of risk very far down."
Speakers & Organizations Mentioned
| Speaker | Title / Role | Organization / Country |
|---|---|---|
| Virginia Dignam | Co-host / Chair | ACM Technology Policy Council |
| Gina Matthews | Co-host / Chair | ACM Technology Policy Council |
| Dr. [Name withheld—chairman of the board] | National AI strategy lead | National Institute of Information and Communication Technology, Mozambique |
| Dame Wendy Hall | Registered Professor of Computer Science, former UN advisor | University of Southampton; UN High-Level Expert Advisory Body on AI |
| Professor Yan Yanidis | President of ACEN; Technical expert | University of Athens |
| Sara Hooker | Co-founder and President | Adaptation Labs; (formerly DeepMind, Cohere) |
| Jibu Elias | Researcher and activist | Musla Foundation (implied) |
| Niha Kumar | Associate Professor, HCI | Georgia Tech; President of ACM SIGCHI |
| Mar Hook | President and Policy Director | Center for AI and Digital Policy |
| Rasmus Anderson | AI advisor to leaders | Tony Blair Institute for Government |
| Tom Romangh | Director of Policy | ACM; former think tank researcher (Washington DC) |
Other organizations/bodies referenced:
- UNESCO (ethics principles, Mozambique partnership)
- National Physical Laboratory (UK equivalent of NIST)
- UK AI Security Institute
- UN High-Level Expert Advisory Body
- Bletchley Declaration/Summit
- India AI Impact Summit (hosting government)
Technical Concepts & Resources
Frameworks & Tools
- Model Cards: Documentation providing provenance, evaluations, limitations, and capabilities of AI models
- Data Cards: Documentation on data sources, processing, and composition
- System Cards: Higher-level documentation covering system-level evaluations and decisions
- Red teaming: Adversarial testing of AI systems for safety vulnerabilities
- Algorithmic alignment: Technical approach to ensuring AI objectives match human values
- AI measurement / AI metrology: Proposed new field for systematically studying sociotechnical systems ("social machines")
Policy & Governance Concepts
- UNESCO principles on AI ethics: Framework adopted by Mozambique and other pilot countries
- The 51% rule: Concept that regulatory action requires 51% political will across political spectrums
- Sociotechnical systems / "social machines": Artifacts created by the intersection of technology and society (originating from Tim Berners-Lee's web science work)
- Multidisciplinary governance: Integration of law, social sciences, education, labor, ethics, and community input
- Institutional accountability: Transparency and continuous human oversight of algorithmic decision-making
Regulatory & National Efforts
- Mozambique: Drafting national AI strategy; data policy and implementation strategy; cyber security strategy revision; data center regulations; cloud computing regulations; digital government interoperability framework
- United Kingdom: National Physical Laboratory's center for AI measurement; AI Security Institute
- France: Proposed age limit (<15) for social media
- Australia: Ban on under-16 social media use (6-month evidence gathering period mentioned)
- Spain: Proposed age restrictions (16+)
- California: Started requirements for regulatory artifacts and documentation
Global Initiatives
- AI for Good Conference (Geneva, July): UN dialogue on AI governance
- Series of AI Safety Summits: Bletchley (initial), followed by 4+ additional summits, trending toward greater precision on safety definitions and real-world impact
Emerging Research Areas
- AI psychosis: Term referenced by Jibu Elias regarding elderly users' interactions with AI models (validity unclear per speaker)
- Longitudinal studies on social media harm: Emphasis on data collection and evidence over time, not snapshots
- Non-consensual deepfake detection and enforcement: Recent success with platforms (e.g., Elon Musk's Grok) stopping creation of non-consensual intimate imagery
Methodological & Process Insights
- Evidence-based regulation requires time: Behavioral and social impacts take longer to measure than immediate technical performance metrics (6-month snapshots are insufficient).
- Intersectional lens: The panel repeatedly noted gender, language, geographic, and economic dimensions of AI harms are inseparable—safety cannot be addressed in silos.
- Tradeoff transparency: Systems serving billions of people will have static, monolithic models with inherent tradeoffs; the ask is to explicitly document which groups/contexts are deprioritized.
- Post-summit action: Organizers indicated intent to produce a shared report documenting priorities, practical issues, heuristics, methodologies, and measures to continue dialogue beyond the event.
Critical Gaps & Unresolved Questions
- How to implement "continuous human oversight" at scale without creating new bottlenecks or reproducing power imbalances?
- What enforces model/data/system card requirements if companies resist or provide incomplete information?
- How to prevent regulatory capture by well-resourced tech companies during rule-making processes?
- What happens when cultural or linguistic contexts have conflicting values about acceptable AI behavior?
- How to ensure marginalized communities have resources and standing to participate in AI governance, not just representation?