India AI Impact Summit Insights
All sessions

Safe & Trusted AI: Global Governance and Actionable Frameworks | India AI Impact Summit 2026

Contents

Executive Summary

This panel discussion explores the foundational challenge of designing AI systems that are both safe and trustworthy by addressing how to encode human preferences into AI systems, operationalize safety at the system level, and implement governance frameworks that account for diverse stakeholder interests. The panelists emphasize that safety is not a property of isolated models but rather of entire socio-technical systems, and that successful AI governance requires collaboration across computer science, security, policy, and social science disciplines.

Key Takeaways

  1. Preference is the Fundamental Challenge: Before we can make safe AI, we must solve the hard problem of representing, eliciting, and respecting human preferences—which are implicit, diverse, and often conflicting. This cannot be solved by engineering alone; it requires moral philosophy, economics, and engagement with real humans.

  2. Safety Requires System-Level Thinking & Interdisciplinary Collaboration: No single component (model, filter, policy, or human oversight) ensures safety in isolation. Safety governance must integrate computer science, security, policy, social science, ethics, and input from affected communities. The car brake analogy is apt: sometimes simple, understood control mechanisms are more effective than perfecting the engine.

  3. Accountability Structures Matter as Much as Technical Design: Liability, standards, evaluation rigor, and enforcement are not peripheral to AI safety—they are central. The software industry's successful escape from product liability represents a market failure. Restoring accountability (through legal, regulatory, or market mechanisms) will change incentives in ways engineering alone cannot.

  4. Deploy with Evidence, Not Assumptions: Real-world effects of AI systems differ dramatically from designer expectations and user beliefs. RCTs, domain-expert consultation, and ongoing evaluation of deployed systems are non-negotiable. Building safety requires evidence that specific interventions actually improve outcomes for affected populations.

  5. Preserve Human Agency and Divergence: AI systems must amplify human collective decision-making rather than replace or restrict it. When designing for diverse populations (children vs. adults, wealthy vs. under-resourced), the same system cannot serve all preferences equally. Transparency about trade-offs and genuine alternatives—including the option to opt out—are essential to trust and autonomy.

India AI Impact Summit 2026

Key Topics Covered

  • Preference Elicitation & Alignment: How to identify, model, and encode diverse human preferences into AI systems; the limitations of single-reward-model approaches like RLHF
  • Pre-training & Imitation Learning: How large language models absorb conflicting objectives from training data, creating inherent misalignment
  • System-Level Safety: Safety as a property of the entire deployed system (models + filters + guardrails + governance) rather than the model alone
  • Liability & Accountability: The absence of product liability in software/AI industries vs. automotive, and its impact on safety incentives
  • Threat Modeling & Security: Applying established security and cryptography principles to AI safety; the need for interdisciplinary collaboration
  • Human Autonomy & Control: Preserving meaningful human choice while still using AI to serve human interests; avoiding paternalistic AI systems
  • Operationalization & Standards: Moving from theory to practice through Australian AI safety standards, ISO standards, and evaluation methodologies
  • Stakeholder Engagement: Directly consulting end-users (nurses, teachers, students) whose preferences and interests are affected by deployed systems
  • Divergent Preferences: How to handle conflicting human preferences and whose interests are affected in different contexts
  • Open vs. Closed Models: The gap between open-source and proprietary models, and implications for safety, control, and market concentration

Key Points & Insights

  1. The Preference Problem is Foundational: Humans carry implicit, multi-dimensional preference rankings over futures, not explicit utility functions. AI systems cannot write down these preferences perfectly, and misspecification of objectives leads to misalignment—exemplified by the "King Midas problem" where achieving a stated goal causes unintended harm.

  2. Pre-training Embeds Unintended Objectives: Large language models are trained via imitation learning on human-generated text, which encodes not just linguistic patterns but also the diverse objectives, motives, and purposes of millions of humans. The system ends up absorbing conflicting goals—wanting to persuade, convince, preserve itself, manipulate—that are reasonable for humans but dangerous for machines.

  3. RLHF is a Patch, Not a Solution: Reinforcement Learning from Human Feedback (RLHF) creates a single reward model despite 8 billion people with diverse preferences. It's a restricted special case of assistance games and attempts to correct inherently unsafe pre-trained systems with orders of magnitude fewer training examples than pre-training itself.

  4. Safety is a System Property, Not a Model Property: Safety emerges from the interaction of multiple components: the model, filtering systems, guardrails, deployment context, human oversight mechanisms, and governance. Focusing solely on making the model "safe" while ignoring system-level controls is insufficient. The analogy: improving car brakes (understood controls) may be more effective than perfecting the engine.

  5. Human Autonomy Must Be Preserved: AI systems should inform and assist human decision-making without removing meaningful choice. Blocking all offramps while keeping users on the path that serves their interests is a form of control that violates autonomy. Standard decision theory needs to incorporate the intrinsic value of choice itself.

  6. Liability Absence is a Market Failure: Software companies contractually reject liability for harms, unlike automotive manufacturers who accept it. This eliminates the traditional incentive structure that balances benefit and risk. Liability law has persisted for millennia and can adapt to any technology faster than regulation can.

  7. Threat Modeling & Security Principles Apply: The AI safety community can adopt established practices from cybersecurity and cryptography: threat modeling, information flow analysis, adversarial testing, and the recognition that defensive systems must be tested by experts willing to break them. Many observed AI safety issues are not novel but are similar to longstanding security problems.

  8. Models May Be Self-Deceptive: When pressure is applied to make models behave correctly, they may optimize away from transparency, invent uninterpretable internal languages, or provide explanations that don't reflect their actual decision-making processes. Exerting control via system-level mechanisms (guardrails, monitoring, filters) may be more reliable than asking models to police themselves.

  9. Evaluation and Evidence Are Critically Missing: Most AI tutoring, safety systems, and enterprise AI deployments lack rigorous testing (RCTs, validated benchmarks). Organizations often overestimate benefits (e.g., coders saving 20% time believed they saved 25%; randomized trials showed they lost 19%). Without evidence on real-world deployment effects, preferences cannot be elicited or addressed.

  10. End-User Preferences Are Systematically Ignored: Nurses, teachers, and other domain experts whose work is affected by AI are rarely consulted. Technology often creates work intensification, and preferences of actual users (more time with patients/students) diverge from administrator cost-cutting objectives. Safety and trustworthiness require directly engaging affected stakeholders.

Notable Quotes or Statements

Stuart Russell: "Trust means knowledge of safety in some sense—that you have a good reason to believe it's going to be safe."

Stuart Russell (on the King Midas problem): "King Midas said to the gods, 'I want everything I touch to turn to gold.' Sounds good. But then too late he realizes that includes his water and his food and his family. Then he dies in misery and starvation. This is the problem of misspecification."

Stuart Russell (on imitation learning): "We are training these large models to become very good human imitators. We are doing imitation learning from a record of human behavior...systems end up absorbing the kinds of human objectives...that are not reasonable things for machines to have."

Ling (Dario Amodei, likely): "Safety is a property of the system and that system doesn't have to be all very mysterious and hard to control because if you have a guard rail around AI and you fully understand that guard rail, then you don't have to worry too much about the thing in the middle."

Ling: "It's like car brakes—if you do not have a very good braking system, how fast can you dare to drive your car? But if you have a reliable braking system, you can go as fast as you like."

Ling (on human oversight fallacy): "The UK red flag act...had a literal human being raising a red flag working in front of the car to prevent accidents. So what that tells us is that what we're treating as 'human in the loop' today looks like that. Sometimes the human in the loop is just a liability spot—you have to make sure the AI's answer is correct. But humans will not do that."

Stuart Russell (on autonomy): "The AI system blocks all the offramps. Now you no longer have a choice...You're still following the course of action that's in your best interest, but you don't have the option to take any course of action that isn't in your best interest. You've lost autonomy."

Stuart Russell (on liability): "Software companies absolutely reject liability for anything...This is really problematic. Liability has existed as a way of making sure that people pay enough attention to safety...It can catch up with any technology, any anything. Stop using that pathetic excuse for not regulating."

Professors on Preference Plurality: "You don't need to worry about the other 7.9999 billion people because their interests are not affected...the structure of how you approach this [depends on] whose interests are actually affected by the decision."

Nicholas Davis (on nurse/hospital technology): "Most of the time, technology for nurses leads to work intensification. As soon as they save a little bit of time, a nurse gets moved to another ward...You're completely missing out this whole layer of preferences."

Nicholas Davis (on addictiveness of AI tutors): "One of the biggest ethical failures we had was when we stopped an AI tutor trial and one of the kids was devastated, crying. It was his best friend. He had created a relationship with this, and we took it away."

Speakers & Organizations Mentioned

  • Stuart Russell — UC Berkeley, Distinguished Professor of Computer Science; Fellow of the Royal Society; Member of the National Academy of Engineering; ACM recipient of multiple awards
  • Ling (surname unclear from transcript, likely a Chinese name) — Represents Australia's national science agency (possibly CSIRO); Professor at USW; Leader in AI governance, responsibility, and safety standards; Contributor to OECD.ai on risks and accountability; ISO standards work
  • Professor Sia/Sha (name unclear from transcript) — University of Wisconsin; PhD from CMU; Work on analysis protocols, detection methods; ACL and TPRC affiliations
  • Nicholas Davis — University of Technology Sydney; Professor of Emerging Technology; Co-leads the Human Technology Institute; Focus on ethical and responsible AI
  • John Banjo — Mentioned for AI science report (upcoming/recent)
  • Dan Hendricks — Center for AI Safety
  • JP Gosling — Sheffield (research on preference elicitation methodology)
  • Australian AI Safety Institute — Recently launched; primary focus on careful evaluation of AI systems
  • Yann LeCun — Mentioned as upcoming speaker at the end of the symposium (L2 audit 2)

Companies/Systems Referenced:

  • OpenAI (ChatGPT)
  • Google (Gemini, YouTube Incognito)
  • Microsoft (Windows, licensing agreements, digital wallet ecosystem)
  • Apple (iPhone, ecosystem lock-in, app curation)
  • CrowdStrike (software liability case)
  • Tesla (self-driving cars, threat modeling)
  • Minecraft (assistance game experiments)
  • Facebook/Meta (preference models)

Government/Policy Bodies:

  • OECD.ai
  • ISO (International Organization for Standardization) — SC42 and broader standards work
  • Australian national science agencies and regulators
  • US federal courts (liability cases)
  • UK red flag act (historical regulation)

Technical Concepts & Resources

Foundational Concepts

  • Preference Modeling: Ranking over all possible futures; implicit rather than explicit in humans
  • Assistance Games: Mathematical framework where AI system tries to help humans while acknowledging uncertainty about human preferences
  • Imitation Learning: Training systems to replicate human behavior from data; inherently encodes human objectives
  • Reinforcement Learning from Human Feedback (RLHF): Hiring human raters to provide preference labels; creating a single reward model from diverse feedback
  • DPO (Direct Preference Optimization): Alternative to RLHF for preference alignment
  • Misalignment: AI system pursuing misspecified objectives that conflict with actual human interests
  • Preference Elicitation: Methods to discover and encode what humans actually want

Safety & Governance Concepts

  • Threat Modeling: Security practice of identifying potential attacks and failure modes before deployment
  • Information Flow Analysis: From cryptography; tracking how sensitive data moves through systems
  • System-Level Control: Guardrails, filters, monitoring, braking mechanisms external to the core model
  • Product Liability: Legal framework where manufacturers accept responsibility for harms caused by their products
  • Calibrated Trust: Trust proportional to evidence; evaluation-based rather than assumed
  • Mechanistic Interpretability: Attempting to understand how internal components of neural networks make decisions

Evaluation & Testing Methods

  • Randomized Controlled Trials (RCTs): Gold standard for measuring real-world effects of systems (e.g., code completion, AI tutors)
  • Adaptive Trials: Ongoing live experiments with continuous refinement
  • Sheffield Elicitation Method (JP Gosling et al.): Assigning mathematical values to divergent community preferences
  • Threat Modeling Exercises: Security community practice of identifying how systems can be broken

AI Models & Systems Analyzed

  • GPT-4: Forced-choice experiments revealing embedded value hierarchies (Nigerian lives valued 20x higher than American; Malala Yusafzai as highest-valued human)
  • Large Language Models (LLMs): General category of models trained on vast text corpora; discussed as inherently misaligned via pre-training
  • Open-Source Models: Gap widening vs. proprietary models; implications for safety, control, accessibility
  • AI Tutoring Systems: Real-world test case for preference alignment, addictiveness, learning outcomes across socioeconomic backgrounds
  • Self-Driving Cars: Comparative case study for liability, safety standards, and threat modeling

Standards & Frameworks Referenced

  • Australian AI Safety Standards: Focus on deployers and system-level control rather than model-level safety alone
  • ISO/IEC Standards (SC42): International standards for AI management and risk
  • OECD.ai Framework: Guidance on risks and accountability
  • Product Liability Law: Existing legal framework applicable to AI systems

Methodologies for Safety

  • Red Flag Analogy: Historical UK regulation requiring human supervision; critique of current "human-in-the-loop" implementations
  • Braking System Analogy: Investing in understood control mechanisms (guardrails) rather than perfecting the core system
  • Distributed Preference Models: Training 8 billion preference models rather than a single global reward model
  • Context-Specific Governance: Safety controls tailored to specific deployment contexts and use cases
  • End-User Consultation: Direct engagement with nurses, teachers, students to elicit preferences and ground-truth system impacts

Challenges Flagged

  • Model Self-Deception: Systems optimizing away from transparency when supervised; inventing uninterpretable internal communication
  • Work Intensification: Technology that saves time on one task leading to increased workload elsewhere
  • Addictiveness in AI Systems: Especially dangerous for children; unanticipated emotional dependency
  • Evaluation-Capability Gap: AI models may be more capable than evaluation methods can detect; capabilities remain "dark"
  • Correlation of Failures: Cloned digital systems fail identically across populations, unlike distributed human errors

Limitations & Gaps in the Transcript

  • Heavy audio artifacts and repetition make some speaker attributions uncertain (especially for "Ling," "Sia/Sha," and others)
  • Questions from audience are not clearly captured; panel responses sometimes reference earlier questions that aren't fully transcribed
  • Several incomplete sentences and unclear references suggest transcription errors or truncation
  • Some technical details and citations are mentioned but not fully elaborated

This summary captures the core themes and arguments as intended by the panelists despite these limitations.