HETEROGENEOUS COMPUTE FOR DEMOCRATIZING ACCESS TO AI

Contents

Executive Summary

This panel discussion explores how heterogeneous compute architecture—distributing AI inference across devices, edge cloud, and data centers—can democratize AI access while addressing critical infrastructure, energy, and security challenges in India and globally. The speakers emphasize that a hybrid approach combining on-device inference, edge processing, and centralized computing is essential for achieving cost efficiency, energy sustainability, and national resilience, particularly for countries facing constraints in land, water, and power resources.

Key Takeaways

Heterogeneous compute is the economically and environmentally sustainable model for AI scaling. Concentrating compute in megadata centers is not feasible; distribution across devices and edge is mandatory.
Inference at the edge (not training) is India's immediate opportunity. With 300+ startups building applications, focus should shift to efficient inference deployment in devices and edge cloud using sovereign, domain-specific models.
Security and resilience require mathematically-verifiable trust, not assumptions. Building trusted AI systems for critical infrastructure demands formal definitions of trust, adversarial protections, and user-level data governance.
Energy efficiency and infrastructure capacity are hard constraints, not optimization targets. Power usage efficiency (PUE), cooling architecture, and hybrid renewable energy are non-negotiable design requirements, especially in resource-constrained regions.
Enterprise AI adoption hinges on three pillars: infrastructure fit-for-purpose, security/safety guardrails, and high-quality accessible data. Organizations must discover AI assets, scan for vulnerabilities, prevent unauthorized access to sensitive information, and leverage proprietary datasets.

Key Topics Covered

Heterogeneous Compute Architecture: Distributed inference across devices, edge, and data centers rather than centralized cloud processing
Voice & Multilingual AI: Supporting 14+ languages for natural user interface on devices
Device-Level Inference: Running large models (10B parameters) on smartphones and sub-1B models on wearables
Network Connectivity Resilience: AI systems must function reliably with or without network connectivity
Enterprise AI Adoption Barriers: Infrastructure constraints, power consumption, compute limitations, and networking challenges
Security & Safety in AI: Model vulnerabilities, adversarial AI, hallucinations, toxicity, and data protection
Data Sovereignty & Sovereign Models: Building country-specific LLMs using high-quality, locally-controlled datasets
Energy Efficiency & Infrastructure: Power usage efficiency (PUE), cooling strategies, and hybrid renewable energy solutions
Critical Infrastructure Protection: Applying AI to public systems while maintaining security and resilience
India-Specific Challenges: Land, water, and power constraints; leapfrogging opportunities; SMB and education sector applications

Key Points & Insights

Hybrid AI is Essential, Not Optional: A three-tier model (devices → edge cloud → data centers) optimizes for different workload types rather than concentrating all compute in centralized locations. This reduces overall infrastructure burden and improves resilience.
Current Device Capabilities Are Underutilized: Modern smartphones can run 10B-parameter multimodal models, and smartglasses can run sub-1B models on 24-hour charge cycles. This capability remains underexploited in commercial applications.
Network Invariance is a Critical Design Requirement: AI experiences should function consistently regardless of network connectivity quality. This mandates edge inference capabilities as a fallback, not just an optimization.
Power Consumption is the Binding Constraint: In data centers, power distribution breaks down as: 40% cooling, 40% compute, 20% connectivity. As workloads intensify (1GB to 10GB+ in 5 years), the entire infrastructure model must be reimagined. Liquid cooling becomes necessary above ~25kW per rack, adding complexity and cost.
Data Sovereignty Requires Rethinking Model Training: Public foundation models trained on public data cannot serve all enterprise and government needs. High-quality, sensitive, or domain-specific datasets owned by organizations and nations must be leveraged to build specialized "machine GPTs" for local deployment and inference.
Security & Safety Are Distinct Problems Requiring Different Solutions:
- Safety = ensuring models behave as intended (preventing hallucinations, toxicity)
- Security = defending against external adversarial attacks and model poisoning
- Both require visibility across the stack and mathematically-defined trust frameworks
Trust is Context-Dependent and Non-Mathematical in Traditional Definitions: Trust fails standard equivalence relations (reflexive, symmetric, transitive) and varies by context and time. Building secure systems requires redefining trust as mathematically verifiable properties rather than assumptions.
Shadow AI Applications Pose Enterprise Risk: Organizations lack visibility into internally-deployed AI applications and models. Enterprises must implement asset discovery, vulnerability scanning, and guardrails to manage proliferation of unvetted AI tools.
Environmental & Geopolitical Constraints Drive Architecture Choices: Countries like India cannot rely on unlimited power, land, and water. Hybrid energy solutions, efficient cooling (air cooling up to 25kW, then liquid cooling), and edge distribution are not luxuries but necessities.
India's 300+ GenAI Startups Show Strength at Application Layer: India leads in GenAI applications and should focus on sovereign models and edge deployment rather than competing on foundational model scale, leveraging leapfrogging advantages in underserved verticals (education, SMB, rural connectivity).

Notable Quotes or Statements

On device capabilities: "Today, we can run up to a 10 billion parameter model, multimodal model, state-of-the-art on a smartphone and a sub 1 billion parameter model in your glasses without necessarily charging your device the whole day. It's once every 24 hours." — Qualcomm speaker (Duda Prasad)
On network resilience: "Do you want your AI user experience to be invariant to the quality of the communications that you have at that point in time or do you want it to depend on it? Obviously you want it to be invariant. That means you must have the ability to run inference directly on devices."
On trust definition: "Trust is not reflexive. Trust is not symmetric. Trust is not transitive. Trust is context dependent. I trust you on something I don't trust you on something else. It is temporal—morning I trust you, evening I don't trust you." — Dr. Kamakoti
On data sovereignty: "The enterprises, the government has the best data sets. So why can't we use those data sets? ...We can build GPTs using that...and get quality use of AI. Without data, which is the fuel for AI, you can't really move forward." — Arun Shetty, Cisco
On India's opportunity: "There is this great moment for us [to do] leapfrogging in India...this is a great moment for us" — Gokul, speaker on edge deployment
On critical infrastructure: "You need to know right. If I'm going to make a model that has understood the entire data, then someone who uses that model...should they need to know that data? That's a very important question." — Dr. Kamakoti

Speakers & Organizations Mentioned

Speaker	Affiliation	Role/Focus
Duda Prasad	Qualcomm	Heterogeneous compute, device-level inference, hybrid AI strategy
Arun Shetty	Cisco	Enterprise AI adoption, security/safety, infrastructure solutions
Dr. Kamakoti	(Indian academic institution implied)	Critical infrastructure, security, policy, sovereign models
Gokul	(Not fully identified)	Edge inferencing, domain-specific models, power/infrastructure optimization
Minister Vashnav	Indian Government	Policy framework, national AI strategy
(Unnamed panelists/moderators)	Intel (Sarah mentioned at end)	Supporting roles

Institutional References:

Qualcomm
Cisco
USC (power demand projections)
Indian government policy bodies
~300 GenAI startups in India

Technical Concepts & Resources

Concept	Context
Heterogeneous Compute	Distributed inference architecture spanning devices, edge cloud, and data centers
Hybrid AI	Qualcomm's term for combining device, edge, and server inference
Multimodal Models	AI models handling multiple data types (voice, text, video); scaling up to 10B parameters on phones
Edge Inference	Running AI inference on edge devices/cloud rather than centralized servers
Sovereign Models	Country/organization-specific LLMs trained on proprietary, high-quality, locally-controlled datasets
Machine GPT	Domain-specific generative models built on enterprise/government datasets
Power Usage Efficiency (PUE)	Ratio of total facility power to compute power; target: as close to 1.0 as possible
Adversarial AI	Malicious attacks that poison models or degrade performance
Model Hallucinations	AI outputs that are plausible but factually incorrect
Shadow AI Applications	Unvetted, undiscovered AI tools deployed within organizations without governance
Deep Packet Inspection (DPI)	Network security technique; current signature-based approach insufficient for dynamic malware
Liquid vs. Air Cooling	Air cooling viable up to ~25kW/rack; beyond that, liquid cooling required
Multimodal Models (14+ Languages)	Voice-based interfaces supporting 14+ native languages for natural interaction
Foundation Models	Large pre-trained models (e.g., ChatGPT) trained on public data

Infrastructure Metrics:

Data center power distribution: 40% cooling + 40% compute + 20% connectivity
Sub-1B parameter models: fit smartglasses with 24-hour battery
10B parameter models: run on flagship smartphones
100-300B parameter models: deployable on edge cloud with air-cooled servers

Policy & Implementation Implications

For Policymakers: Recognize land, water, and power as binding constraints; support hybrid renewable energy and distributed compute infrastructure investment.
For Enterprises: Implement AI asset discovery, vulnerability scanning, and guardrails; leverage proprietary data for sovereign model development.
For India Specifically: Focus on GenAI application leadership and edge deployment rather than competing on foundation model scale; seize leapfrogging opportunity in underserved verticals (education, SMB, rural).
For Security/Critical Infrastructure: Build mathematically-verifiable trust frameworks; deploy heterogeneous security architectures (different security solutions for different compute tiers).