Beyond GPUs: The Future of AI Compute

Contents

Executive Summary

This panel discussion at an AI summit in India explores compute infrastructure for AI beyond GPUs, emphasizing that AI requires a holistic ecosystem encompassing networking, energy efficiency, security, standards, and edge devices. The panelists argue that sustainable AI infrastructure requires thoughtful architecture, international standards, and distributed compute models—particularly relevant for the Global South where connectivity is variable and resource constraints are real.

Key Takeaways

AI infrastructure is a system problem, not a chip problem. Success requires orchestrating compute, memory, networking, power, security, and software. Isolating GPUs ignores 50% efficiency losses in interconnects and memory access.
Edge AI is critical for inclusive AI in the Global South. On-device processing (privacy, offline capability, lower latency) is essential where connectivity is unreliable, data sovereignty is sensitive, and users demand low latency—not an afterthought or feature for rich markets only.
Modularity and responsible design beat brute-force scaling. Building thoughtfully from day zero (sustainability, efficiency, modular architecture) costs less and scales faster than experimenting, then retrofitting. This applies to data centers, networks, and chip design.
Standards are sovereignty tools, not restrictions. International consensus standards (like USB-C or AI safety standards via ISO/BIS) enable countries to protect citizens and access global innovation. Isolation is expensive and slows development.
Talent & skills bottlenecks are real but addressable. India needs system architects, chip designers, and developers across physics, mechanical engineering, and software. Reimagining education and making tools accessible (e.g., AI-assisted code generation) can expand the talent pool.

Key Topics Covered

AI Compute Architecture: Moving beyond GPU-centric thinking to holistic system design (compute + memory + network + power)
Edge AI & Decentralized Computing: Processing at device level vs. centralized data centers; use cases in healthcare, agriculture, accessibility
Connectivity & Network Infrastructure: Fiber optics, deterministic latency, software-defined networks, quality of service guarantees
Sustainability & Energy Efficiency: Liquid cooling, modular design, responsible scaling; India's projected 6-7 GW data center capacity by 2032
Standards & Sovereignty: Balancing national interests with global interoperability; role of BIS and ISO in AI standards
Skills & Talent Development: Physics, semiconductor design, system architecture, and educational reimagining for AI-native economies
Economics & Business Models: Cost optimization through modularity, responsible design, and unit economics; avoiding wasteful overprovisioning
Privacy by Design: On-device processing as privacy safeguard; data residency concerns
Policy Frameworks: Market signals, innovation incentives, and lessons from Vietnam and Singapore

Key Points & Insights

Ecosystem Trust > Individual Components: Jason (ITI Council) emphasized that trust in the entire tech stack—from data centers to applications to security—is foundational, not just trust in a single AI service. Data and technology must cross borders while maintaining sovereignty.
AI is a System, Not Just a GPU: Bhavna (HPE) used a city planning metaphor: GPUs are tall buildings, but you also need roads (data), transport (networks), and planning (architecture). Poor integration of any component bottlenecks the entire system.
Networking & Memory Are Constraint Bottlenecks: Naveen (Marvell) revealed that networking and memory limit compute efficiency to <50% today. Connectivity has evolved from copper to optical interconnects; interconnect optimization is as critical as processor design.
Edge AI Enables Access in Resource-Constrained Contexts: Wilson (Google) and Becky (Qualcomm) highlighted that on-device AI (e.g., Gemini Nano, Android AI Core) provides:
- Privacy by design (data doesn't leave device)
- Functionality during poor/no connectivity
- Practical applications in rural healthcare (Roxa Health), accessibility (HearSight), and agriculture
Custom Silicon > Off-the-Shelf Accelerators for Scale: Naveen stressed that different workloads (audio, video, text, 3D input/output; training, fine-tuning, inference) require custom processing units optimized for power, performance, thermal, and cost—not just procurement of generic GPUs/TPUs.
Deterministic Latency Is a Service Differentiator: Abhishek (Airtel) argued that telcos must move from "best effort" to guaranteed QoS (Quality of Service). Software-defined networking can shift power back to users to demand specific latency/speed guarantees, not just advertised maximums.
Sustainability Must Be Built In, Not Bolted On: Bhavna noted that designing for efficiency from day zero (via liquid cooling, modular architecture) is cheaper and faster than retrofitting. HPE's 300+ patents in liquid cooling exemplify this.
India's Data Center Scaling Requires Responsible Growth: Sujit (Sienna) projected India will reach 6.5–7 GW by 2032 (vs. earlier 3 GW forecast), driven by AI. Efficiency through scale-out, interconnect scaling (100G → 1.6T), and green energy (nuclear) is critical.
Global Standards Enable Sovereignty, Not Restrict It: Jason reframed sovereignty as protection of constituents through standards (like USB-C charging ports), not isolation. BIS has published 25,000+ standards but many aren't internationally recognized, limiting Indian hardware export.
Developer Accessibility Drives App Ecosystem: Wilson highlighted that making AI accessible to app developers (e.g., "agentic mode" in Android Studio for code generation) and reimagining education are essential to creating an AI-powered economy across professions (doctors, lawyers, farmers, plumbers).

Notable Quotes or Statements

Jason (ITI Council): "It's all down to trust… Trust crosses borders. Data crosses borders... We need to take an international approach."
Bhavna (HPE): "AI is actually a system. You know when whether you talk about networking, data security, it's about the software, it's about the compute… if one part fails or gets weak the whole system slows down."
Naveen (Marvell): "Compute is your Ferrari on Bangalore road and if the traffic blocks it that's not the best efficiency of the compute." [On interconnect bottlenecks]
Wilson (Google): "Most people... are interacting with AI through devices... that's how most people are experiencing AI is on devices."
Sujit (Sienna): "The highway is what fiber connectivity comes through… when you have to get AI to the edge what you need is a nanosecond of availability before it touches you."
Abhishek (Airtel): "We have to make that shift [from passive carriers] to programming your digital infrastructure... guaranteeing quality of service."
Jason (ITI Council): "Sovereignty is very important... but we really should look to global standards particularly around AI to make sure we can access innovations from around the world."

Speakers & Organizations Mentioned

Speaker	Organization	Role
Jason	Information Technology Industry Council (ITI)	CEO; global tech advocacy body
Bhavna Agarwal	Hewlett Packard Enterprises (HPE)	Senior VP & MD; servers, connectivity
Naveen	Marvell Technologies	Country Head; semiconductor design
Wilson White	Google	Policy, Asia-Pacific region
Becky Frasier	Qualcomm	(Specific title not given; edge AI focus)
Sujit	Sienna (telecom equipment & connectivity)	Software/vertical lead
Abhishek Bisval	Airtel (AirTEL Digital Services)	Chief Business Officer
Van/Vivven	Cone Advisory (implied moderator)	Moderator; thought leader

Institutions/Programs Referenced:

BIS (Bureau of Indian Standards)
ISO (International Organization for Standardization)
Google's AI Hub in Visakhapatnam ($15B investment, 1 GW data center, submarine cable, renewable energy)
Qualcomm AI Hub
Android Studio, Android AI Core, Gemini Nano (Google)
Willow quantum chip (Google)

Technical Concepts & Resources

AI Models & Frameworks

Gemini Nano: Smallest version of Google's Gemini LLM; deployed on-device via Android AI Core
Large Language Models (LLMs) vs. Small Language Models (SLMs)
Training, Fine-tuning, Inference: Different compute/workload profiles requiring different architectures

Hardware & Infrastructure

GPUs, TPUs, NPUs: Traditional accelerators; not optimized for all workloads
Custom silicon: Workload-specific processors for power/performance optimization
Liquid cooling: High-density compute cooling; HPE has 300+ patents
Interconnects: Evolution from copper to optical; speeds scaling from 100G → 200G → 800G → 1.6T
Smart NICs, Security Accelerators, Crypto Accelerators, Co-processors: Specialized hardware in clusters
Submarine cables: Data center-to-data center connectivity
Quantum computing: Willow chip (Google); emerging for optimization and QoS challenges

Networking & Software

Software-Defined Networks (SDN): Enable deterministic latency, QoS guarantees
Operational Support Systems (OSS): Bind compute, availability, and applications together
Agentic AI: Autonomous agents for network optimization, fault diagnosis
Android Studio "Agentic Mode": AI-assisted code generation for developers
AI-native Architecture: Infrastructure designed from day zero for AI workloads, not retrofitted

Standards

BIS (Bureau of Indian Standards): India's standards body; 25,000+ published standards; 5 AI standards proposed to ISO
ISO Standards: International consensus-based standards for interoperability, security, safety
USB-C/Universal Standards: Example of global standards enabling innovation without restricting sovereignty

Use Cases & Applications

Healthcare: Roxa Health (on-device medical note-taking, analytics in rural clinics); HearSight (audio-to-mobility for visually impaired)
Agriculture: Crop robotics (insect detection), drones (drought detection, watering)
Edge Devices: Phones, cars, smart glasses, factory robotics, wearables
Scam Call Screening: On-device detection without data leaving device
Image Processing at Edge: Instagram-like latency-sensitive applications

Energy & Sustainability

Data Center Capacity in India:
- Projected 2030: 6–7 GW (vs. earlier 3 GW forecast)
- Projected 2032–2035: 16 GW
Scale-out efficiency: Technology solutions (e.g., DeComs) can reduce power utilization by ~70%
Green energy: Nuclear, renewables; holistic data center design
Modular design: Avoiding overprovisioning and waste

Policy & Skills

Vietnam Model: 4 IoT connections per citizen by 2030; ambitious smartphone milestones
Singapore (Sea Lion, Merion goals): Government-set targets for edge AI deployment
Skill Requirements: Physics, chemistry, materials science, mechanical/thermal engineering, electrical engineering, computer architecture, system thinking, semiconductor manufacturing
Education Reimagining: Preparing workforce for AI-powered economy across professions

Note: The transcript contains some audio artifacts and repetitions (e.g., "dependencies" repeated thrice), which have been interpreted contextually. Names and titles are transcribed as spoken and may contain minor pronunciation variations.