About this position
About the Role
We are building the next generation of AI infrastructure: a large-scale inference platform powered
by custom AI accelerators capable of serving frontier dense and Mixture-of-Experts (MoE)
foundation models at datacenter scale. As a Principal Software Engineer, you will define and lead
the architecture of the software stack that transforms hundreds to thousands of accelerator ASICs
into a unified inference system. You will drive the design of distributed runtimes, communication
libraries, scheduling systems, memory management, and model-serving infrastructure that deliver
industry-leading throughput, latency, and efficiency. This is a highly cross-functional leadership
role working across silicon architecture, networking, compiler, runtime, infrastructure, and AI
research teams.
What Makes This Role Unique
This role sits at the intersection of distributed systems, AI infrastructure, networking, compilers,
and computer architecture. You will help shape the software foundation for a new class of AI
supercomputers and define how future foundation models are served efficiently at scale. The ideal
candidate is equally comfortable discussing transformer architectures, network topology, runtime
scheduling, memory systems, and accelerator microarchitecture, and can translate those insights
into production-grade software that powers large-scale AI services.
Responsibilities
Technical Leadership
• Define the software architecture for large-scale AI inference systems.
• Lead the evolution of the distributed runtime stack for dense and MoE foundation models.
• Establish long-term technical strategy for performance, scalability, reliability, and
programmability.
• Influence hardware-software co-design decisions across future accelerator generations.
Distributed Inference Runtime
• Architect runtime systems supporting:
o Tensor parallelism
o Pipeline parallelism
o Expert parallelism
o Data parallelism
o Multi-rack inference execution
• Design systems for efficient prefill and decode execution.
• Drive continuous batching and request scheduling strategies.
• Optimize TTFT, TPOT, throughput, utilization, and QoS.• Experience in AI accelerator heterogenous programming models like CUDA/HIP/Triton
implementing device management, memory management, Kernel launching, Error handling
and diagnostics, Kernel optimization techniques with tiling and fused kernels.
MoE Infrastructure
• Lead design of token routing and expert execution systems.
• Define expert placement and replication strategies.
• Develop mechanisms for expert load balancing and congestion control.
• Optimize communication and compute overlap across distributed expert clusters.
Communication and Fabric Software
• Architect high-performance collective communication libraries.
• Design software interfaces for custom accelerator fabrics.
• Develop scalable implementations of:
o AllReduce
o AllGather
o ReduceScatter
o Broadcast
o Expert dispatch and gather
• Drive latency and bandwidth optimization across accelerator clusters.
Memory Systems
• Define architecture for distributed KV-cache management.
• Drive memory placement, eviction, and tiering policies.
• Optimize utilization across local memory, pooled memory, and storage tiers.
• Create runtime services for dynamic memory allocation and workload balancing.
Performance Engineering
• Establish performance methodology, benchmarks, and KPIs.
• Analyze bottlenecks across:
o Compute
o Memory bandwidth
o Communication fabric
o Expert routing
o Scheduling layers
• Lead root-cause analysis of production performance issues.Infrastructure and Production Systems
• Drive reliability, observability, telemetry, and debugging architecture.
• Define health monitoring and fleet management capabilities.
• Partner with infrastructure teams to operate inference systems at scale.
• Establish production readiness standards and operational excellence practices.
• System and Infrastructure experience - Experience implementing Virtualization
software, Rack controllers, Pcie/Memory controllers, device life cycle management
software, Experience with NCCL/RCCL/MPI collective communications, Scale Up
networking with UALoE/ESUN/NVLink, scale out networking with RoCEv2, Network
congestion control techniques like PFC/DCQCN.
Qualifications
Required
related field.
• BS, MS, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or
• 12+ years of software engineering experience.
• 5+ years building distributed systems at scale.
• Strong expertise in C++, Rust, or similar systems programming languages.
• Deep understanding of large-scale AI inference systems.
• Experience with networking and distributed communication systems.
• Proven track record leading architecture across multiple engineering teams.
• Experience driving projects from concept through production deployment.
Preferred
• Experience with large language model serving systems.
• Experience with vLLM, SGLang, TensorRT-LLM, Triton, PyTorch Distributed, NCCL, or
equivalent technologies.
• Experience designing runtime systems for custom accelerators, GPUs, TPUs, or AI ASICs.
• Knowledge of transformer architectures, sparse models, and MoE systems.
• Experience with RDMA, InfiniBand, RoCE, PCIe, Ethernet fabrics, or custom interconnects.
• Experience in compiler technologies, graph optimization, or kernel scheduling.
• Familiarity with HBM, memory hierarchy design, and cache management strategies.
Success Metrics
In the first 12–24 months, this leader will be expected to:
• Deliver industry-leading TTFT and TPOT performance.
• Achieve high accelerator utilization across production clusters.• Enable scalable execution of frontier dense and MoE models.
• Improve throughput per rack and throughput per watt.
• Reduce communication overhead across distributed inference deployments.
• Establish a robust and production-ready AI runtime platform.
• Influence future hardware generations through software-driven architectural insights.
Location and commute
San Jose, CA .
Plan your route with Google Maps. Travel times and transportation options vary; confirm the work location with the employer before commuting.
How to apply
Review the vacancy and complete the application on the employer or recruitment partner’s website. Requisition ID: WJ-3254371238.