Site Reliability Engineer, Ai & Hpc Infrastructure

As a Site Reliability Engineer, you will be responsible for maintaining and improving our infrastructure to ensure engineering teams across Autopilot/AI & Dojo have the necessary tools and resources to be productive. This includes managing/operating our HPC clusters, monitoring compute/GPU/network metrics, Linux troubleshooting & performance tuning, and collaborating with our Data Center team to coordinate the smooth operation of hundreds of servers & bring up new GPU capacity. Your work will directly facilitate neural network training at scale, streamline FSD development, and enable Dojo to become the most powerful supercomputer to date.

• Proficiency in Python, Golang and/or Bash
• Proficiency with Linux fundamentals and performance optimizations (Ubuntu/RHEL OS)
• Demonstrable knowledge of TCP/IP, IPoIB, Linux operating system internals, filesystems, disk/storage technologies and storage protocols
• Experience collaborating with network and data center teams for large scale cluster builds
• Experience with configuration management software (Ansible, etc.), systems monitoring & alerting (Prometheus, Grafana, Telegraf, Splunk, etc.), and/or administering HPC workload managers
• Experience with high-throughput low-latency networks, GPU-based computing systems, and/or high performance storage systems
• Experience with Slurm, LSF and storage management of distributed parallel file systems a plus
• Bachelor's Degree in Computer Science, Computer Engineering, Electrical Engineering, Physics or proof of exceptional skills in related field
• 3+ years of additional equivalent experience or evidence of exceptional ability related to the position