HPC Systems Software Engineer

Lawrence Berkeley National Lab's (LBNL) NERSC Division has an opening for an HPC Systems Software Engineer to join the team.

We are looking for system developers and Software Engineers to join our team at the National Energy Research Scientific Computing Center (NERSC) to help architect, deploy, configure, and operate large scale, leading-edge HPC systems. You will work collaboratively to develop and operate large-scale compute and storage platforms to drive scientific research and progress. Working with other teams at NERSC, other laboratories, and vendors you will develop innovative solutions that enable science as well as improve the state of HPC practice on an international stage.  Using your knowledge, skills, and experience, you will help research, evaluate, and develop new technologies to support NERSC's mission of accelerating scientific discovery through high performance computing and data analysis into the exascale era.

What You Will Do:

• Participate in team-oriented agile development and management process for HPC systems

• Develop and maintain APIs to securely expose system functionality to end users

• Automate common tasks and processes to continuously improve HPC systems management

• Analyze system issues and develop solutions to improve end-user experience

• Be part of a team that installs, tests, maintains and manages HPC systems

• Assist with technology evaluation of systems and system architecture

• Work with vendors to prioritize, develop and enhance their technologies in order to better meet the needs of our users

• Be part of team providing on-call rotation for 24x7 HPC system support

Additional Responsibilities as needed:

• Provide leadership and technical guidance to group members, and members of other groups at NERSC.

• Recommend and lead implementation and deployment efforts for system improvements that enhance reliability, stability, usability, performance and security.

• Identify and evaluate emerging HPC technologies and explore new features that would create new capabilities and enhance system performance and usability.

• Participate in working/user/advocacy groups and represent NERSC and its interests to the broader HPC community.

What is Required (to be hired at a level 3):

• Typically requires a minimum of 8 years of related experience with a Bachelor's degree in Computer Science or STEM-related field; or 6 years and a Master's degree; or equivalent experience. 

• Minimum of 2 years of experience with systems programming in linux environment or management of large-scale Linux-based systems in a high-performance computing, cloud computing, or hyper-scale environment.

• Experience with C, bourne shell, and Python3 programming languages.

• Demonstrated ability to work independently as well as collaboratively in large projects, and contribute to an active and respectful intellectual environment.

• Excellent oral and written communication skills.

• Ability to derive technical solutions in a collaborative environment to meet end user requirements or needs. 

If hired at a level 4:

• A minimum of 12 years linux experience with a minimum of 6 years of experience with the management of large-scale linux based systems in a HPC or Cloud environment, or have a strong community presence for developing one of the key technologies in use on NERSC systems.

• Demonstrated excellent systems programming skills and strong knowledge of Linux internals.

• Demonstrated ability to successfully lead complex projects.

Desirable qualifications include experience in two or more of the following:

• Development of kubernetes microservices using technologies like helm or loftsman for deployment.

• Operations of kubernetes, etcd.

• Infrastructure as code solutions like argo, terraform, ansible, puppet, salt.

• Rust or Go programming language.

• Gitlab or Github Continuous Integration and Project Management.

• Agile process, scrum.

• Linux kernel interfaces, cgroups, ebpf.

• Installation, configuration, monitoring, and tuning of workload management systems such as Slurm, PBSPro, or GridEngine.

• Monitoring solutions such grafana, prometheus, ldms.

• HPC systems administration.

• HPC applications analysis, MPI.

• Specialized networking (infiniband, slingshot, high-speed networks).

• Lustre, SpectrumScale (GPFS) or other parallel file systems.

Many of these desirable qualifications can be developed on the job. As a result, applicants missing several of the desired qualifications are still encouraged to apply.

Want to learn more about Berkeley Lab's Culture, Benefits and answers to FAQs? Please visit: https://recruiting.lbl.gov/

If hired at a level 3:

This full salary range of this position is between $10,307 to $17,393 per month and is expected to pay between a targeted range of $11,595 to $14,172 per month depending upon candidates' full skills, knowledge, and abilities, including education, certifications, and years of experience.

If hired at a level 4:

This full salary range of this position is between $11,738 to $19,808 per month and is expected to pay between a targeted range of $13,205 to $16,140 per month depending upon candidates' full skills, knowledge, and abilities, including education, certifications, and years of experience.

Notes:

• This is a full-time, career appointment, exempt (monthly paid) from overtime pay.

• This position will be hired at a level commensurate with the business needs and the skills, knowledge, and abilities of the successful candidate.

• This position may be subject to a background check. Any convictions will be evaluated to determine if they directly relate to the responsibilities and requirements of the position. Having a conviction history will not automatically disqualify an applicant from being considered for employment.

• Common for Posting Export Control access role: This position will involve access to hardware, commodities, and technical information subject to export control regulations including, but not limited to, the Export Administration Regulations ("EAR") and/or International Traffic in Arms Regulations ("ITAR"). Accordingly, any hiring decision may depend in part on Berkeley Lab's ability to obtain or rely on federal government authorizations as required, if you are not a U.S. citizen, lawful permanent resident of the U.S. ("green card holder"), asylee, refugee, or other qualifying protected individual as defined by 8 U.S.C. 1324b(a)(3).

• This position is not eligible for visa sponsorship

• This position requires substantial on-site presence, but is eligible for a flexible work mode, and hybrid schedules may be considered. Hybrid work is a combination of performing work on-site at Lawrence Berkeley National Lab, 1 Cyclotron Road, Berkeley, CA and some telework. Individuals working a hybrid schedule must reside within 150 miles of Berkeley Lab. Work schedules are dependent on business needs. In rare cases, full-time telework or remote work modes may be considered.

Based on University of California Policy - SARS-CoV-2 (COVID-19) Vaccination Program and U.S Federal Government requirements, Berkeley Lab requires that all members of our community obtain the COVID-19 vaccine as soon as they are eligible. As a condition of employment at Berkeley Lab, all Covered Individuals must Participate in the COVID-19 Vaccination Program by providing proof that vaccination requirements have been met or submitting a request for Exception or Deferral. Visit covid.lbl.gov for more information.

Berkeley Lab is committed to Inclusion, Diversity, Equity and Accountability (IDEA) and strives to continue building community with these shared values and commitments. Berkeley Lab is an Equal Opportunity and Affirmative Action Employer. We heartily welcome applications from women, minorities, veterans, and all who would contribute to the Lab's mission of leading scientific discovery, inclusion, and professionalism. In support of our diverse global community, all qualified applicants will be considered for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, age, or protected veteran status.

Equal Opportunity and IDEA Information Links:

Know your rights, click here for the supplement: Equal Employment Opportunity is the Law and the Pay Transparency Nondiscrimination Provision under 41 CFR 60-1.4.