Senior Software Engineer - Infrastructure and GPU Monitoring

Software Engineer III, Infrastructure, Google Cloud

Minimum qualifications:

• Bachelor’s degree or equivalent practical experience.
• 2 years of experience with software development in one or more
  programming languages (such as Go, C++, Python) with systems-level integration and distributed systems.
• Experience with monitoring and observability stacks like Prometheus, Grafana, Elasticsearch, Logstash, Kibana (ELK), or VictoriaMetrics.
• Experience with Linux internals, server hardware components, and out-of-band management (e.g., Intelligent Platform Management Interface (IPMI), Redfish).

Preferred qualifications:

• 3 years of experience with data structures and algorithms in an academic or industry settings.
• Experience with GPU architecture and management libraries (e.g., DCGM, or ROCm).
• Experience in working with large-scale data centers or cloud provider infrastructure, and contributions to open-source monitoring or infrastructure projects.
• Experience with containerization and orchestration (e.g., Docker, Kubernetes).
• Knowledge of hardware telemetry protocols and data ingestion pipelines, and distributed systems and how to monitor them.

About the job

Google's software engineers develop the next-generation technologies that change how billions of users connect, explore, and interact with information and one another. Our products need to handle information at massive scale, and extend well beyond web search. We're looking for engineers who bring fresh ideas from all areas, including information retrieval, distributed computing, large-scale system design, networking and data storage, security, artificial intelligence, natural language processing, UI design and mobile; the list goes on and is growing every day. As a software engineer, you will work on a specific project critical to Google’s needs with opportunities to switch teams and projects as you and our fast-paced business grow and evolve. We need our engineers to be versatile, display leadership qualities and be enthusiastic to take on new problems across the full-stack as we continue to push technology forward.

In this role, you will play a role in ensuring the reliability and performance of computing assets. You will build and scale the systems that monitor hardware health, with a focus on Graphics Processing Unit (GPU) diagnostics and infrastructure observability. You will impact the up-time of server fleets by developing automated alerting, dashboards, and diagnostic tools. You will be at the intersection of hardware and software, solving stability tests for the next generation of Artificial Intelligence (AI) and compute infrastructure. You will influence the health and efficiency of GPU deployments in a changing environment.Google Cloud accelerates every organization’s ability to digitally transform its business and industry. We deliver enterprise-grade solutions that leverage Google’s cutting-edge technology, and tools that help developers build more sustainably. Customers in more than 200 countries and territories turn to Google Cloud as their trusted partner to enable growth and solve their most critical business problems.

Responsibilities

• Design and maintain software solutions for real-time monitoring of server infrastructure and GPU health.
• Develop and refine diagnostic suites to identify pre-failure indicators and performance bottlenecks in GPU clusters.
• Build and optimize monitoring pipelines, ensuring alerts are actionable and noise is minimized.
• Create automated recovery workflows to manage hardware faults without manual intervention.
• Partner with hardware vendors and site reliability teams to integrate low-level hardware telemetry into high-level monitoring platforms.

Experience Level

Senior Level