Description
GovTech is the lead agency driving Singapore's Smart Nation initiatives and public sector digital transformation. As the Centre of Excellence for Infocomm Technology and Smart Systems (ICT & SS), GovTech develops the Singapore Government's capabilities in Data Science & Artificial Intelligence, Application Development, Smart City Technology, Digital Infrastructure, and Cybersecurity.
We are looking for a hands-on storage engineer to operate and maintain the Ceph-based storage services supporting our on-premises Kubernetes and OpenShift platform. The role focuses on operating and improving production storage services, including monitoring, maintenance, upgrades, capacity management, hardware replacement, performance troubleshooting, and failure recovery.
Responsibilities:
- Operate and maintain Ceph and OpenShift Data Foundation environments.
- Monitor cluster health, capacity, latency, throughput, placement groups, and device status.
- Manage OSD and node replacement, recovery, rebalancing, backfill, scrubbing, and routine maintenance.
- Diagnose degraded placement groups, slow operations, quorum issues, device failures, and network-related storage problems.
- Plan and perform Ceph/ODF upgrades, expansion, patching, and configuration changes.
- Support block and shared-file storage through ODF, Ceph CSI, RBD, and CephFS.
- Troubleshoot persistent-volume provisioning, attachment, mounting, expansion, and performance issues.
- Maintain dashboards, alerts, runbooks, capacity plans, and recovery procedures.
- Test recovery from disk, node, service, and network failures.
- Coordinate server, disk, firmware, and network maintenance with infrastructure teams.
- Participate in production incident response and root-cause analysis.
Requirements:
- Experience developing or operating distributed-storage platforms.
- Strong Linux systems administration and troubleshooting skills.
- Strong understanding of distributed-storage concepts, including quorum, data placement, replication, failure domains, recovery, rebalancing, and block, file, or object storage.
- Experience with storage monitoring, capacity planning, upgrades, expansion, and component replacement.
- Experience benchmarking storage workloads and analysing performance bottlenecks.
- Ability to diagnose storage issues across software, Linux, networks, and physical hardware.
- Knowledge of HDD, SSD, NVMe, HBA, firmware, and storage-network considerations.
- Automation experience using Ansible, Python, shell, or similar tools.
- Working knowledge of TCP/IP, bandwidth, latency, packet loss, and network redundancy.