Description
Electronic Arts creates exceptional entertainment experiences that inspire players and fans worldwide. Here, everyone is part of the story – a large international community where creativity flourishes, new perspectives are welcomed, and ideas matter. At EA, we believe games are powerful because they bring together multiple ways people engage: play, watch, create, and connect.
As a Senior Machine Learning Data Developer, you will own the systems that prepare and deliver multimodal game data for large-scale model training. You will set the architecture for these training input pipelines and improve data throughput, accelerator utilisation, reproducibility, and recovery from failures.
Your responsibilities will initially focus on hands-on individual contribution and technical leadership, with potential to take on people-management responsibilities as the team grows. You will partner with researchers and central data and compute teams to turn evolving model requirements into reusable training capabilities.
Responsibilities:
- Architect and build shared training input pipelines from certified game capture to accelerators, covering storage layout, decoding, sharding, sampling, packing, batching, caching, prefetching, and host-to-device delivery across multiple training frameworks.
- Define schemas, data contracts, encodings, compression, and standards for dataset versioning, lineage, quality, coverage, acceptance criteria, and reproducible splits.
- Build deterministic loaders and checkpoint-safe iteration that recover from worker or hardware failures and corrupt data, including elastic restarts and changes to cluster size or parallelism.
- Design reproducible dataset mixtures and balanced batches across modalities, resolutions, sequence lengths, titles, and capture conditions, accounting for worker/rank locality and distributed parallelism.
- Benchmark and enforce throughput, data-wait time, step time, and accelerator utilisation and stalls. Model FLOPs utilisation (MFU) where applicable, per-rank skew, scaling efficiency, restart time, reliability, and cost targets through thousands of accelerators.
- Diagnose bottlenecks and intermittent failures across storage, network I/O, CPU, memory, decoding, preprocessing, and accelerators, including stragglers, faulty hardware, corrupt samples, and imbalanced batches.
- Integrate compilers, runtimes, optimised kernels, low-precision training, and hardware-accelerated media processing. Maintain efficient data delivery and recovery under FSDP/HSDP, tensor, context, and pipeline parallelism across heterogeneous accelerator environments.
- Lead training-pipeline architecture reviews, mentor engineers, and align priorities with research leads. Translate model and data-access requirements into scalable systems building on shared infrastructure services, while researchers retain model-research ownership.
Qualifications:
- 20+ years of development experience, including owning production systems for distributed-training input, with measurable improvements in throughput, accelerator utilisation, step time, reliability, recovery time, or cost.
- Expertise building custom PyTorch training and ingestion systems or adapting TorchTitan, JAX/XLA with MaxText, Megatron Core/NeMo, or equivalent stacks for single jobs spanning thousands of accelerators.
- Expertise in FSDP2/HSDP, tensor/context/pipeline parallelism, mixed-precision training, and optimisation on one or more accelerator platforms such as GPUs, TPUs, or AWS Trainium.
- Pre-training experience and an understanding of how model architecture, data representation, batching, and orchestration affect training-system design and efficiency.
- Expertise building large-scale multimodal streaming or hardware-accelerated ingestion with Ray Data, Mosaic Streaming, WebDataset, NVIDIA DALI, NVDEC, or equivalent technologies.
- Deep knowledge of sharding, sampling, packing, caching, prefetching, deterministic data mixing, checkpoint-safe iteration, and elastic restart; experience with schemas, dataset versioning, lineage, quality, and reproducible training/evaluation.
- Experience profiling I/O, CPU, memory, network, distributed workers, and accelerators, including diagnosing cross-layer failures and intermittent faults.
Salary:
- British Columbia: $204,600 - $287,500 CAD
- California: $227,300 - $344,700 USD
Benefits: We offer a comprehensive benefits package, including:
- Medical, dental, and vision insurance
- Life insurance and disability insurance
- Retirement plan
- Paid time off and sick leave
- Opportunities for bonuses and other incentive programs