Description
Shield AI is a venture-backed defense-tech company with the mission of protecting service members and civilians with intelligent systems. Its products include Hivemind autonomy software, V-BAT and X-BAT aircraft, and Aechelon simulation and synthetic reality technologies.
You will build the backend services that make model development workflows reliable. This involves connecting training runs to their data, code, and configuration, managing model artifacts, and helping machine-learning engineers spend less time reconstructing what happened in an experiment.
Responsibilities
- Build experiment-tracking and model-registry services, integrating tools such as MLflow where appropriate.
- Connect exact dataset, code, and configuration versions to training runs, checkpoints, and model versions.
- Develop APIs and adapters that connect training jobs, data services, and artifact storage.
- Make interrupted or failed workflows recoverable through reliable state handling, retries, and useful diagnostics.
- Make checkpoints, model artifacts, and evaluation results easy to find and retrieve for further training, evaluation, or deployment.
- Work directly with machine-learning engineers to simplify their workflows and reduce manual record keeping.
- Deliver tested services with observability and compatible upgrades across supported cloud and customer-managed environments.
Key Outcomes
- Engineers can trace a model version back to the data, code, configuration, and run that produced it.
- Checkpoints and evaluation evidence remain available when jobs fail or workflows change.
- Developers can compare experiments and retrieve model artifacts through consistent APIs.
- Teams can reproduce results without manually reconstructing experiment history.
Requirements
- Experience building production backend services for machine-learning platforms or data-intensive workflows.
- Strong software engineering skills in Python or a comparable backend language.
- Experience designing APIs, metadata models, and persistent state for services used by other engineers.
- Experience handling storage, partial failures, retries, and consistency in distributed systems.
- Understanding of training runs, checkpoints, and model versioning.
- Ability to work directly with machine-learning users and deliver maintainable software that solves their workflow problems.
Preferred Qualifications
- Experience with MLflow, Weights & Biases, model registries, or comparable experiment-management tools.
- Experience integrating Kubernetes jobs, object storage, or machine-learning workflow systems.
- Experience with reproducibility, data provenance, or evaluation records.
- Experience supporting open-source software or on-premises and air-gapped deployments.
Why Join Us
The AI Factory is the foundation for how Shield AI develops and deploys autonomy. As a Senior Staff Engineer, you will help build the reference architecture used internally by our engineering teams and delivered to customers building their own AI capabilities. Your work will directly impact how quickly researchers can iterate, how efficiently compute resources are utilized, and how AI systems move from experimentation to operational deployment.