Distributed Training Engineer for Large Language Models - Remote

About the Role

We are looking for a Distributed Training Engineer for Large Language Models to join our innovative team at Poolside. This remote position offers an exciting opportunity to work on cutting-edge AI technologies and contribute to the development of Artificial General Intelligence (AGI). As a Distributed Training Engineer, you will play a vital role in building and optimizing our distributed training and inference systems for Large Language Models (LLMs).

What You'll Do

• Identify, study, and troubleshoot hardware problems during training at scale.
• Minimize GPU idle time during faults, both operationally and strategically.
• Design and develop tools and add-ons to accelerate training recovery.
• Improve the performance and reliability of checkpointing.
• Write high-quality code in Python (PyTorch), Cython, C/C++, and CUDA API.

Requirements

• Understanding of Large Language Models (LLM).
• Basic knowledge of Transformers and deep learning fundamentals.
• Strong engineering background with programming experience in Linux API and Linux kernel.
• Proficient in Python with libraries such as numpy, PyTorch, or Jax.
• Knowledge of NCCL and distributed systems.
• Strong algorithmic skills and critical thinking.

Nice to Have

• Experience with fault tolerance and observability.
• Familiarity with Kubernetes (K8s) stack.

What We Offer

• Fully remote work with flexible hours.
• 37 days/year of vacation and holidays.
• Health insurance allowance for you and your dependents.
• Company-provided equipment.
• Well-being, learning, and home office allowances.
• Frequent team get-togethers.
• A diverse and inclusive people-first culture.