Senior Software Engineer - SparkFlow Framework (Remote)

About the Role

We are seeking a Senior Software Engineer - SparkFlow Framework (Remote) to join our dynamic team. In this role, you will contribute to the SparkFlow enterprise data processing framework built on Apache Spark, focusing on implementing new features and enhancing existing components. This position offers a unique opportunity to work remotely while being part of an innovative project that delivers AI-enabled capabilities to simplify framework usage.

What You'll Do

• Build and enhance functional features in the SparkFlow framework, including sources, targets, transformations, governance, and control capabilities.
• Implement and refine framework extension points such as APIs, configurations, and libraries to improve composability and reuse.
• Improve developer experience by simplifying configuration patterns, reducing onboarding friction, and enhancing diagnostics and observability hooks.
• Develop AI-enabled solutions that assist developers, such as guided configuration generation, validation tools, and troubleshooting accelerators.
• Contribute to Unity control plane integration by implementing adapters, operators, automation, and integration testing.
• Participate in code reviews, design discussions, and operational support activities as needed.

Requirements

• Strong hands-on engineering experience with Apache Spark using Scala and/or Java (Python is a plus), including Spark SQL-based processing.
• Experience building frameworks or libraries, including API and abstraction design.
• Working knowledge of CI/CD and engineering fundamentals, including Git, build tooling, and unit/integration testing.
• Experience with enterprise data ecosystem components such as Hadoop/Hive, Kafka, and cloud storage or warehouse patterns.
• Strong problem-solving skills and the ability to work collaboratively in a team environment.

Nice to Have

• Experience improving pipeline onboarding and deployment patterns, such as config-driven artifacts, launcher scripts, and scheduler integration.
• Familiarity with governance capabilities, including audit trail capture, metadata or lineage integration, and data-in-motion controls.
• Cloud or hybrid experience, such as Google Cloud Platform Dataproc patterns supporting Spark workloads.

What We Offer

• Competitive salary range of $120,000 - $150,000 per year.
• Visa sponsorship available for qualified candidates.
• Flexible remote work options with a hybrid model (3 days on-site, 2 days remote).
• Opportunities for professional development and career growth.
• Collaborative and innovative work environment.