Senior Data Engineer - Databricks

Accountabilities:

In this role, you will own the design, development, optimization, and reliability of large-scale data pipelines that power advanced analytics and AI-driven applications. You will ensure data platforms remain secure, scalable, and efficient while collaborating with cross-functional teams to deliver high-quality solutions.

• Own production support for Databricks-based data platforms, including monitoring, alerting, incident response, and SLA management across critical data workflows.
• Design and develop high-performance data pipelines using Databricks, PySpark, SQL, and Python to ingest, transform, and deliver ERP and CRM data at scale.
• Optimize Databricks workloads by improving processing performance, reducing compute costs, and enhancing pipeline efficiency through measurable improvements.
• Lead modernization efforts by migrating legacy ETL/ELT workflows into scalable Databricks-based architectures with automated deployment and operational processes.
• Build and maintain Delta Lake architectures, including schema design, partitioning strategies, incremental processing, and data quality enforcement.
• Support customer onboarding by provisioning, validating, and securing tenant-specific data pipelines while ensuring reliable data delivery.
• Implement strong data governance practices, including access controls, security standards, compliance requirements, and privacy protections for enterprise data.
• Establish data observability practices to monitor pipeline health, ensure data integrity, and support accurate machine learning and predictive analytics outcomes.
• Partner with engineering, product, architecture, and infrastructure teams to ensure seamless integration across the broader technology ecosystem.
• Support globally distributed operations through incident management, on-call rotations, technical documentation, runbooks, and continuous improvement initiatives.
• Apply CI/CD best practices for data engineering workflows, including automated testing, version control, and reliable deployment processes.
• Contribute to technical strategy by sharing knowledge, improving engineering standards, and helping evolve platform capabilities.

Requirements:

The ideal candidate is an experienced data engineer with strong expertise in modern cloud data platforms, Databricks ecosystems, and production-scale pipeline development. You bring a combination of technical depth, problem-solving ability, and collaboration skills to build reliable data solutions in a fast-moving environment.

• 4+ years of experience in data engineering roles, building and maintaining production-grade data pipelines.
• At least 2 years of hands-on experience with Databricks and the Apache Spark ecosystem across Azure and/or AWS environments.
• Strong proficiency in PySpark, SQL, and Python, with experience delivering scalable data solutions under SLA requirements.
• Hands-on experience with Delta Lake, including schema evolution, ACID transactions, optimization strategies, and incremental or streaming processing patterns.
• Experience tuning Databricks workloads, managing compute resources, and improving production pipeline performance.
• Strong knowledge of PostgreSQL, including query optimization, schema design, and production data integration.
• Experience supporting legacy ETL technologies such as SSIS, Informatica, or custom SQL/Python-based workflows.
• Experience working with multi-tenant architectures, including tenant isolation, performance optimization, and enterprise data privacy requirements.
• Solid understanding of data governance, security, compliance, access management, and protection of sensitive business information.
• Ability to collaborate effectively with data scientists, product teams, infrastructure engineers, and enterprise architecture stakeholders.
• Experience with cloud-based data platforms, automation, and infrastructure practices is highly valuable.
• Familiarity with ML feature engineering, feature stores, or predictive analytics workflows is a plus.
• Experience with Databricks Serverless optimization, cross-cloud data access, customer onboarding automation, or Infrastructure-as-Code patterns is preferred.
• Databricks Data Engineer certification or equivalent industry certification is a plus.
• Strong ownership mindset, adaptability, curiosity, and willingness to learn new technologies in a collaborative environment.

Benefits:

• Comprehensive healthcare coverage for employees and families.
• 401(k) savings plan with employer matching contributions.
• Unlimited paid time off.
• Paid parental leave.
• Online legal services support.
• Financial planning resources.
• Discounted pet insurance options.
• Corporate benefit program offering travel, entertainment, and lifestyle discounts.
• Health and wellness reimbursement program.
• Travel discount opportunities.
• Access to educational resources and career development programs.
• Employee referral bonus program.
• Opportunities for professional growth, learning, and career advancement within a merit-based environment.
• Flexible remote work opportunities.