Data Engineer – Databricks

Why Join AspenView?

At AspenView, we’re more than a nearshore IT partner—we’re a people-first, purpose-driven company that believes great culture drives great outcomes. We’re passionate about connecting talent and technology to deliver measurable value for clients—and meaningful career paths for our people.

Here’s what you can expect:

• Competitive base
• Comprehensive benefits and wellness support
• Flexible work model: hybrid, remote, or in-office
• Real growth opportunities and leadership visibility
• Inclusive, respectful culture that blends U.S. innovation with Colombian heart
• A company that listens, invests in you, and celebrates wins together

About the Role

We are seeking a skilled Data Engineer – Databricks to design, build, and optimize high-throughput, production-grade data pipelines on the Databricks Lakehouse platform. This is a full-time, remote position open to mid-senior and senior candidates across Latin America.

In this role, you will implement Medallion Architecture patterns (Bronze, Silver, Gold layers) to process structured and unstructured healthcare data—including Electronic Health Records (EHR/Epic), FHIR streams, claims data, and clinical research datasets. You will work closely with data architects, clinical analysts, and cloud engineers to build secure, HIPAA-compliant ETL/ELT pipelines using PySpark, Delta Lake, and Databricks Workflows.

What You Will Do

• Lakehouse Pipeline Engineering: Design, develop, and maintain automated ETL/ELT data pipelines on Databricks utilizing PySpark, Delta Lake, and Databricks Workflows or Delta Live Tables (DLT).
• Data Architecture Implementation: Implement Medallion Architecture principles to clean, transform, and aggregate raw clinical/operational datasets into analytics-ready models for BI reporting and predictive analytics.
• Healthcare Data Integration: Ingest and normalize healthcare data sources, including Epic EHR databases (Clarity/Caboodle), FHIR/HL7 data feeds, and third-party claims systems.
• Data Governance & Security: Enforce row/column-level access control and data lineage using Databricks Unity Catalog, ensuring strict compliance with HIPAA, HITRUST, and internal data governance protocols for Protected Health Information (PHI).
• Performance Tuning & Optimization: Optimize compute cluster utilization, SQL query performance, Delta table indexing (Z-Ordering/Partitioning), and memory management to control cloud infrastructure costs and lower pipeline latency.
• Cross-Functional Collaboration: Partner with US-based data scientists, business intelligence developers, and clinical stakeholders to understand data requirements and deliver reliable data products.

What You Bring

Experience

• 4+ years of production experience in data engineering, with at least 2+ years of hands-on expertise building scalable solutions on Databricks.

Technical Expertise

• Databricks & Lakehouse Ecosystem: Strong mastery of PySpark, Delta Lake, Databricks SQL, and Unity Catalog.
• Programming & Querying: Advanced proficiency in Python and SQL for complex data manipulation and performance optimization.
• Cloud Infrastructure: Hands-on experience with managed cloud data storage and computing services (AWS S3 / EMR or Azure ADLS Gen2 / Data Factory).
• Healthcare Interoperability (Preferred): Familiarity with data standards (FHIR, HL7, OMOP, ICD-10/CPT codes) or EHR data structures (Epic/Cerner).
• DevOps & Quality: Experience with Git, CI/CD deployment pipelines, and data validation frameworks (e.g., Great Expectations).

Language Proficiency

• Advanced/Fluent English (C1/C2) for daily active collaboration with Boston-based technology and analytics teams.

Soft Skills & Competencies

• Analytical Rigor: Uncompromising attention to data quality, consistency, and structural integrity.
• Mission-Driven Alignment: Genuine passion for applying technology and data engineering to improve healthcare outcomes and health equity.
• Autonomous Problem Solver: Proactive approach to diagnosing pipeline failures, identifying performance bottlenecks, and recommending architecture improvements.
• Clear Technical Communication: Ability to articulate data architecture choices and pipeline designs clearly to technical peers and non-technical healthcare stakeholders.