Data engineering - databricks sr. new (madrid)

NEORIS, now part of EPAM, is a Digital Accelerator helping companies step into the future. With more than 20 years of experience as the trusted Digital Partner of some of the world’s most recognized brands, we are a global team of 4,000+ professionals across 11 countries. Our multicultural and startup-driven culture fosters innovation, continuous learning, and the creation of high-impact solutions for our clients.

**We are looking for**:Senior Data Engineer - Azure Databricks**:
**Main Responsibilities**:

- Design, develop, and optimize end-to-end ETL/ELT data pipelines using Azure Databricks and Spark (SQL, PySpark).
- Build scalable ingestion processes for structured and semi-structured data from various sources including APIs, databases, and file systems.
- Implement complex data transformation logic and publish clean datasets to Azure Data Lake or other cloud storage for analytics consumption.
- Leverage Databricks Workflows and automation tools (e.g., Apache Airflow) for orchestrating and scheduling pipeline execution.
- Ensure best practices for performance tuning, including partitioning, caching, and efficient Spark configurations.
- Implement robust data governance and security practices using Unity Catalog and Azure-native capabilities, including encryption and RBAC.

**Requirements**:
**Mandatory**:

- +5 years of proven experience as a Data Engineer working with large-scale data processing in cloud environments.
- Expertise in building and optimizing data pipelines using **Azure Databricks**, **Spark (SQL & PySpark)**, and **Databricks Workflows**.
- Strong hands-on experience with **Azure Data Lake**, **Blob Storage**, and **Azure Synapse Analytics**.
- Solid understanding of data governance practices and tools, including **Databricks Unity Catalog**.
- Demonstrated experience in performance tuning and optimization of Spark jobs.
- Working knowledge of **infrastructure as code** tools like Terraform and monitoring using **Azure Monitor**.
- English proficiency (B2 or above).

**Nice to Have**:

- Experience with Delta Lake for managing data versioning and incremental data loads.
- Familiarity with CI/CD pipelines and DevOps practices in Azure environments.
- Relevant certifications in Microsoft Azure or Databricks.
- Experience with Apache Airflow or similar orchestration tools.

**We offer**:

- Permanent contract with a competitive salary.
- Versátil work model with remote work possibilities.
- Personalized career plan and continuous learning (certifications, English training, etc.).
- Participation in stable, technically advanced projects.
- Flexible schedule and work-life balance culture.
- Social benefits tailored to your needs.

Mercedes Manzano
LI-MM4