At our unit “Data Foundation - Big Data Management,” we aim to offer organizations a robust and scalable solution for managing and deriving insights from vast quantities of data. Utilizing Azure's PaaS components, our platform streamlines the deployment and handling of Big Data workloads, empowering our clients to make data-driven decisions and propel business expansion. Our team is responsible for creating and maintaining the Big Data platform built on Azure PaaS components like EDL (Enterprise Data Lake). We collaborate closely with stakeholders to understand their needs and develop solutions that meet their expectations. The team also ensures the platform's scalability, reliability, and security, staying updated with the latest technologies and trends in the big data domain.
About the Job :
We are seeking a highly motivated and detail-oriented DataOps Engineer responsible for Data Distribution with Databricks Unity Catalog, as well as maintaining and developing CI/CD pipelines within an ambitious team.
RESPONSIBILITIES
* Design, implement, and maintain data distribution strategies using Unity Catalog with Azure Databricks and Azure Data Factory.
* Maintain and monitor data distribution across various platforms and systems with Azure Data Factory.
* Develop, extend, and optimize data distribution pipelines.
* Implement quality control measures to ensure data accuracy and consistency.
* Collaborate with the data science team to understand their data needs and provide suitable solutions.
* Troubleshoot data-related issues and optimize data distribution for performance and scalability.
* Develop and maintain technical documentation related to data distribution processes.
* Test new features and improvements in Unity Catalog, Azure Databricks, and Azure Data Factory, and recommend their implementation if beneficial.
* Communicate fluently in English (spoken and written).
QUALIFICATIONS
* Bachelor's degree in Computer Science, Information Technology, Engineering, Business, or related fields.
* At least 5 years of relevant experience as a DataOps Engineer using Azure, Python, Azure Data Factory, and Databricks.
* Experience working in an Agile environment with a DevOps mindset.
* Strong knowledge of automation and scripting with tools like Azure DevOps pipelines and Azure Data Factory.
* Deep understanding of Big Data concepts and experience with Databricks.
* Strong software engineering skills to maintain and extend Data Distribution Pipelines.
* Knowledge of Unity Catalog for Data Distribution via Databricks.
NICE TO HAVE
* Experience with event streaming, message brokers, and event-driven architectures (e.g., Kafka, RabbitMQ).
* Experience managing solutions via command line utilities and improving automation practices (DevOps, CI/CD, Terraform).
* Knowledge of integrating Data Catalog tools like Collibra with Big Data Platforms.
* Vendor certifications (e.g., Microsoft, Linux) are a plus.
* Strong interpersonal, written, and verbal communication skills, and a team-oriented attitude.
BENEFITS
* A secure work environment prioritizing health, safety, and wellbeing.
* Flexible work schedule and home-office options.
* Opportunities for learning and development.
* 23 holiday days per year plus 5 additional days (readjustment).
* Collaborative, innovative, and trustful work environment.
* International team involvement and global project participation.
* Relocation assistance to Madrid.
* Opportunity to develop and support custom business intelligence software across various topics.
Join BASF — where the chemistry is right.
We foster innovative solutions, sustainable actions, and connected thinking. Be part of our success story, develop your future with us, and thrive in a diverse, inclusive global team.
At BASF, our total compensation package goes beyond salary and includes elements to support your personal and professional growth. Click here to learn more.
J-18808-Ljbffr
#J-18808-Ljbffr