Overview:
We are seeking a highly skilled and experienced Senior Data Engineer with a minimum of 5 years working with Databricks and Lakehouse architecture to join our team in the consumer finance industry. The successful candidate will play a critical role in mapping requirements, designing, implementing, and maintaining scalable data solutions, and will ensure seamless integration and operation of CI/CD pipelines.
Key Responsibilities:
* Design, develop, and optimize robust data architectures using Databricks and Lakehouse principles to support large-scale and complex data analytics needs.
* Implement and maintain CI/CD pipelines to ensure continuous integration and delivery of data solutions, ensuring data quality and operational efficiency.
* Collaborate with cross-functional teams (ideally Finance) to understand data requirements, map data through distributed systems, and translate these into technical solutions that align with business objectives.
* Manage and optimize data storage and retrieval systems to ensure performance and cost-effectiveness.
* Develop, maintain, and document ETL/ELT processes for data ingestion, transformation, and loading using industry best practices.
* Ensure data security and compliance, particularly within the context of financial data, adhering to relevant regulations and standards.
* Troubleshoot and resolve any data-related issues, ensuring high availability and reliability of data systems.
* Evaluate and incorporate new technologies and tools to improve data engineering practices and productivity.
* Mentor junior data engineers and provide technical guidance to the wider team.
* Contribute to the strategic planning of data architecture and infrastructure.
Required Qualifications and Experience:
* Bachelor’s degree in Computer Science, Information Technology, or a related field. A Master’s degree is a plus.
* Minimum of 5 years of professional experience as a Data Engineer or in a similar role within the finance industry, demonstrating experience of working with consumer finance data models.
* Proficient in using Databricks for data engineering and analytics.
* Strong experience with Lakehouse architecture and its optimization.
* Highly proficient in programming languages such as Python
* Demonstrable expertise in implementing and managing CI/CD pipelines for data solutions
* Solid experience with cloud platforms (e.G., AWS, Azure, or GCP), and their data services.
* Deep understanding of data warehousing concepts and technologies (e.G., Snowflake, Redshift).
* Strong knowledge of ETL/ELT processes and tools.
* Solid experience of utilising PowerBI or similar visualisation tools
* Experience working with big data technologies and frameworks (e.G., Spark)
* Excellent problem-solving skills and a proactive approach to data engineering challenges.
* Strong communication skills with the ability to articulate complex technical concepts to non-technical stakeholders.
Desirable Skills:
* Certifications in Databricks or cloud technologies.
* Experience with machine learning pipelines and model deployment.
* Knowledge of regulatory requirements in the finance industry, such as GDPR or PCI-DSS.
* Experience with agile development methodologies, such as Scrum or Kanban.