Senior AI Test Automation Engineer – Hybrid Madrid
We are looking for a Senior AI Test Automation Engineer to lead and evolve the test automation strategy for an advanced AI & MLOps platform. The role requires strong hands‑on expertise in Python‑based automation, design of scalable frameworks, and quality across complex systems including APIs, UIs, data pipelines, and AI/ML‑driven components such as LLMs and RAG systems.
Responsibilities
- Own the design, architecture, and evolution of the test automation framework for the AI & MLOps platform
- Define and implement scalable, reusable automation patterns (UI, API, data, BDD)
- Automate test cases using Python stack (Playwright, pytest, requests, pytest‑bdd/Behave) applying clean code practices
- Design and execute AI/ML‑specific tests (LLM evaluation, RAG validation, non‑deterministic outputs)
- Integrate and orchestrate automated tests within CI/CD pipelines (GitLab), ensuring reliable execution across environments
- Provision and manage test environments using Docker and Kubernetes
- Plan and execute regression, smoke, and release test suites across DEV, INT, UAT, and pre‑PROD
- Monitor execution quality, manage flaky tests, and ensure trustworthy test results
- Analyze defects, perform root cause analysis, and report issues with clear evidence in Jira
- Ensure full traceability from requirements to test execution and reporting
- Collaborate with cross‑functional teams (ML Engineers, MLOps, Data Engineers, Product Owners)
- Act as Test Automation Ambassador, promoting best practices and guiding teams on tooling and strategy
- Ensure compliance with test policies, audit requirements, and data privacy standards
Qualifications
- 5–7+ years of experience in test automation with strong hands‑on Python expertise
- Proven experience designing and owning automation frameworks (not just writing tests)
- Solid experience with Playwright, pytest, API testing, and BDD frameworks
- Experience integrating automation into CI/CD pipelines (GitLab or similar)
- Knowledge of Docker and basic Kubernetes concepts
- Understanding of microservices, APIs, authentication, and distributed systems
- Exposure to AI/ML systems or strong interest in testing LLM‑based and AI‑driven applications
- Experience with test reporting, parallel execution, and flaky test handling
- Familiarity with Agile/SAFe environments and DevOps practices
- Experience with Jira and test management tools (e.g., Xray)
- Strong communication, ownership, and collaboration skills
- Awareness of security, data privacy, and working in regulated environments (finance, healthcare, etc.)
- Fluent English
Work Location
Hybrid position in Madrid – 3 days at the client’s office in Sol area, next to the underground.
Work Schedule
Full‑time, 40 hours per week.
Benefits
- 23 days of annual leave plus the 24th and 31st of December as discretionary days
- Health Care Plan, teleworking compensation, life and accident insurances
- Retribución Adaptable Program (meals, kindergarten, transport, online English lessons, Health Care Plan)
- Free access to several training platforms
- Professional stability and career plans
- Referral program compensation
- Option to pick between 12 or 14 payments per year
- Real work–life balance measures (flexibility, WFH or remote work policy, compacted hours during summertime)
- Club platform discounts and gym access discounts
Equal Opportunity Statement
In UST we are committed to equal opportunities in our selection processes and do not discriminate based on race, gender, disability, age, religion, sexual orientation or nationality. We have a special commitment to disability & inclusion, so we are interested in hiring people with disability certificates.
#J-18808-Ljbffr