About Us
Zylon (https://www.zylon.ai/) transforms company knowledge into immediate productivity gains through a secure, private AI workspace. Our on-premise deployment ensures complete data sovereignty, cost control, and full customization by running local AI models with no vendor lock-in.
As creators and maintainers of the popular open-source project PrivateGPT (https://privategpt.dev/) with more than 55K Github stars, we are committed to leveraging the latest AI technologies to drive innovation and deliver exceptional value to our customers.
As an early-stage startup serving clients in financial, manufacturing, engineering, and other regulated industries, we're looking for talented individuals to help drive our growth in a company where we celebrate diversity and are committed to creating an inclusive environment for all employees.
Role Overview
We're seeking an experienced AI Engineer to join our team and play a crucial role in developing and enhancing our Private AI platform. You'll work on the whole AI stack, ranging from the GPU management and inference server, to leveraging and optimizing open-source AI models for on-premise bare metal deployments where we have complete control of the data pipeline and the agentic system running on top of it. All of this while ensuring high performance and security for our enterprise clients in regulated industries.
Key Responsibilities
* Design, develop, and deploy AI systems that can run efficiently in on-premise environments with local models, without depending on 3rd parties like OpenAI, Amazon Bedrock, etc.
* Optimize Small Language Models (SLMs) for private deployments with attention to performance and resource constraints.
* Contribute to the architecture of our AI platform, ensuring scalability and security.
* Implement advanced prompt engineering techniques and secure data processing pipelines for knowledge extraction and transformation.
* Research, design and implement agentic strategies to improve AI model interaction and user experience.
* Stay current with the latest advancements in GenAI research and identify opportunities for bringing additional value to our product, as well as developing within the team a deep understanding of the capabilities, limitations, and availability of the different AI models.
* Work together with the Product & Engineering teams, actively participating not only in the implementation but also in the product definition, prioritization, scoping, and validation.
* Work closely with client success teams to troubleshoot and resolve technical issues.
Requirements
* 5+ years of experience in software or/and AI engineering roles with special focus on GenAI for the last 2 years
* Experience with on-premise deployment of AI systems.
* Strong background in integrating and fine-tuning open-source LLMs
* Proficient in Python and relevant ML/AI frameworks (PyTorch, LlamaIndex, LangChain, etc.)
* Familiarity with Agents strategies and agentic frameworks, as well as different RAG implementations
* Experience with vector databases and efficient retrieval systems
* Knowledge of containerization and deployment technologies (Docker, Kubernetes)
* Strong problem-solving skills and attention to detail
* Excellent English communication skills and ability to collaborate effectively with cross-functional teams
* Eager to be up-to-date with new techniques, models and advances within the GenAI field
* Lean and creative mindset, ready to iterate fast, learn and adapt
* Customer-focused mindset
Nice to Have
* Experience working with vLLM + NVIDIA Triton architecture
* Understanding of NVIDIA drivers and CUDA platform to power NVIDIA GPUs for AI
* Knowledge of evaluation and observability frameworks (LangSmith, Opik, Arize Phoenix, Ragas, etc.) for GenAI
* Knowledge of Model Context Protocol (MCP) for building agentic systems.
* Experience with model quantization and optimization techniques
* Experience in startups or early-stage companies
* Background in working with financial, manufacturing, or engineering domains
What We Offer
* Opportunity to shape the future of private AI in regulated industries
* Competitive salary and equity package
* Work with cutting-edge AI technologies
* Direct impact on product development and company growth
* Collaborative team environment focused on innovation
* Full-remote position, within Europe
* Flexible schedule (40h/week)
* 23 days of PTO
* Necessary equipment to work
* Periodic on-site team building events