Automation Test Engineer (AI & Chatbot Testing)

Automation Test Engineer (AI & Chatbot Testing)

Location: UAE

Job Summary

We are looking for an experienced Automation Test Engineer to test and improve AI-powered customer support and troubleshooting solutions. The role focuses on test automation, API testing, AI quality validation, and performance testing to ensure reliable and accurate customer experiences.

Key Responsibilities

AI & Chatbot Testing

• Develop automated tests for AI chatbots and RAG (Retrieval-Augmented Generation) systems.
• Validate response accuracy, relevance, and AI performance.
• Measure AI quality metrics such as hallucination rates and answer relevance.
• Perform regression testing when AI models or knowledge bases are updated.

Automation & Functional Testing

• Automate testing of troubleshooting workflows and decision trees.
• Validate end-to-end customer support scenarios.
• Simulate customer interactions and verify expected outcomes.

API & Integration Testing

• Build and execute automated API tests.
• Validate integrations between AI platforms and backend systems.
• Create mock services and test data where required.

Performance & Security Testing

• Conduct load and performance testing of conversational AI systems.
• Test AI guardrails, prompt injection risks, and unsafe responses.
• Identify performance and reliability issues.

CI/CD & Quality Assurance

• Integrate automated tests into Azure DevOps pipelines.
• Track quality metrics and generate test reports.
• Work closely with developers, AI engineers, and product teams.

Requirements

• 4+ years of experience in QA Automation or SDET roles.
• 1+ year of experience testing AI, Chatbots, LLMs, or Agentic AI solutions.
• Strong programming skills in Python, TypeScript, or C#.
• Experience with REST APIs, GraphQL, WebSockets, Postman, and WireMock.
• Experience with Azure DevOps and CI/CD pipelines.
• Knowledge of RAG, semantic search, and vector databases.
• Experience with MQTT or IoT telemetry is a plus.
• Exposure to embedded systems or consumer electronics is preferred.