Automated AI Tester

Job Title: Automated AI Tester

Location : Dubai

Industry : Payment Solutions / BFSI / Retail / Healthcare

Employment Typ e: Full Time

About Kayana

Kayana is one of the fastest-growing companies, empowering small and medium-sized businesses with cutting-edge technology to compete with larger enterprises in the ordering and payments space. As a disruptive force in the industry, we provide everything from essential point-of-sale systems to advanced solutions like self-service kiosks, enabling brands to scale and succeed.

Our mission is simple : to help businesses thrive through innovative techn ology. With operations across the USA, UK, Europe, Australia, and the Middle East, Kayana is proud to foster an inclusive and dynamic culture. Every employee is supported with a personalized development plan, offering clear and achievable career paths for long-term growth.

Important Work Arrangement Note

This role requires full-time, on-site work from Monday to Saturday (6 days a week) . Candidates should be comfortable with a fully office-base d setup, as remote or hybrid work is not available for this position.

Role Overview

We are looking for an experienced Automated AI QA Tester to ensure the reliability, accuracy, safety, and performance of AI-driven ordering and payment workflows at Kayana. The ideal candidate will have 4+ years of experience in QA automation and software testing and be proficient in leveraging AI-powered testing tools and methodologies to perform end-to-end (E2E) testing across conversational AI, ordering, checkout, payment, and integration workflows

Key Responsibilities:

• Build, maintain, and scale automated test frameworks for AI-powered ordering and payment systems.
• Design conversation-based test scenarios for AI ordering assistants (menu browsing, cart building, checkout, cancellations)
• Develop AI-driven and automated E2E test suites that simulate real-world customer journeys.
• Validate payment flows (UPI, cards, wallets, failures, refunds, edge cases)
• Create synthetic user simulations and automated load scenarios to stress-test AI systems at scale.
• Evaluate AI responses for accuracy, consistency, and task completion
• Perform regression testing for AI model updates, prompt changes, and product releases.
• Identify issues in AI reasoning, API integrations, and transaction flows
• Integrate automated tests into CI/CD pipelines
• Collaborate closely with engineering, product, and AI teams to improve product quality and reliability.
• Leverage AI-assisted testing tools and frameworks to increase test coverage, automate test generation, and improve testing efficiency.

Required Skills:

• 4+ years of experience in QA automation, software testing, or test engineering.
• Strong experience designing and implementing automated E2E testing frameworks.
• Demonstrated experience using AI tools and AI-assisted testing methodologies for test creation, execution, analysis, and validation.
• Strong understanding of AI/LLM testing concepts, including Prompt testing, Hallucination detection, Response evaluation, Groundedness validation, AI safety testing &Evaluation metrics
• Experience with automation tools such as: Playwright, Selenium, Cypress & Postman
• Strong programming skills in Python and/or JavaScript.
• Experience with API testing and service integration validation.
• Familiarity with payment systems, fintech workflows, or transactional platforms.
• Ability to design structured test cases for conversational AI systems, chatbots, and AI agents.
• Experience validating multi-step user journeys across web, mobile, and backend systems.
• Strong debugging, root-cause analysis, and problem-solvi