All positions
Quality & EvaluationTokyo · Hybrid

QA Engineer, AI Systems

Test the AI systems we ship for clients, from functional bugs to the subtle failures unique to LLM-based products, using Ledger, our evaluation platform.

Team
Quality & Evaluation
Location
Tokyo · Hybrid
Employment
Full-time
Experience
2+ years
Languages
Business-level English
Apply for this role

What you’ll do

  • Design and run test plans for AI agents, copilots and document or voice systems before and after release.
  • Build and maintain evaluation sets and automated checks in Ledger to catch regressions in model behaviour.
  • Investigate failures and edge cases, from UI bugs to incorrect or unsafe model outputs, and file clear reports engineers can act on.
  • Work with engineers and clients to define what "good" looks like for a system, and track it over time.

What we’re looking for

  • 2+ years in software QA or test engineering, including some test automation.
  • Comfortable reading logs, prompts and model outputs to figure out why something went wrong.
  • Clear, structured bug reports, and a habit of verifying fixes rather than just filing issues.
  • Basic scripting ability (Python or TypeScript) to write test cases and evaluation checks.

Nice to have

  • Experience testing LLM-based or other AI-powered products.
  • Familiarity with evaluation frameworks or LLM-as-judge techniques.
  • Business-level Japanese.

Hiring process

Four rounds, and a clear answer at every step.

Interviews are held online, in English or Japanese.

  1. Start

    Apply

    Email your resume and a cover letter to career@kontextly.ai. We read every application and reply within 5 business days.

    1. Round 130 minutes

      Initial screen

      A short AI interview covering your background, experience and motivation, so we can quickly find a good match on both sides.

    2. Round 290 minutes

      Testing exercise

      Given a sample AI feature, find and document its bugs and edge cases, including a few that are specific to LLM-based behaviour. Use whatever tools you normally use.

    3. Round 360–90 minutes

      Technical interview

      A conversation with two engineers. We review your coding exercise, work through a system design problem taken from a real project, and discuss work you are proud of.

    4. Round 445 minutes

      HR interview

      Meet HR and a member of the leadership team to talk about motivation, working style, expectations, salary and start date.

  2. Finish

    Offer

    We aim to decide within a week of the final round. The whole process usually takes 3–4 weeks.

Apply

Apply for QA Engineer.

Email your resume and a cover letter. It takes about two minutes.

How to apply

Email your resume and a short cover letter, both as PDF, to career@kontextly.ai, with “QA Engineer, AI Systems” in the subject line. We read every application and reply within 5 business days.