QA Engineer, AI Systems
Test the AI systems we ship for clients, from functional bugs to the subtle failures unique to LLM-based products, using Ledger, our evaluation platform.
- Team
- Quality & Evaluation
- Location
- Tokyo · Hybrid
- Employment
- Full-time
- Experience
- 2+ years
- Languages
- Business-level English
What you’ll do
- Design and run test plans for AI agents, copilots and document or voice systems before and after release.
- Build and maintain evaluation sets and automated checks in Ledger to catch regressions in model behaviour.
- Investigate failures and edge cases, from UI bugs to incorrect or unsafe model outputs, and file clear reports engineers can act on.
- Work with engineers and clients to define what "good" looks like for a system, and track it over time.
What we’re looking for
- 2+ years in software QA or test engineering, including some test automation.
- Comfortable reading logs, prompts and model outputs to figure out why something went wrong.
- Clear, structured bug reports, and a habit of verifying fixes rather than just filing issues.
- Basic scripting ability (Python or TypeScript) to write test cases and evaluation checks.
Nice to have
- Experience testing LLM-based or other AI-powered products.
- Familiarity with evaluation frameworks or LLM-as-judge techniques.
- Business-level Japanese.
Hiring process
Four rounds, and a clear answer at every step.
Interviews are held online, in English or Japanese.
- Start
Apply
Email your resume and a cover letter to career@kontextly.ai. We read every application and reply within 5 business days.
- Round 130 minutes
Initial screen
A short AI interview covering your background, experience and motivation, so we can quickly find a good match on both sides.
- Round 290 minutes
Testing exercise
Given a sample AI feature, find and document its bugs and edge cases, including a few that are specific to LLM-based behaviour. Use whatever tools you normally use.
- Round 360–90 minutes
Technical interview
A conversation with two engineers. We review your coding exercise, work through a system design problem taken from a real project, and discuss work you are proud of.
- Round 445 minutes
HR interview
Meet HR and a member of the leadership team to talk about motivation, working style, expectations, salary and start date.
- Finish
Offer
We aim to decide within a week of the final round. The whole process usually takes 3–4 weeks.
Apply
Apply for QA Engineer.
Email your resume and a cover letter. It takes about two minutes.
How to apply
Email your resume and a short cover letter, both as PDF, to career@kontextly.ai, with “QA Engineer, AI Systems” in the subject line. We read every application and reply within 5 business days.