OverviewKaseya is the leading provider of complete IT infrastructure and security management solutions for Managed Service Providers (MSPs) and internal IT organizations worldwide powered by AI. Kaseya’s technologies allow organizations to efficiently manage and secure IT to drive sustained business success. Founded in 2000, Kaseya serves customers in over 20 countries and manages over 15 million endpoints worldwide. For more information, visit www.Kaseya.com.Join the dynamic team at Kaseya as a Staff AI QA Engineer. Ensure the reliability, safety, and ethical integrity of our AI platform by crafting advanced testing strategies and automation frameworks tailored for adaptive, large-scale agentic systems and complex integrations. You’ll validate AI behavior, secure data pipelines, and enforce quality standards across distributed systems, helping ensure our intelligent agents operate with precision, fairness, and resilience in real-world environments.What we are looking forWe’re looking for a quality-focused engineer who can think critically about testing complex, adaptive systems. You’re a strong communicator who can clearly articulate quality risks and testing strategies to both technical and non-technical stakeholders. You thrive in collaborative environments, working closely with developers, data scientists, and product managers, and you bring leadership in QA best practices, mentoring, and ethical testing.Essential duties and responsibilitiesDesign and implement comprehensive test strategies for AI agents, platform services, and integrationsDevelop and maintain automated test frameworks for functional, performance, and security testingValidate AI model behavior, including prompt outputs, bias detection, and explainabilityConduct rigorous testing of distributed systems, microservices, and real-time data pipelinesEnsure data quality and integrity across training, inference, and integration workflowsCollaborate with engineering and research teams to define quality gates and ethical testing standardsDrive continuous testing practices in agile development cyclesWhat you’ll bring5–7+ years of experience in QA or Test EngineeringMastery of test automation frameworks (e.g., PyTest, Playwright, Selenium, RestAssured)Strong API testing skills (manual and automated)Experience with performance, load, and high-volume stress testing (e.g., JMeter, k6s, Gatling)Deep understanding of AI/ML model evaluation metrics and methodologiesExperience testing distributed systems and microservices architecturesProficiency in data quality validation for AI training and inference workflowsExperience with contract testing for microservicesAbility to develop test strategies for non-deterministic and adaptive behaviors in AI systems (e.g., prompt validation, guardrails, bias detection)Familiarity with AI model testing tools (e.g., CheckList, Giskard, Deepchecks)Experience with test planning, execution, and reporting in agile environmentsBachelor’s degree in Computer Science, Software Engineering, or a related fieldPreferred: Certifications in QA Automation, AI/ML Testing, or Ethical AI (e.g., ISTQB, Certified Ethical Emerging Technologist)Additional training in AI safety, fairness, or model validation is a plusNice to HaveExperience with AI red teaming and adversarial testing techniquesExplainability testing and validation for XAI methods (e.g., LIME, SHAP)Usability testing for AI-driven UIs and agent interactionsFamiliarity with data privacy testing techniques (e.g., differential privacy validation)Security testing for AI models (e.g., model inversion, membership inference, poisoning attacks)Experience testing AI ethics, fairness, and biasKnowledge of formal verification methods for AIExperience building synthetic data generation pipelines for testingAdditional InformationKaseya provides equal employment opportunity to all employees and applicants without regard to race, religion, age, ancestry, gender, sex, sexual orientation, national origin, citizenship status, disability, veteran status, marital status, or any other characteristic protected by applicable law.Seniority levelMid-Senior levelEmployment typeFull-timeJob functionEngineering and Information TechnologyIndustriesSoftware Development
#J-18808-Ljbffr