Overview
Senior AI Agent Engineer - Voice AI at Zendesk. The role focuses on building intelligent, autonomous voice agents that can listen, understand, and speak with human-like fluency. You will design the cognitive architecture for voice applications, enabling reasoning, planning, and execution of complex tasks through low-latency spoken dialogue. You will also communicate complex technical concepts to both technical and non-technical stakeholders.
What You Will Do
* Design and develop robust, stateful, and scalable voice-first AI agents using Python, optimized for real-time voice interactions, managing turn-taking, interruptions, and low-latency responses.
* Integrate real-time Speech-to-Text (STT), Text-to-Speech (TTS), and Voice Activity Detection (VAD) services to create a seamless conversational flow.
* Connect voice agents with existing enterprise systems, databases, and third-party APIs to create powerful, end-to-end automated workflows initiated and managed through voice.
* Establish and own evaluations for voice agent performance and behavior, iterating to improve performance, reliability, and user experience.
* Build end-to-end conversational flows with reasoning, planning, and dynamic tool use beyond pre-scripted voice experiences.
* Collaborate with product managers, ML scientists, and engineers to deeply understand user needs and voice interaction goals.
* Implement fallback, recovery, and error-handling strategies to address noisy audio input or speech recognition inaccuracies.
* Define and track voice-specific evaluation metrics (e.g., word error rate, latency, conversational naturalness).
* Develop observability tools and guardrails to monitor performance, ensure safety, and handle edge cases in spoken interactions.
* Document development, architecture decisions, and research findings to share knowledge across the team.
Qualifications
* LLM-Oriented System Design: Experience building multi-step, tool-using agents (LangChain, Autogen). Familiar with prompt engineering, context management, and reasoning strategies like Chain-of-Thought and ReAct.
* Voice AI Expertise:
o Experience building low-latency, streaming voice applications. Proficient with real-time STT/TTS models and APIs. Expertise in Voice Activity Detection (VAD), noise suppression, and interruption logic.
o Experience with third-party voice AI APIs (STT/TTS) from providers like OpenAI, Deepgram, ElevenLabs, etc.
o Understanding of latency, timing, and streaming audio constraints.
* Tool Integration & APIs: Comfortable connecting agents to external APIs, tools, and databases in secure environments.
* RAG (Retrieval-Augmented Generation): Building pipelines with vector stores, chunking strategies, and hybrid retrieval.
* Evaluation & Observability: Experience with monitoring tools and evaluation frameworks to score AI agents.
* Safety & Reliability: Familiarity with prompt-injection defenses, guardrails, and failover logic.
* Performance Optimization: Token budgeting and latency management using caching, model routing, etc.
* Programming & Deployment: Proficient in Python and FastAPI. Experience deploying AI apps to cloud platforms (AWS, GCP, Azure) with CI/CD.
Nice-to-have
* M.S. / Ph.D. in Computer Science, NLP, ML, or related field.
* Background in spoken dialogue systems or conversational UX design.
* Familiarity with real-time streaming architectures (e.g., WebRTC, gRPC, socket.io).
* Multilingual ASR/TTS pipeline experience.
About Zendesk
Zendesk builds software for better customer relationships. It empowers organizations to improve customer engagement and understand their customers. Zendesk products are easy to use and implement, giving organizations flexibility to move quickly, focus on innovation, and scale with growth. Zendesk is an equal opportunity employer with diversity and inclusion initiatives across the workplace. We provide reasonable accommodations for applicants with disabilities where required.
Seniority level
* Mid-Senior level
Employment type
* Full-time
Job function
* Sales and Business Development
* Industries: Software Development
#J-18808-Ljbffr