Dice is the leading career destination for tech experts at every stage of their careers. Our client, TekLeaders, Inc, is seeking the following. Apply via Dice today!

AI Observability engineer

Location: remote contract

AI Observability Engineer Tasks

Design and implement endtoend observability for AI agents, models, MCPs, and data pipelines
Instrument agents for traces, metrics, and logs covering prompts, tool calls, responses, latency, errors, and cost
Monitor agent behavior, reliability, and performance across single and multiagent systems
Build and operate an evaluation framework (offline + continuous) for agentic systems
Define offline golden test suites, regression sets, and scenariobased evaluations
Implement continuous, inproduction evaluations to detect quality and safety drift with alerts and thresholds\
Implement AI quality and safety metrics (hallucination rate, grounding accuracy, tool success rate, confidence scores)
Detect and alert on model drift, data drift, and concept drift impacting agent outcomes
Implement HumanintheLoop (HITL) review workflows for approvalgated agent actions
Enforce and log approvals for sensitive or highrisk tool actions
Define HITL triggers using confidence thresholds, escalation policies, and reviewer queues
Feed human feedback back into prompt updates, retrieval tuning, and agent policy improvements
Instrument MCPs for request/response observability and correlate MCP telemetry with agent traces
Integrate observability and evaluation checks into CI/CD pipelines to enable safe rollout, canarying, and rollback
Build dashboards and alerts for agent health, quality, safety, and usage trends
Ensure security, privacy, and compliance observability, including PII detection and audit logging
Optimize observability cost and performance across logs, metrics, traces, and evaluation runs
Experience implementing AI observability using AWS cloud services and opensource tooling