PHOENIX-ARIZE
Phoenix Arize OSS LLM observability OpenTelemetry.
Definition
Phoenix Arize key features: (1) Tracing: OpenTelemetry-based instrumentation via auto-instrumentation libraries (openinference-instrumentation-openai + openinference-instrumentation-langchain + openinference-instrumentation-llama-index + etc.), traces collected by Phoenix backend (local + SaaS), web UI shows hierarchical traces. (2) OpenInference standard: semantic conventions for LLM operations OpenTelemetry, standardize trace attributes for cross-tool compatibility (Phoenix + Langfuse + LangSmith + DataDog + custom tools all consume same trace format), Arize AI leadership donating OpenInference to CNCF/OpenTelemetry. (3) Evaluations: LLM-as-Judge built-in templates (correctness + relevance + toxicity + hallucination + custom criteria), Python evaluators, Pairwise evaluators. (4) Datasets: manage curated test datasets, import production traces -> dataset for evaluation. (5) Experiments: track experiment runs with metrics across datasets versions, comparison side-by-side. (6) Prompt Playground: iterate prompts side-by-side LLMs (OpenAI + Anthropic + Mistral + Cohere). (7) Embedding analysis: visualise embedding spaces (UMAP + t-SNE) detect drift + clusters, unique strength Phoenix vs other tools. (8) Self-hosted: Phoenix OSS Docker + Helm Kubernetes deploy. Customers: Tinder + Toast + Cresta + IBM + Notion AI ~10000+ deployments 2024.
Origin
Arize AI founded 2020 in Mill Valley California by Jason Lopatecki (ex-Uber Eats) + Aparna Dhinakaran (ex-Tubi Engineering) ; Series A $19M 2020 + Series B $38M 2022 + Series C $70M 2024 (TCV lead) ; Phoenix OSS launched 2022 ; ~10000+ Phoenix deployments 2024.
Example in context
Notion AI uses Phoenix Arize for AI features observability: OpenInference instrumentation traces millions LLM calls/day across Notion AI features (writing assist + Q&A + summarization), embedding analysis Phoenix UMAP visualises user queries clusters identify product opportunities + drift detection model performance degradation.
Related terms
- LangSmith — alternative LLM observability.