GitHub - garg-ankush/scipe: SCIPE is a powerful tool for evaluating and diagnosing LLM (Large Language Model) graphs or chains.
llm-evaluationllm-chainsroot-cause-analysisdebugging
Abstraction: Python tool for root-cause diagnosis of failing LLM chain nodes
Key points:
- SCIPE (
pip install scipe) identifies the most impactful failing node in a LangGraph-based LLM chain using LLM-as-judge evaluation - Distinguishes independent failures (node itself causes the problem) from dependent failures (propagated from upstream nodes)
- Algorithm traverses from the most downstream node upstream, computing conditional failure probabilities, stopping at the root cause
- Requires no ground-truth labels — only input/output pairs for each node and a compiled LangGraph structure
- Configuration specifies model (e.g.,
claude-3-haiku-20240307) via LiteLLM, node-to-column mappings, and output path - Output is an
EvaluationResultwithroot_cause,debug_path, and per-node failure probabilities as JSON
Connections: Langchain · Claude · LLM Evaluation · AI Agents