SkillSpec makes agent skills followable, testable, and provable with Doctor risk reports, guided imports, structured contracts, and alignment proof.

Operational doctrine for practical AI systems design.

Benchmarking Open-Ended Inference Optimization by AI Agents

Evaluation Infrastructure for AI Agents

Core engine behind Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

Learn to evaluate AI products for production — 21 hands-on lessons on evals, metrics, fairness, agents, red teaming, and release decisions for working PMs.

Agent skill for AI agent development

Collection of frameworks and tools for AI evalations, including tool-use, agentic AI, MCP, and multimodal

Governed multi-LLM Responsible AI control plane for prior-authorization decision support, with PHI-safe audit trails, HITL review, evidence packets, and deterministic governance evals.

Eval-first AI agent that triages property maintenance emails. The real work is the eval system around it: trace-driven error analysis, code graders and validated LLM-as-judge (TPR/TNR), component and end-to-end evals, a failure taxonomy, and a CI regression gate. LangGraph, FastAPI, Langfuse.

AI agent workspace architecture, demonstrated end-to-end in Claude Code: roles library, persistent memory, hooks, scheduled agents, self-audits, loop selection, and measurement-gated self-improvement. Interactive tour, fork-ready samples.

Benchmark LLM jailbreak resilience across providers with standardized tests, adversarial mode, rich analytics, and a clean Web UI.

Open-source toolkit for assessing whether an AI workflow is ready for production: governance, RAG quality, evals, observability, human review, cost, risk, and business value.

Local side-by-side eval workbench that turns one AI product evaluation idea into eval cases, comparable evidence, and a decision-grade report.

FDE Consultants Protocoles — open-source Forward Deployed Engineer + DeepSCR protocol: turn any coding agent (Claude Code, Codex, Cursor) into a certifying engineer with verifiable AI Assurance Scores, MCP tools, and a public trust registry. Apache-2.0.

Playbook for PMs shipping AI products with PRDs, evals, HITL, launch gates, cost, and observability.

Offline, auditable benchmark for one-shot LLM market decisions.

Unofficial TypeScript starter for deterministic local contract testing around Foundry-oriented workflows with Themis.

Frontend for Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

Rust based CLI tool for estimating success rates when using LLM judges for evaluation.