How We Use Claude Code Skills to Run 1,000+ ML Experiments a Day

claude-codeskillsml-experimentsteam-knowledgeagentic-codingworkflow

Abstraction: Claude Code skills registry for team ML knowledge

Key points:

Sionic AI built a shared "skills" registry so Claude Code captures team ML experiment knowledge; installed via /plugin marketplace add and /plugin install.
Two custom commands drive the loop: /retrospective reads a session and opens a PR with a SKILL.md, plugin.json, and references/scripts; /advise searches the registry before new work.
Every skill template mandates a "Failed Attempts" table — failure documentation is referenced more than success paths; descriptions must be specific with trigger conditions (e.g. exact error strings like vllm_skip_weight_sync).
Concrete example: GRPO training with external vLLM server on ms-swift, gemma-3-12b-it, 8×A100; a ColBERT/FDE param search found ksim=4 works, d_proj=32 loses information.
Base-100 addition transformer case: 40x more params (77K→3.18M) gave 98x exact-match gain; RoPE theta=100 best for short sequences; minimum viable ~253K–500K params.
"Spec-driven modeling": a TECHSPEC.md research contract; infra called Creep Colony schedules GPU jobs on HashiCorp Nomad; W&B queried via MCP; GitHub Actions validate structure and regenerate marketplace.json.

Connections: Sionic AI · Claude Code · Claude Code Skills · Knowledge Management · Agentic Coding