back to slides
Workshop reference · session 1 · "Where good prompts live" deep-dive

Team prompt repository

The plain-Git pattern for managing prompts as team assets. Folder convention, PR template, semver per prompt, eval tooling tiers. Plain-Git first. SaaS only when you outgrow markdown.

Sources cited inline. Last updated 2026-05-02 · WebMD HS workshop · author: Tony Bùi
jump: Folder structure PR template Versioning Eval tooling Sources
Folder structure
subfolder per pattern

One subfolder per pattern (not per task). Prompt + examples + eval grouped together. README at the top defines conventions.

How to use: create prompts/ at the root of your team's repo (or a dedicated your-team-prompts repo). One subfolder per pattern (reviewer / synthesizer / translator …). Strict file names so tooling and humans agree.
prompts/ ├── README.md ← index + conventions ├── _conventions.md ← schema, versioning, PR rules ├── reviewer/ │ ├── prompt.md ← the master prompt │ ├── CHANGELOG.md ← human-readable history │ ├── examples/ ← input/output pairs │ │ ├── 01-prd-section.md │ │ └── 02-spec-review.md │ └── eval.yaml ← Promptfoo regression tests ├── synthesizer/ │ └── ... (same structure) └── translator/ └── ... (same structure)

Synthesized from Promptfoo's examples/ directory + apxml's version-control guidance. The senior-team adaptation: subfolder per pattern, prompt+examples+eval grouped together.

  • Subfolder per pattern, not per task reviewer/ is reusable across PRDs, specs, briefs. member-consent-prd-review/ is a one-shot. Patterns compound; tasks don't.
  • Prompt + examples + eval grouped When you change prompt.md, the diff sits next to its eval cases. PR reviewers can see "did the regression coverage update?" in one glance.
  • README at the top is the index First file a new teammate opens. Lists every pattern, last-updated date, owner. Without it, the repo becomes a graveyard within 6 months.
  • _conventions.md captures the rules once Schema for prompt.md frontmatter, semver rules, PR review template. Underscore prefix sorts it near the top alongside README.
  • Lowercase, hyphenated folders reviewer/ not Reviewer/. synthesizer/ not Synthesizer_v2/. Cross-platform safe, grep-friendly.
  • Domain-named, not project-named reviewer/ is the pattern. member-consent-prd/ is the use case. Patterns are reusable; use cases are not. Name folders after patterns.
  • Strict file names — don't bikeshed prompt.md, eval.yaml, CHANGELOG.md, examples/. So Promptfoo can find eval.yaml automatically. So a script can grep every prompt.md across the repo. Naming is infrastructure.
  • Underscore-prefix for meta files _conventions.md, _glossary.md. Sorts above pattern folders. Signals "read me before patterns."

Per kriasoft/Folder-Structure-Conventions: naming is a contract with tooling. Break it once and grep stops working.

PR review template
4 questions every prompt PR must answer

No PR without naming the failure mode. No merge without eval coverage. No major version without explicit schema-change call-out.

  • 1. What failure mode does this defend? No PR without naming the bug. Forces the author to cite either an incident or a research source — e.g. "sycophantic plan-validation, arxiv 2411.15287."
  • 2. Eval cases added/updated? Regression coverage for the change. If you can't write a failing test before the fix, you don't understand the failure. Promptfoo's CI integration runs eval.yaml on every PR — failed evals block merge.
  • 3. Output schema changed? Forces the semver decision. Schema change = major bump (v1.x → v2.0). New constraint, same schema = minor (v1.1.0 → v1.2.0). Wording fix only = patch (v1.2.0 → v1.2.1).
  • 4. Tested against last 3 model versions? Prompts break silently when models update. A prompt that passes on Claude 4.5 may fail on 4.7. The eval suite must run against your live model set, not just one.
How to use: save as .github/PULL_REQUEST_TEMPLATE/prompt-change.md in your repo. GitHub auto-fills it on every PR opened against the prompts/ path.
## What failure mode does this defend? [Name the specific bug or research source — e.g. "Sycophantic plan-validation, arxiv 2411.15287"] ## Eval cases added/updated? - [ ] Added regression case to `eval.yaml` - [ ] Tested against Claude 4.7, GPT-4.5, Gemini 2.5 ## Output schema changed? - [ ] No — minor version bump - [ ] Yes — major version bump (v1.x → v2.0) ## Tested against last 3 model versions? - Claude: ✓ - GPT: ✓ - Gemini: ✓ ## CHANGELOG entry written? - [ ] yes
What a good PR looks like — fictional but realistic. PR title: reviewer: add PDPL constraint (v1.1.0 → v1.2.0)
## What failure mode does this defend? Reviewer was missing PDPL (VN data privacy) compliance flags on member-data sections. Caught in QA on PRD-2026-04 — reviewer cleared a section that included raw [member health condition] in an analytics export. Source: internal incident QA-0429 + PDPL Article 13. ## Eval cases added/updated? - [x] Added 3 PDPL test cases to `reviewer/eval.yaml` - prd-section-with-raw-condition (must flag) - prd-section-with-anonymized-condition (must pass) - prd-section-cross-border-transfer (must flag) - [x] Tested against Claude 4.7, GPT-4.5, Gemini 2.5 — all pass ## Output schema changed? - [x] No — minor version bump (v1.1.0 → v1.2.0) - Same defect-table columns. New [COMPLIANCE — PDPL] tag added inside existing "Compliance flags" section. ## Tested against last 3 model versions? - Claude 4.7: ✓ - GPT-4.5: ✓ - Gemini 2.5: ✓ ## CHANGELOG entry written? - [x] yes — see `reviewer/CHANGELOG.md` v1.2.0

The "Source: internal incident QA-0429" line is the audit trail. Two years from now, when someone asks "why does the reviewer flag PDPL?", the answer is one git-blame away.

Versioning
semver per prompt · date-tags lose

Each prompt carries its own semver. Major = breaking output schema. Minor = new constraint. Patch = wording fix. Tag the whole repo at deployment milestones.

  • MAJOR — breaking output schema v1.x.x → v2.0.0. The defect-table grew a column. The output is now JSON, was markdown. Anything downstream that parses the output may break. Forces consumer review.
  • MINOR — new constraint or capability v1.1.0 → v1.2.0. Added "[COMPLIANCE — PDPL] tag for VN data handling." Same schema, more rules. Backward-compatible — old consumers still work.
  • PATCH — wording fix only v1.2.0 → v1.2.1. Typo. Clarified "must" vs "should." No behavioral change expected. Eval suite must still pass — that's the proof.
  • Date-only tags lose information 2026-05-04 tells you when, not what. Was that a typo fix or a schema rewrite? Semver makes the breaking-vs-non-breaking call explicit at tag time, not git-archeology time.
"Associating commit hashes with evaluation results lets you definitively link specific prompt versions to observed performance metrics." — apxml, Version Control for Prompts
One CHANGELOG.md per prompt, sitting next to prompt.md. The "Defends:" line connecting changes to research/incident sources is the audit trail regulators want.
# reviewer/CHANGELOG.md ## v1.2.0 — 2026-05-15 - Added [COMPLIANCE — PDPL] tag for VN data handling - Eval: added 3 PDPL test cases - Defends: internal QA-0429 + PDPL Article 13 ## v1.1.0 — 2026-05-04 - Added "adversarial posture" rule to defend sycophancy - Defends: arxiv 2411.15287 ## v1.0.0 — 2026-05-04 - Initial — workshop kickoff
  • Per-prompt semver in prompt.md frontmatter Each prompt carries its own version, independent of the repo. Reviewer at v1.2.0 while Synthesizer is at v3.0.1 — no problem.
  • Repo-level tag at deployment milestones Tag the whole repo when you ship a Project to a customer or update production: git tag v2026.05.04-webmd. Captures the snapshot every prompt was at on that day.
  • Two coordinates: who-and-when (repo tag) + what-and-why (per-prompt semver) Six months later, you can answer both "what was running for WebMD on 2026-05-04?" (checkout the repo tag) and "what changed in reviewer between v1.1 and v1.2?" (read the per-prompt CHANGELOG).
Eval & tooling tiers
3 tiers · adopt on team-size signals

Start with markdown checklists. Graduate to Promptfoo when you hit ~10 prompts. Add observability or SaaS only when production traffic / non-engineer iteration forces it.

Markdown checklist + human review $0 · plain-Git only

Adopt when: <10 prompts, <3 reviewers, no production traffic yet. This is where you start. This is also where most teams should stay longer than they think.

What it looks like: reviewer/eval.md with 5–10 numbered "test cases" — each a snippet of input + the expected behavior in prose. Human runs them by eye on every PR. The PR template's "eval cases added/updated?" checkbox enforces it.

"Plain-Git is durable. SaaS comes when you outgrow markdown. Start where your team already is." — workshop heuristic, source: research synthesis
Promptfoo — YAML eval, CI-ready $0 · open-source

Adopt when: 10–50 prompts, you want automated regression on every PR, you have at least one engineer comfortable with YAML and GitHub Actions.

What it looks like: eval.yaml per pattern. CI runs promptfoo eval on every PR — failed assertions block merge. Free, self-hosted, no vendor lock-in. The de-facto OSS standard. promptfoo.dev

Langfuse — observability layer above Promptfoo self-host or SaaS

Adopt when: production traffic is live, you need to A/B test prompt versions against real users, you want tracing across multi-step chains.

What it looks like: Langfuse ingests Promptfoo eval results + production traces. Dashboards show "v1.1 vs v1.2 latency / cost / quality." Self-hostable — hedges SaaS lock-in. langfuse.com

PromptLayer — visual editor + versions for non-engineers $$$ · SaaS

Adopt when: PMs / copywriters / domain experts need to iterate on prompts without git, and you've already outgrown markdown. Not before.

What it looks like: prompts live in PromptLayer's DB (not your repo). Web UI for editing + version diff + test runs. Engineers fetch via API at runtime. Trade-off: pace up, sovereignty down. promptlayer.com

"PromptLayer excels at getting prompts out of your application's code and into a centralized, governed registry where they can be versioned, tested, and safely deployed." — Nearform, Prompt Management Systems Compared
Caveat — tooling churn is high. PromptLayer / Langfuse / Helicone / Maxim all launched 2023–2024. Some won't exist in 2027. Plain-Git is the only durable substrate. SaaS-managed prompts don't live in your repo — fine for pace, bad for sovereignty. Hedge with self-hostable Langfuse if you need a UI layer. — research synthesis, 2026-05-02

Key sources cited

Workshop · WebMD Health Services Vietnam · 04/05/2026 tony · vibery.app