diff --git a/.agents/skills/Readme-builder/SKILL.md b/.agents/skills/Readme-builder/SKILL.md new file mode 100644 index 0000000..8dfe119 --- /dev/null +++ b/.agents/skills/Readme-builder/SKILL.md @@ -0,0 +1,277 @@ +# SKILL.md — Universal Repository README Generator + +A reusable specification that any repo can drop in to produce a **production-grade, industry-standard README**. Save the content below as `SKILL.md` (or `.github/SKILL.md`) in your repo. It defines: + +1. The **principles** a great README must follow +2. The **canonical section structure** (with what each section must contain) +3. **Templates and snippets** ready to copy +4. A **checklist** to validate the result +5. Optional **automation hooks** for CI/AI assistants + +--- + +````markdown +# SKILL: Repository README Generation + +> **Purpose**: A canonical, opinionated specification for producing a high-quality `README.md` +> for **any** repository — library, service, CLI, plugin, model, dataset, or monorepo. +> Use this file as a contract for humans **and** AI coding assistants. + +--- + +## 1. Guiding principles + +A README must answer five questions in the first 60 seconds: + +1. **What** is this project? (one sentence) +2. **Why** does it exist? (the problem it solves) +3. **Who** is it for? (target audience) +4. **How** do I use it? (install + minimal example) +5. **Where** do I go next? (docs, contributing, license) + +Quality bar: + +- **Skimmable**: clear hierarchy, short paragraphs, tables and code blocks over prose. +- **Self-contained**: a newcomer can install, run, and contribute without leaving the README. +- **Truthful**: every command, badge, link, and version must work at the time of commit. +- **Accessible**: alt text on images, descriptive link text, no jargon without definition. +- **Versioned**: kept in sync with code; PRs that change behavior must update the README. +- **Localizable**: keep README.md in English; translations as `README..md`. + +--- + +## 2. Canonical section structure + +Sections marked **REQUIRED** must appear in every repo. Others are conditional. + +| # | Section | Status | When to include | +|---|---|---|---| +| 1 | Project header (name, tagline, badges, hero image) | REQUIRED | Always | +| 2 | Table of contents | RECOMMENDED | README > 200 lines | +| 3 | Overview / What & Why | REQUIRED | Always | +| 4 | Key features | REQUIRED | Always | +| 5 | Demo / Screenshots / Diagrams | RECOMMENDED | UI, CLI, or architecture worth showing | +| 6 | Architecture | RECOMMENDED | Multi-component or non-trivial systems | +| 7 | Quick start | REQUIRED | Always | +| 8 | Installation | REQUIRED | Always | +| 9 | Usage / Examples | REQUIRED | Always | +| 10 | Configuration | CONDITIONAL | If config exists | +| 11 | API reference / CLI reference | CONDITIONAL | If applicable | +| 12 | Project structure | RECOMMENDED | Repos with > ~10 top-level entries | +| 13 | Development & contributing | REQUIRED | Always | +| 14 | Testing | REQUIRED | Always | +| 15 | Deployment / Release | CONDITIONAL | Services, packages, plugins | +| 16 | Performance & benchmarks | OPTIONAL | Performance-sensitive projects | +| 17 | Security | REQUIRED | Always (link to `SECURITY.md`) | +| 18 | Roadmap | RECOMMENDED | Active projects | +| 19 | FAQ / Troubleshooting | RECOMMENDED | Public projects | +| 20 | Changelog | REQUIRED | Link to `CHANGELOG.md` | +| 21 | Contributors / Acknowledgements | RECOMMENDED | Always | +| 22 | License | REQUIRED | Always | +| 23 | Contact / Support | REQUIRED | Always | + +--- + +## 3. Section-by-section specification + +### 3.1 Project header +- **H1 = exact project name** (no emoji prefix in H1; use one in the tagline if you must). +- **One-sentence tagline** under the title. +- **Badges row** (≤ 6, in this order): build status, coverage, package version, downloads, license, language version. +- Optional **hero image / logo** centered, max width 320px. + +```markdown +

project-name

+

One-sentence value proposition.

+ +

+ Build status + Coverage + Version + License +

+``` + +### 3.2 Table of contents +Auto-generate with a tool (e.g., `markdown-toc`) when README exceeds 200 lines. + +### 3.3 Overview / What & Why +- 2–4 short paragraphs. +- Explicitly state the **problem**, the **solution**, and **non-goals**. +- Avoid marketing fluff; lead with concrete capability. + +### 3.4 Key features +- 4–8 bullets, each ≤ 12 words, action-oriented (verb-first). +- Group by theme if more than 8. + +### 3.5 Demo / Screenshots / Diagrams +- Animated GIF or short MP4 for UI/CLI projects (≤ 5 MB). +- Mermaid/PlantUML/D2 diagrams over static images when possible (renderable in GitHub). + +### 3.6 Architecture +- Use a **C4 container** or **component** diagram for systems. +- Caption with one sentence describing what the diagram shows. +- Keep diagrams in `/docs/diagrams` and embed as code or rendered SVG. + +```markdown +*High-level architecture — request flow from client to data store* + +```mermaid +flowchart LR + Client --> API --> Service --> DB[(Database)] +``` +``` + +### 3.7 Quick start +- A **single, copy-pasteable** block that takes a user from zero to a working result in ≤ 60 seconds. +- Show expected output. + +```bash +# Install +pipx install project-name + +# Run +project-name --help +``` + +### 3.8 Installation +- Cover **all** supported install paths: package manager, source, Docker, binary. +- State **prerequisites** (OS, language version, system libs) explicitly. +- Provide a verification command (e.g., `project-name --version`). + +| Method | Command | +|---|---| +| pip | `pip install project-name` | +| Docker | `docker run ghcr.io/org/project-name:latest` | +| Source | `git clone … && make install` | + +### 3.9 Usage / Examples +- Start with the **simplest meaningful example**, then progress to advanced. +- Each example: minimal, runnable, with expected output. +- For libraries, show **import + call + result**. +- For services, show **request + response**. +- For CLIs, show **command + stdout**. + +### 3.10 Configuration +- Document **every** environment variable, flag, and config file key in a table. +- Include **type**, **default**, **required?**, and **description**. + +| Key | Type | Default | Required | Description | +|---|---|---|---|---| +| `LOG_LEVEL` | string | `info` | no | Logging verbosity | +| `API_KEY` | string | — | yes | Authentication token | + +### 3.11 API / CLI reference +- For libraries: link to generated docs (Sphinx, TypeDoc, Rustdoc, godoc). +- For CLIs: include `--help` output or a generated reference. +- For HTTP APIs: link to OpenAPI/Swagger; include 1–2 representative endpoints inline. + +### 3.12 Project structure +- Show top-level layout with one-line annotations. +- Skip generated/boilerplate folders. + +```text +. +├── src/ # Library source +├── tests/ # Unit + integration tests +├── docs/ # User-facing documentation +├── examples/ # Runnable examples +├── scripts/ # Dev and CI helpers +└── pyproject.toml # Build config +``` + +### 3.13 Development & contributing +- **Local setup** in ≤ 5 commands. +- **Coding standards** (link to `CONTRIBUTING.md`, `STYLEGUIDE.md`). +- **Branching model** (trunk-based, GitFlow, etc.). +- **Commit conventions** (e.g., Conventional Commits). +- **PR checklist** link. + +### 3.14 Testing +- One command to run all tests. +- Separate commands for unit / integration / e2e if applicable. +- Coverage target stated explicitly. +- How to add a new test. + +### 3.15 Deployment / Release +- Versioning scheme (SemVer). +- Release process (tag → CI → registry). +- Rollback procedure for services. + +### 3.16 Performance & benchmarks +- Reproducible commands. +- Hardware/environment used. +- Comparison table vs. alternatives if relevant. + +### 3.17 Security +- Link to `SECURITY.md` (vulnerability reporting policy). +- Supported versions table. +- Known limitations / threat model summary. + +### 3.18 Roadmap +- Link to issues, project board, or milestones. +- 3–5 near-term items inline. + +### 3.19 FAQ / Troubleshooting +- 5–10 real questions from issues/support. +- Format: question as H3, concise answer. + +### 3.20 Changelog +- Link to `CHANGELOG.md` (keep-a-changelog format). +- Latest version highlights inline. + +### 3.21 Contributors / Acknowledgements +- Use [all-contributors](https://allcontributors.org/) or `CONTRIBUTORS.md`. +- Credit upstream projects, sponsors, inspirations. + +### 3.22 License +- SPDX identifier + link to `LICENSE` file. +- One line: `Released under the [MIT License](LICENSE).` + +### 3.23 Contact / Support +- Issue tracker for bugs / features. +- Discussion forum / Slack / Discord for questions. +- Security contact (email or form) — never use public issues. + +--- + +## 4. Cross-cutting requirements + +### 4.1 Companion files (must exist alongside README) +- `LICENSE` +- `CONTRIBUTING.md` +- `CODE_OF_CONDUCT.md` (Contributor Covenant) +- `SECURITY.md` +- `CHANGELOG.md` +- `.github/ISSUE_TEMPLATE/`, `.github/PULL_REQUEST_TEMPLATE.md` +- `CITATION.cff` for academic / research projects + +### 4.2 Style rules +- **Headings**: sentence case; one H1 only; no skipped levels. +- **Line length**: soft-wrap at ~100 chars for diff-friendliness. +- **Code blocks**: always specify language for syntax highlighting. +- **Links**: descriptive text (no "click here"); prefer relative links inside the repo. +- **Images**: store under `/docs/assets` or `/.github/assets`; always include `alt`. +- **Emoji**: use sparingly; never as the only signifier of meaning. +- **Tone**: second person, active voice, present tense. + +### 4.3 Accessibility +- Alt text on every image and diagram. +- Sufficient color contrast in custom badges. +- Don't rely on color alone to convey status. + +### 4.4 Localization +- Primary README in English at repo root. +- Translations: `README.fr.md`, `README.ja.md`, etc., linked from a language switcher at the top. + +--- + +## 5. Validation checklist + +Use this before merging any README change. + +- [ ] H1 matches package/repo name exactly +- [ ] Tagline ≤ 120 characters +- [ ] All required sections present +- [ ] Quick start works on a clean machine +- [ ] All commands tested (copy-paste-run) \ No newline at end of file diff --git a/.agents/skills/architecture-discovery/SKILL.md b/.agents/skills/architecture-discovery/SKILL.md new file mode 100644 index 0000000..b7c0c53 --- /dev/null +++ b/.agents/skills/architecture-discovery/SKILL.md @@ -0,0 +1,47 @@ +--- +name: architecture-discovery +description: Understand the repository architecture, execution flow, dependencies, and security-critical components before making changes. +--- + +# architecture-discovery + +You are a Senior Software Architect performing brownfield system analysis. + +Your objective is to understand the repository before proposing modifications. + +## Usage + +Use this skill before: + +- Refactoring +- Feature development +- Security analysis +- Documentation updates +- Test generation + +## Steps + +1. Read README and project documentation. +2. Identify application entry points. +3. Locate public APIs. +4. Identify SQL validation workflow. +5. Identify configuration loading logic. +6. Build a component map. +7. Trace request execution flow. +8. Identify security-critical modules. +9. Document module responsibilities. +10. Create an architecture summary. + +## Output Format + +### Repository Overview + +### Component Map + +### Execution Flow + +### Security-Critical Components + +### Technical Debt Observations + +### Recommended Improvement Areas \ No newline at end of file diff --git a/.agents/skills/benchmark-suite-builder/SKILL.md b/.agents/skills/benchmark-suite-builder/SKILL.md new file mode 100644 index 0000000..96ed40d --- /dev/null +++ b/.agents/skills/benchmark-suite-builder/SKILL.md @@ -0,0 +1,54 @@ +--- +name: benchmark-suite-builder +description: Build a comprehensive SQL attack benchmark suite and detection effectiveness report for the project. +--- + +# benchmark-suite-builder + +You are a Security Benchmark Engineer. + +Your objective is to build a measurable evaluation framework for SQL Data Guard. + +## Usage + +Use this skill when creating a flagship hackathon contribution. + +## Steps + +1. Review current detection capabilities. +2. Create attack categories. +3. Generate attack corpus. +4. Build automated benchmark execution. +5. Measure detection results. +6. Identify missed attacks. +7. Identify false positives. +8. Generate benchmark reports. +9. Create CI integration recommendations. +10. Produce demo-ready output. + +## Attack Categories + +- Classic SQL Injection +- UNION Injection +- Blind SQL Injection +- Boolean Injection +- Stacked Queries +- Comments +- Nested Queries +- Restriction Bypass +- Obfuscated Payloads +- Encoding Attacks + +## Output Format + +### Benchmark Summary + +### Detection Results + +### Missed Payloads + +### False Positives + +### Improvement Recommendations + +### CI Integration Plan \ No newline at end of file diff --git a/.agents/skills/documentation-engineer/SKILL.md b/.agents/skills/documentation-engineer/SKILL.md new file mode 100644 index 0000000..ab03ebc --- /dev/null +++ b/.agents/skills/documentation-engineer/SKILL.md @@ -0,0 +1,46 @@ +--- +name: documentation-engineer +description: Generate contributor-friendly technical documentation, architecture guides, onboarding material, and implementation explanations. +--- + +# documentation-engineer + +You are a Technical Writer and Open Source Maintainer. + +Your objective is to improve developer onboarding and project understanding. + +## Usage + +Use this skill when: + +- Updating project documentation +- Explaining architecture +- Creating onboarding guides +- Preparing hackathon deliverables + +## Steps + +1. Analyze repository structure. +2. Identify major components. +3. Explain validation workflow. +4. Explain injection detection workflow. +5. Document configuration options. +6. Document extension points. +7. Create onboarding instructions. +8. Add examples where appropriate. +9. Generate diagrams when useful. +10. Ensure documentation matches implementation. + +## Output Format + +### Architecture Overview + +### Validation Engine + +### Injection Detection + +### Configuration + +### Extensibility + +### Contributor Guide \ No newline at end of file diff --git a/.agents/skills/hackathon-submission/SKILL.md b/.agents/skills/hackathon-submission/SKILL.md new file mode 100644 index 0000000..75b79d7 --- /dev/null +++ b/.agents/skills/hackathon-submission/SKILL.md @@ -0,0 +1,256 @@ +# SKILL.md — A4I Hackathon Submission Markdown Generator + +A specification for producing the **A4I Hackathon submission markdown** — a clear, +evidence-backed write-up of the work done, aimed at **upper management and technical +reviewers**. It follows the official template format **exactly** and is filled **only** with +facts that can be verified from the repository or supplied by the team. + +> Persona: a senior solution designer (20+ yrs) presenting delivered work to leadership. +> Every claim is concrete, sourced, and skimmable. No marketing language. + +--- + +## 1. Non-negotiable rules (read first) + +1. **No hallucination. No assumptions.** State only what is (a) supplied by the user, or + (b) directly verifiable in the repo (code, docs, tests, CI, git history). If neither, do + **not** invent — insert an explicit placeholder: + `> ⚠️ NEEDS INPUT: ` +2. **Cite the evidence.** Where practical, point to the source: a file path + (`src/...`), a test count, a CI workflow, a PR/commit link. Reviewers trust traceable claims. +3. **Follow the attached format exactly** — same H1/H2/H3 headings, same order, same section + numbers (1–8). Do not add, rename, reorder, or drop sections. Section 8 is optional and may + be omitted only if empty. +4. **Distinguish fact from intent.** Delivered work = present tense, evidenced. Plans/ideas = + clearly labelled as future/next-step, never stated as done. +5. **Management-readable.** Lead each section with the point; keep paragraphs short; prefer + bullets and tables. Define any acronym on first use. + +--- + +## 2. The exact output format + +Reproduce this structure verbatim (headings and numbering must match the official template): + +``` +# {Project Name} +## 1. Pull/Merge request +## 2. Problem & Scope +### Context +### Problem to solve +### Why it matters +### In scope +### Out of scope +### Assumptions +## 3. Use Cases Delivered +### Use case 1 (Description / Expected outcome / What was implemented) +### Use case 2 … +## 4. Implementation Overview +### Architecture +### Key technical choices +## 5. Coding Assistant Usage +### Tools used +### Approach +### Agentic features implemented +### Key learnings +## 6. Quality & Testing +### Testing approach +### Code quality +## 7. Technical Documentation +## 8. Appendix (optional) +``` + +### What each section must contain — and where to source it + +| Section | Must contain | Primary source | If unknown | +|---|---|---|---| +| **Project Name** (H1) | The actual project/repo name | repo name, `README`, manifest | placeholder | +| **1. Pull/Merge request** | The PR/MR link(s) for the work | user, or `git` remote + branch → compare URL (confirm) | `⚠️ NEEDS INPUT` | +| **2 · Context** | Existing system, stakeholders, constraints | README "overview", docs | `⚠️ NEEDS INPUT` | +| **2 · Problem to solve** | The concrete problem | README "why" section | `⚠️ NEEDS INPUT` | +| **2 · Why it matters** | Business/technical impact, risks | README, SECURITY, domain docs | `⚠️ NEEDS INPUT` | +| **2 · In scope** | What the hackathon work covered | git diff/log of the branch, user | `⚠️ NEEDS INPUT` | +| **2 · Out of scope** | What was deliberately excluded | user (a decision, not derivable) | `⚠️ NEEDS INPUT` | +| **2 · Assumptions** | Key team assumptions | user (a decision, not derivable) | `⚠️ NEEDS INPUT` | +| **3. Use Cases Delivered** | One block per use case: *Description / Expected outcome / What was implemented* | code, README features, examples, tests | `⚠️ NEEDS INPUT` | +| **4 · Architecture** | Components and how they fit | code structure, README architecture, diagrams | describe only what exists | +| **4 · Key technical choices** | Frameworks, tools, design decisions + rationale | manifests/deps, code, README | state choice; flag rationale if unstated | +| **5 · Tools used** | Coding assistant(s) used | user; corroborate via `.agents`/`.claude`, commit co-authors | `⚠️ NEEDS INPUT` | +| **5 · Approach** | Prompting, context usage, workflows | user | `⚠️ NEEDS INPUT` | +| **5 · Agentic features implemented** | Agentic features used + for which use cases | user; corroborate via skills/workflows in repo | `⚠️ NEEDS INPUT` | +| **5 · Key learnings** | What worked, limitations | user | `⚠️ NEEDS INPUT` | +| **6 · Testing approach** | Test strategy (unit/integration/…) | `test/`, CI workflows, test configs | describe what exists | +| **6 · Code quality** | Refactoring, readability, maintainability work | linters/formatters config, CI, git history | state only what's evidenced | +| **7. Technical Documentation** | README-style: setup, usage, structure, key details | README, ONBOARDING, docs/ | summarize + link | +| **8. Appendix** | Extra scripts, links, prompts | user, repo | omit if empty | + +> Sections **2 (Out of scope, Assumptions)** and **5 (Approach, Agentic features, Key learnings)** +> describe human decisions and process — they are **rarely derivable from the repo**. Expect to +> ask the user or leave `⚠️ NEEDS INPUT`. Never fabricate them. + +--- + +## 3. Intake questions (ask the user — optional, but strongly advised for §2 and §5) + +Ask once, in a single batched prompt. Make clear each is optional, and apply the precedence +**user answer → verifiable repo evidence → `⚠️ NEEDS INPUT` placeholder** (never invention): + +- **PR/MR link?** (for §1) — else derive a compare URL from the remote/branch and ask to confirm. +- **Out of scope?** and **Assumptions?** (for §2) — decisions only the team knows. +- **Coding assistant(s) used, and how** — tools, prompting/context approach, workflows (§5). +- **Agentic features used, and for which use cases?** (§5) +- **Key learnings — what worked, what didn't?** (§5) +- **Anything for the Appendix?** (§8) + +Everything else (problem, why, use cases, architecture, tech choices, testing, docs) should be +**drafted from the repo first**, then shown to the user to confirm or correct. + +--- + +## 4. Repo-sourcing playbook (for everything not answered) + +Mine, in order, and record where each fact came from: + +1. `README*` / `ONBOARDING*` / `docs/` → project name, context, problem, why, features, docs. +2. Manifests / deps (`pyproject.toml`, `package.json`, `pom.xml`, `go.mod`, …) → tech choices. +3. Code tree (`src/`, packages) → architecture, components, delivered use cases. +4. `examples/` → concrete use cases and expected outcomes. +5. `test/` + test configs → testing approach; **count** suites/files for evidence. +6. CI workflows (`.github/workflows`) → testing automation, quality gates, release. +7. Lint/format configs (`.editorconfig`, ruff/eslint/black, pre-commit) → code-quality claims. +8. `.agents/` `.claude/` + commit co-authors + git log on the branch → corroborate §5 and scope. + +**Rule:** if mining doesn't surface a fact and the user didn't give it, write +`⚠️ NEEDS INPUT: …` — do not approximate. + +--- + +## 5. Ready-to-fill template + +Copy this, fill it, and save per §6. Keep headings/numbers exactly as below. Replace each +`{{…}}`. Replace any field you cannot source with the `⚠️ NEEDS INPUT` line. Duplicate the +"Use case" block as many times as there are real use cases. + +```markdown +# {{Project Name}} + +## 1. Pull/Merge request +{{PR/MR link — e.g. https://github.com///pull/}} + +## 2. Problem & Scope + +### Context +{{Existing system, stakeholders, constraints — sourced from README/docs.}} + +### Problem to solve +{{The concrete problem this work addresses.}} + +### Why it matters +{{Business, technical, and risk impact.}} + +### In scope +{{What the hackathon work covered (back with branch diff/commits).}} + +### Out of scope +{{What was deliberately not covered.}} + +### Assumptions +{{Key assumptions the team made.}} + +--- + +## 3. Use Cases Delivered + +### Use case 1 +**Description:** {{what it is}} +**Expected outcome:** {{the intended result}} +**What was implemented:** {{what actually shipped — cite files/examples/tests}} + +### Use case 2 +**Description:** {{…}} +**Expected outcome:** {{…}} +**What was implemented:** {{…}} + +--- + +## 4. Implementation Overview + +### Architecture +{{Overall architecture and main components. Include a diagram only if one exists or can be +drawn faithfully from the code.}} + +### Key technical choices +{{Frameworks, tools, and main design decisions — with rationale where stated.}} + +--- + +## 5. Coding Assistant Usage + +### Tools used +{{Coding assistant(s) used.}} + +### Approach +{{How they were used — prompting, context usage, workflows.}} + +### Agentic features implemented +{{Agentic features used, and for which use cases.}} + +### Key learnings +{{What worked well and the limitations encountered.}} + +--- + +## 6. Quality & Testing + +### Testing approach +{{Testing strategy — unit/integration/etc. Cite suite/file counts and CI.}} + +### Code quality +{{Refactoring, readability, maintainability improvements — only what's evidenced.}} + +--- + +## 7. Technical Documentation +{{README-style detail: setup, usage, structure, key implementation details. Summarize and +link to the in-repo docs.}} + +--- + +## 8. Appendix (optional) +{{Additional scripts, links, prompts, or supporting materials. Omit this section if empty.}} +``` + +--- + +## 6. File naming & output + +- Name the file using the official convention: + **`A4I Hackathon__MarkdownFile.md`** (ask for the team name; placeholder if unknown). +- Output one markdown file. Do not split across files. +- After generating, **list every `⚠️ NEEDS INPUT` item** back to the user so they can fill gaps. + +--- + +## 7. Writing principles (leadership audience) + +- **Lead with the answer**, then support it — reviewers skim. +- **Quantify** ("9 test suites", "4 integrations") rather than qualify ("robust", "extensive"). +- **Tables and bullets** over long prose; one idea per bullet. +- **Plain language**; expand acronyms on first use. +- **Traceable**: link PRs, files, and docs so claims can be checked. +- **Honest scope**: separate delivered work from intent; don't overstate. + +--- + +## 8. Validation checklist + +- [ ] H1 + sections 1–8 present, named and numbered exactly as the official template. +- [ ] Section 1 contains a real PR/MR link (or `⚠️ NEEDS INPUT`). +- [ ] Every factual claim is sourced from the repo or the user — nothing invented. +- [ ] Unknowns are explicit `⚠️ NEEDS INPUT` lines, not guesses. +- [ ] Delivered vs. planned work is clearly distinguished. +- [ ] Use-case blocks each have Description / Expected outcome / What was implemented. +- [ ] §5 reflects only what the team confirmed about assistant usage. +- [ ] Testing/quality claims cite concrete evidence (counts, configs, CI). +- [ ] File named `A4I Hackathon__MarkdownFile.md`. +- [ ] All `⚠️ NEEDS INPUT` items surfaced to the user after generation. diff --git a/.agents/skills/quadrant-onepager/SKILL.md b/.agents/skills/quadrant-onepager/SKILL.md new file mode 100644 index 0000000..e067cb2 --- /dev/null +++ b/.agents/skills/quadrant-onepager/SKILL.md @@ -0,0 +1,362 @@ +# SKILL.md — 4-Quadrant One-Pager Slide Generator + +A reusable, **repo-agnostic** specification for producing a **single, presentation-ready +4-quadrant one-pager** that matches the **A4I Hackathon** dark template. Drop this in *any* +repo — any language, any stack — and a human or AI assistant can generate a crisp, demo-grade +slide that summarizes the project on one page. It works from **optional user answers** (§4), +falling back to **repo-derived content** (§5) for anything left unanswered. + +> Written from the perspective of a senior presentation designer: the slide must read in +> **15 seconds**, survive a projector, and look intentional — not auto-generated. + +--- + +## 1. When to use this skill + +Use it to generate a **one-page executive summary** of a project, hackathon entry, or demo: + +- End-of-hackathon submission slide. +- Project showcase / stakeholder one-pager. +- "What we built and why it matters" leave-behind. + +The output is **one 16:9 slide**, four quadrants, no more. If the story needs two slides, +it does not belong in this format — tighten the story instead. + +--- + +## 2. The template anatomy + +A header band on top, then a 2×2 grid of quadrant cards. Read clockwise from top-left: + +``` +┌──────────────────────────────────────────────────────────┐ +│ {DECK TITLE} │ ← header band +│ {italic subtitle} │ +├───────────────────────────┬──────────────────────────────┤ +│ ① Problem & opportunity │ ② Hypothesis │ +│ · Problem to solve │ · Hypothesis on assistants │ +│ · Why it matters (stakes) │ · How AI was used │ +│ │ · Best practices applied │ +│ │ · Spec-driven approach (opt) │ +│ │ · Skills / framework / tools │ +├───────────────────────────┼──────────────────────────────┤ +│ ③ Use cases & results │ ④ Value & Impact (HERO) │ +│ · Key use cases delivered │ · Business value generated │ +│ · Measurable outcomes │ · Reusability potential │ +│ │ · Next steps for scaling │ +└───────────────────────────┴──────────────────────────────┘ +``` + +### What each quadrant must answer + +| # | Quadrant | The question it answers | Mandatory content | +|---|---|---|---| +| ① | **Problem & opportunity** | *Why does this exist?* | The problem; why it matters (impact, stakes) | +| ② | **Hypothesis** | *What did we bet on & how did we build it?* | Hypothesis on coding assistants; how AI was used (prompts, features, workflows); engineering best practices; **mandatory: skills / framework / tools**; *optional: spec-driven approach (SpecKit, BMAD…)* | +| ③ | **Use cases & results** | *What did we actually deliver?* | Key use cases delivered; measurable outcomes (code quality, test coverage, …) | +| ④ | **Value & Impact** | *So what — why should anyone care?* | Business value; reusability potential; next steps for scaling | + +> **Quadrant ④ is the hero.** It gets the bright (near-white) border so the eye lands on +> impact last. Everything else uses the teal border. + +--- + +## 3. Design system (tuned to the A4I template) + +Faithful tokens extracted from the reference slide. Keep these unless the client rebrands. + +| Token | Hex | Use | +|---|---|---| +| `--page-bg` | `#232839` | Outer slate background | +| `--frame` | `#f4f6fb` | Thin outer frame line | +| `--panel-bg` | `#0a1322` | Near-black navy behind the cards | +| `--card-bg` | `#06182a` | Quadrant card fill | +| `--card-border` | `#1f5d78` | Standard quadrant border (teal) | +| `--hero-border` | `#e8eef5` | Hero quadrant border (Value & Impact) | +| `--title` | `#ffffff` | Deck + quadrant titles | +| `--subtitle` | `#c7cedb` | Italic header subtitle | +| `--body` | `#cfd9e6` | Bullet text | +| `--bullet` | `#4f9fd6` | Bullet dot (blue) | + +**Type:** system sans (Segoe UI / Helvetica / Arial). Deck title ~40px bold; quadrant +title ~26px bold centered; body ~18px. **Canvas:** 1280×720 (16:9), scalable to 1920×1080. + +**Layout rules:** equal-size quadrants, generous inner padding (~28px), rounded corners +(~10px), 20–24px gap between cards. Titles centered; bullets left-aligned. + +--- + +## 4. Intake questions (ask the user — ALL OPTIONAL) + +Before generating, ask the user the questions below **once** (a single batched prompt — e.g. +`AskUserQuestion` — not one at a time). Make it explicit that **every answer is optional**: + +> *"Answer any of these to steer the slide. Skip any — or all — and I'll derive that content +> from the repo's code, docs, tests, and git history."* + +**Golden rule of precedence:** for each field, **use the user's answer if given; otherwise +auto-derive it from the repo (§5); otherwise mark `[TODO: confirm]`.** Never block on an +unanswered question — a skipped answer is a signal to fall back, not to stop. + +| Field | Question to ask | Maps to | If skipped → fallback | +|---|---|---|---| +| **Deck title** | "Slide title?" | Header | Repo / project name | +| **Subtitle** | "Subtitle or event name? (e.g. a hackathon)" | Header | "4-quadrant one-pager" | +| **Problem** | "What problem does this solve, and who does it hurt?" | ① | Infer from README / docs | +| **AI assistant(s)** | "Which AI coding assistant(s) did you use?" | ② | Infer from repo (skills dir, CI, commit co-authors) or omit | +| **How AI was used** | "How did you use it — key prompts, features, workflows?" | ② | Infer from skills / config / commit history or omit | +| **Skills / framework / tools** | "Which skills, frameworks, or tools? (e.g. MCP, SpecKit, BMAD)" | ② | Infer from `.agents`/`.claude`, manifests, deps | +| **Best practices** | "Engineering practices to highlight? (tests-first, CI, reviews)" | ② | Infer from `test/`, CI workflows, PR setup | +| **Metrics** | "Any numbers to feature? (coverage %, tests, time saved)" | ③ | Count from repo (test files, integrations); never invent | +| **Use cases** | "Key use cases / features delivered?" | ③ | Infer from README features + code structure | +| **Business value** | "Business value — who benefits, risk/$$ avoided?" | ④ | Infer cautiously from README "why" section | +| **Next steps** | "Next steps for scaling?" | ④ | Infer from roadmap / TODO / open issues, else `[TODO: confirm]` | +| **Output format** | "HTML (best fidelity) or editable PPTX?" | — | Default HTML (Method A) | + +> The **only** quadrant that is hard to fully auto-derive is **② Hypothesis** (it's about +> *how the humans worked with AI*). If the user skips everything, still produce a complete +> slide from the repo, and clearly flag any ② bullets that are inferred or `[TODO: confirm]`. + +--- + +## 5. Content-gathering playbook (the repo-derived fallback) + +For every field the user did **not** answer, mine the source material — then write. + +1. **Read the repo signal**, in this order: `README.md`, `ONBOARDING.md`/onboarding docs, + `docs/`, build/manifest files (`pyproject.toml`, `package.json`, `pom.xml`, `go.mod`, …), + `test/` (count tests, coverage), CI workflows (`.github/workflows`), agent/skill configs + (`.agents/`, `.claude/`), and recent `git log` (including co-authors → AI assistants). For a + hackathon, also read any submission notes. +2. **Extract evidence for each quadrant** — pull *concrete, quantified* facts, not adjectives: + - ① the named problem + who it hurts (regulations, risk, cost). + - ② AI tools/skills/frameworks evidenced in the repo, plus engineering practices (tests, CI, reviews). + - ③ shipped features + numbers (N test files, X% coverage, integrations, supported targets). + - ④ business value, what's reusable, the next scaling step. +3. **If a fact isn't in the source and the user didn't supply it, don't invent it.** Mark it + `[TODO: confirm]` and tell the user what's missing rather than fabricating metrics. +4. **Merge, don't duplicate:** when the user gives a partial answer, blend it with repo + evidence (e.g. user says "we used Claude Code" → you add the specific skills you found). + +### Writing principles (presentation-grade) + +- **≤ 5 bullets per quadrant**, **≤ 10 words per bullet**. If it wraps to 3 lines, cut it. +- **Verb-first, present tense.** "Blocks injection" > "The system is able to block injections." +- **Quantify** wherever possible ("9 test suites", "3 integrations", "1 runtime dependency"). +- **No paragraphs, no sub-bullets.** One idea per line. +- **Parallel structure** within a quadrant (all bullets start the same grammatical way). +- Lead each quadrant with its strongest point — people read the first bullet, skim the rest. + +--- + +## 6. Output method A — self-contained HTML (PRIMARY, pixel-faithful) + +Highest fidelity to the template and trivial to preview/export. Produce **one `.html` file**. +Replace the `{{PLACEHOLDERS}}`; keep one `
  • ` per bullet; delete unused `
  • `s. + +```html + + + + +{{DECK_TITLE}} — one-pager + + + +
    +
    +

    {{DECK_TITLE}}

    +

    {{DECK_SUBTITLE}}

    +
    +
    +
    +

    Problem & opportunity

    +
      +
    • {{PROBLEM_1}}
      • +
    • {{PROBLEM_2}}
      • +
    +
    +
    +

    Hypothesis

    +
      +
    • {{HYP_1}}
      • +
    • {{HYP_2}}
      • +
    • {{HYP_3}}
      • +
    • {{HYP_4}}
      • +
    +
    +
    +

    Use cases and results

    +
      +
    • {{RESULT_1}}
      • +
    • {{RESULT_2}}
      • +
    +
    +
    +

    Value & Impact

    +
      +
    • {{VALUE_1}}
      • +
    • {{VALUE_2}}
      • +
    • {{VALUE_3}}
      • +
    +
    +
    +
    + + +``` + +### Export the HTML to an image / PDF / PPT + +- **Quick preview:** open the file in any browser. +- **PNG (crisp):** the slide is exactly 1280×720 — screenshot the `.slide` element, or use a + headless browser (e.g. `playwright`/`puppeteer` `screenshot`, or Chrome + `--headless --screenshot --window-size=1280,720`). +- **PDF:** browser → Print → Save as PDF, landscape, margins none, scale 100%. +- **Into PowerPoint:** paste the PNG full-bleed onto a blank 16:9 slide, **or** use Method B + for a natively editable deck. + +--- + +## 7. Output method B — editable PPTX by cloning the template (PREFERRED for PPT) + +When the client needs a **native, editable PowerPoint**, do **not** rebuild the design from +scratch — **clone the bundled template and replace only the text.** This guarantees pixel-perfect +fidelity (rounded cards, hero border, fonts, colours, bullet glyphs all preserved automatically). + +- **Template file:** `template.pptx`, shipped next to this `SKILL.md`. It contains one 16:9 + slide: a header textbox + a group of four quadrant textboxes (titles + `Problem & opportunity`, `Hypothesis`, `Use cases and results`, `Value & Impact`). +- **Requires:** `pip install python-pptx`. +- **How it works:** the script walks the shape tree (recursing into groups), matches each + quadrant textbox by its **title paragraph**, keeps the title + any spacer, and clones the + first bullet paragraph for each new bullet (so styling is inherited, not re-specified). + +Edit `DECK_TITLE`, `DECK_SUBTITLE`, and the `QUADRANTS` dict, then run it. It writes +`one-pager.pptx` and never touches the template. + +```python +from copy import deepcopy +from pptx import Presentation +from pptx.oxml.ns import qn + +TEMPLATE = "template.pptx" # ships beside this SKILL.md +OUTPUT = "one-pager.pptx" + +# ---- edit this block ---- +DECK_TITLE = "Your project name" +DECK_SUBTITLE = "Event / one-line subtitle" +QUADRANTS = { # keys MUST match the template's quadrant titles + "Problem & opportunity": ["…", "…"], + "Hypothesis": ["…", "…", "…"], + "Use cases and results": ["…", "…"], + "Value & Impact": ["…", "…", "…"], +} +# ------------------------- + +def para_text(p): return "".join(r.text for r in p.runs) + +def set_para_text(p_el, text): + runs = p_el.findall(qn("a:r")) + runs[0].find(qn("a:t")).text = text + for extra in runs[1:]: p_el.remove(extra) # keep first run's formatting only + +def fill_textbox(tf, bullets): + txBody, paras = tf._txBody, tf.paragraphs + tmpl = next((p._p for p in paras[1:] if para_text(p).strip()), None) + if tmpl is None: return + for p in paras[1:]: # drop old bullets, keep title + spacers + if para_text(p).strip(): txBody.remove(p._p) + for b in bullets: # clone styled bullet for each new line + newp = deepcopy(tmpl); set_para_text(newp, b); txBody.append(newp) + +def walk(shapes): + for sh in shapes: + if sh.shape_type == 6: yield from walk(sh.shapes) # 6 = group + elif sh.has_text_frame: yield sh + +def set_header(tf): + runs = [r for p in tf.paragraphs for r in p.runs] + if runs: + runs[0].text = DECK_TITLE + if len(runs) > 1: runs[-1].text = DECK_SUBTITLE + +prs = Presentation(TEMPLATE) +for sh in walk(prs.slides[0].shapes): + head = para_text(sh.text_frame.paragraphs[0]).strip() + if head in QUADRANTS: fill_textbox(sh.text_frame, QUADRANTS[head]) + elif head.startswith("A4I Hackathon"): set_header(sh.text_frame) +prs.save(OUTPUT); print("OK wrote", OUTPUT) +``` + +> **Why cloning beats building:** the A4I template uses a grouped layout, custom rounded-corner +> shapes, and a specific font/colour theme that are tedious and error-prone to recreate with +> `add_shape`. Cloning inherits all of it for free — you only ever touch text. +> +> **No template file?** (e.g. a different client deck) Fall back to building from scratch with +> the design tokens in §3 — but prefer obtaining the real `.pptx` and cloning it. + +### Previewing a `.pptx` + +`python-pptx` can't render images. To eyeball the result, open it in PowerPoint, or convert +with LibreOffice if available: `soffice --headless --convert-to png one-pager.pptx`. Otherwise +rely on Method A's HTML/PNG (§6) as the visual proof and ship the `.pptx` for editing. + +--- + +## 8. End-to-end flow (how to run the skill) + +1. **Ask the intake questions** (§4) in one batched, clearly-optional prompt. +2. **Mine the repo** (§5) for every field the user left blank; merge partial answers with + repo evidence rather than overwriting them. +3. **Draft bullets**, enforcing the **≤5 / ≤10-word**, verb-first limits (§5 writing principles). +4. **Pick output** — Method A (HTML, §6) for fidelity or Method B (PPTX, §7) for editability; + use the user's `Output format` answer, else default to HTML. +5. **Generate the file**, then **export to PNG/PDF** (§6) and show the user the result. +6. **Surface gaps** — list any bullets that were inferred or marked `[TODO: confirm]` so the + user can correct them. + +> Works with **zero answers**: skip step 1's responses and the skill still produces a complete, +> repo-derived slide. Works with **full answers**: the user's content takes precedence. Most +> runs are a mix — that's the intended mode. + +--- + +## 9. Validation checklist + +- [ ] Exactly four quadrants, titles match the template (or agreed rename). +- [ ] **Value & Impact** carries the hero (bright) border. +- [ ] ≤ 5 bullets per quadrant; ≤ 10 words per bullet; no wrapped 3-liners. +- [ ] Every claim is sourced from the repo/work — no invented metrics. +- [ ] Quadrant ② names the actual skills / framework / tools (mandatory). +- [ ] Bullets are verb-first, parallel, quantified where possible. +- [ ] Colors match the design tokens; 16:9; nothing clipped or overflowing. +- [ ] Renders cleanly in a browser **and** exports to a legible PNG/PDF at projector size. +- [ ] Fits on **one** page — no scroll, no second slide. diff --git a/.agents/skills/quadrant-onepager/fill_template.py b/.agents/skills/quadrant-onepager/fill_template.py new file mode 100644 index 0000000..c9c7feb --- /dev/null +++ b/.agents/skills/quadrant-onepager/fill_template.py @@ -0,0 +1,118 @@ +"""Fill the A4I 4-quadrant template (sample.pptx) with content, preserving its design. + +Clones the template's textboxes and replaces only the bullet text — the title styling, +colors, fonts, bullet glyphs, rounded cards and hero border all stay exactly as designed. + +Usage: python fill_template.py -> writes one-pager.pptx +""" +from copy import deepcopy +from pptx import Presentation +from pptx.oxml.ns import qn + +TEMPLATE = "template.pptx" +OUTPUT = "one-pager.pptx" + +# ---- edit this block (matches the sample.html dummy data) ---- +DECK_TITLE = "sql-data-guard" +DECK_SUBTITLE = "A4I Hackathon — 4-quadrant one-pager (sample)" + +# keys MUST match the template's quadrant titles (paragraph 0 of each textbox) +QUADRANTS = { + "Problem & opportunity": [ + "LLM-generated SQL can't run as prepared statements", + "SQL injection & data leaks risk GDPR / CCPA fines", + "DB permissions can't express row/column-level rules", + ], + "Hypothesis": [ + "Coding assistants can ship a safety layer faster", + "Built with Claude Code + custom skills & MCP", + "Engineering practices: tests-first, CI on every push", + "Spec-driven via reusable SKILL.md contracts", + "Tools: sqlglot, Flask, flasgger, python-pptx", + ], + "Use cases and results": [ + "Verifies & auto-rewrites unsafe queries pre-execution", + "9 test suites: unit, joins, updates, REST, DuckDB", + "4 integrations: library, REST, MCP wrapper, Dify", + "Multi-dialect parsing (SQLite, Postgres, +)", + ], + "Value & Impact": [ + "Blocks data breaches before they reach the DB", + "Drop-in for any LLM-to-SQL app — zero DB changes", + "Reusable as pip package + Docker image", + "Next: more dialects, policy templates, auth", + ], +} +# -------------------------------------------------------------- + + +def para_text(p): + return "".join(r.text for r in p.runs) + + +def set_para_text(p_el, text): + """Set a cloned 's text into its first run; drop the other runs (keep formatting).""" + runs = p_el.findall(qn("a:r")) + first = runs[0] + first.find(qn("a:t")).text = text + for extra in runs[1:]: + p_el.remove(extra) + + +def fill_textbox(tf, bullets): + txBody = tf._txBody + paras = tf.paragraphs + title_p = paras[0]._p + + # find a non-empty bullet paragraph to use as the styling template + tmpl = None + for p in paras[1:]: + if para_text(p).strip(): + tmpl = p._p + break + if tmpl is None: # no bullets in template box; nothing to clone + return + + # remove every existing bullet paragraph (keep title + any empty spacer paragraphs) + for p in paras[1:]: + if para_text(p).strip(): + txBody.remove(p._p) + + # append new bullets cloned from the template bullet (preserves font/size/colour/glyph) + for b in bullets: + newp = deepcopy(tmpl) + set_para_text(newp, b) + txBody.append(newp) + + +def walk(shapes): + for sh in shapes: + if sh.shape_type == 6: # group -> recurse + yield from walk(sh.shapes) + elif sh.has_text_frame: + yield sh + + +def set_header(tf): + runs = [] + for p in tf.paragraphs: + runs.extend(p.runs) + if not runs: + return + runs[0].text = DECK_TITLE # first run = title line + if len(runs) > 1: + runs[-1].text = DECK_SUBTITLE # last run = subtitle line (after the line break) + + +prs = Presentation(TEMPLATE) +slide = prs.slides[0] + +for sh in walk(slide.shapes): + head = para_text(sh.text_frame.paragraphs[0]).strip() + if head in QUADRANTS: + fill_textbox(sh.text_frame, QUADRANTS[head]) + elif head.startswith("A4I Hackathon") or sh.shape_id == 3: + set_header(sh.text_frame) + +prs.save(OUTPUT) +print(f"OK wrote {OUTPUT}") diff --git a/.agents/skills/quadrant-onepager/one-pager.pptx b/.agents/skills/quadrant-onepager/one-pager.pptx new file mode 100644 index 0000000..3893ae1 Binary files /dev/null and b/.agents/skills/quadrant-onepager/one-pager.pptx differ diff --git a/.agents/skills/quadrant-onepager/sample.png b/.agents/skills/quadrant-onepager/sample.png new file mode 100644 index 0000000..fb07c19 Binary files /dev/null and b/.agents/skills/quadrant-onepager/sample.png differ diff --git a/.agents/skills/quadrant-onepager/template.pptx b/.agents/skills/quadrant-onepager/template.pptx new file mode 100644 index 0000000..145c82a Binary files /dev/null and b/.agents/skills/quadrant-onepager/template.pptx differ diff --git a/.agents/skills/refactoring-advisor/SKILL.md b/.agents/skills/refactoring-advisor/SKILL.md new file mode 100644 index 0000000..5f5c2d8 --- /dev/null +++ b/.agents/skills/refactoring-advisor/SKILL.md @@ -0,0 +1,49 @@ +--- +name: refactoring-advisor +description: Identify maintainability issues, complexity hotspots, code duplication, and safe refactoring opportunities. +--- + +# refactoring-advisor + +You are a Principal Python Engineer focused on maintainability and clean architecture. + +Your objective is to improve code quality without changing behavior. + +## Usage + +Use this skill when: + +- Improving maintainability +- Reducing complexity +- Preparing pull requests +- Evaluating technical debt + +## Steps + +1. Scan repository structure. +2. Identify large files and functions. +3. Detect duplicated logic. +4. Detect tightly coupled modules. +5. Evaluate abstraction quality. +6. Identify complexity hotspots. +7. Recommend incremental refactors. +8. Estimate implementation effort. +9. Assess migration risk. +10. Generate implementation plans. + +## Output Format + +### Finding + +- Component: +- Issue: +- Impact: +- Proposed Refactor: +- Effort: +- Risk: + +### Refactoring Roadmap + +- Quick Wins +- Medium Effort +- High Impact \ No newline at end of file diff --git a/.agents/skills/security-test-generator/SKILL.md b/.agents/skills/security-test-generator/SKILL.md new file mode 100644 index 0000000..51ef7f4 --- /dev/null +++ b/.agents/skills/security-test-generator/SKILL.md @@ -0,0 +1,56 @@ +--- +name: security-test-generator +description: Generate comprehensive pytest-based security and validation tests for SQL validation and injection detection rules. +--- + +# security-test-generator + +You are a Senior QA Engineer specializing in security testing and automated validation. + +Your objective is to maximize security test coverage. + +## Usage + +Use this skill when: + +- Expanding test coverage +- Creating regression tests +- Validating security fixes +- Preparing hackathon deliverables + +## Steps + +1. Analyze existing test suites. +2. Identify uncovered validation rules. +3. Create a test matrix. +4. Generate positive validation tests. +5. Generate negative validation tests. +6. Generate malicious attack tests. +7. Create parameterized pytest scenarios. +8. Ensure every security finding has a regression test. +9. Organize tests by attack category. +10. Document coverage improvements. + +## Test Categories + +- SELECT +- JOIN +- UNION +- CTE +- HAVING +- ORDER BY +- LIMIT +- Nested Queries +- Subqueries +- SQL Injection Payloads +- Restriction Bypass Attempts + +## Output Format + +### Coverage Gap Analysis + +### New Test Cases + +### Regression Tests + +### Coverage Improvement Summary \ No newline at end of file diff --git a/.agents/skills/sql-injection-researcher/SKILL.md b/.agents/skills/sql-injection-researcher/SKILL.md new file mode 100644 index 0000000..cbe8c63 --- /dev/null +++ b/.agents/skills/sql-injection-researcher/SKILL.md @@ -0,0 +1,59 @@ +--- +name: sql-injection-researcher +description: Analyze SQL validation and injection detection logic, identify bypasses, generate attack payloads, and create security findings with remediation recommendations. +--- + +# sql-injection-researcher + +You are a Security Research Engineer specializing in SQL Injection detection, query validation, secure parsers, and defensive controls. + +Your objective is to evaluate the repository's ability to detect, prevent, and mitigate SQL injection attacks and authorization bypasses. + +Focus on practical findings that can be implemented during a hackathon. + +## Usage + +Use this skill when: + +- Reviewing SQL validation logic +- Assessing security controls +- Looking for injection bypasses +- Evaluating parser limitations +- Creating security test suites +- Preparing security-focused hackathon deliverables + +## Steps + +1. Identify all SQL parsing and validation components. +2. Map the validation flow from input to decision. +3. Locate injection detection rules. +4. Review supported SQL constructs. +5. Generate attack payloads covering: + - UNION attacks + - Nested queries + - Comment injection + - Stacked queries + - Boolean injections + - Encoding tricks + - Whitespace manipulation +6. Evaluate expected versus actual behavior. +7. Document false positives and false negatives. +8. Prioritize findings by severity. +9. Propose code-level fixes. +10. Generate regression tests for every finding. + +## Output Format + +### Finding + +- Severity: +- Component: +- Description: +- Attack Example: +- Risk: +- Recommended Fix: + +### Test Cases + +- Positive Tests +- Negative Tests \ No newline at end of file diff --git a/.clinerules/Security-Rule.md b/.clinerules/Security-Rule.md new file mode 100644 index 0000000..91ba580 --- /dev/null +++ b/.clinerules/Security-Rule.md @@ -0,0 +1,65 @@ +You are a Senior Security Engineer, Python Architect, and Open Source Maintainer working on the sql-data-guard project. + +Mission: +Improve security, detection coverage, code quality, test coverage, developer experience, and documentation while preserving backward compatibility. + +Primary Objectives: + +1. Understand repository architecture before proposing changes. +2. Never introduce breaking API changes without documenting them. +3. Prefer incremental improvements over large rewrites. +4. Every code modification must include: + - rationale + - security impact + - tests + - documentation updates +5. Favor AST-based validation over regex-based validation when possible. +6. Identify SQL injection bypass opportunities. +7. Identify unsupported SQL syntax and edge cases. +8. Evaluate false positives and false negatives. +9. Produce implementation-ready pull request plans. +10. Follow existing coding conventions. + +Working Principles: + +- Read code before editing. +- Search entire repository for related functionality. +- Trace execution flow end-to-end. +- Identify security assumptions. +- Validate assumptions using tests. +- Generate tests before major refactors. +- Prefer maintainable code over clever code. + +Security Mindset: + +Act like: +- security researcher +- penetration tester +- secure code reviewer + +Look for: +- SQL injection bypasses +- logical authorization bypasses +- row restriction bypasses +- column restriction bypasses +- nested query bypasses +- UNION attacks +- stacked query attacks +- comment-based attacks +- encoding tricks +- malformed SQL +- parser confusion attacks + +Deliverables: + +For every task produce: + +1. Findings +2. Root Cause +3. Recommended Fix +4. Impact +5. Test Cases +6. Documentation Updates + +Do not stop at identifying issues. +Always propose implementable improvements. \ No newline at end of file diff --git a/.clinerules/workflows/technical-documentation-workflow.md b/.clinerules/workflows/technical-documentation-workflow.md new file mode 100644 index 0000000..17267fa --- /dev/null +++ b/.clinerules/workflows/technical-documentation-workflow.md @@ -0,0 +1,356 @@ +--- + +name: technical-documentation-workflow +description: Generate comprehensive technical documentation for an existing codebase through architecture discovery, execution tracing, dependency analysis, and source-code-driven documentation generation. +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + +# technical-documentation-workflow + +You are a Senior Software Architect, Technical Writer, and Open Source Maintainer. + +Your objective is to generate accurate, maintainable, and comprehensive technical documentation for an existing codebase. + +Documentation must be derived from source code, configuration, tests, and existing project artifacts. + +Never assume implementation details. + +Always validate findings against the repository. + +--- + +# Usage + +Use this workflow when: + +* Documenting an unfamiliar codebase +* Creating architecture documentation +* Improving contributor onboarding +* Generating technical reference material +* Preparing maintenance documentation +* Understanding brownfield systems +* Documenting security-sensitive components + +--- + +# Workflow + +## Phase 1 — Repository Discovery + +### Objectives + +Build a complete understanding of the repository structure and purpose. + +### Tasks + +1. Read all README files. +2. Review existing documentation. +3. Analyze package structure. +4. Identify application entry points. +5. Identify public APIs. +6. Identify CLI interfaces. +7. Identify configuration files. +8. Identify build and deployment artifacts. +9. Identify major dependencies. +10. Create a repository inventory. + +### Deliverables + +Generate: + +* Project purpose +* Key features +* Technology stack +* Repository structure +* Dependency overview +* Build and runtime requirements + +--- + +## Phase 2 — Architecture Discovery + +### Objectives + +Understand the overall system architecture and component boundaries. + +### Tasks + +1. Identify major subsystems. +2. Identify module responsibilities. +3. Identify dependency relationships. +4. Identify external integrations. +5. Identify shared services and utilities. +6. Identify data flow paths. +7. Identify security-sensitive components. + +### Deliverables + +Generate: + +* Architecture overview +* Component catalog +* Dependency map +* Integration map +* Component responsibility matrix + +Include Mermaid diagrams when appropriate. + +--- + +## Phase 3 — Execution Flow Analysis + +### Objectives + +Understand runtime behavior and request processing. + +### Tasks + +Trace: + +1. Startup sequence +2. Configuration loading +3. Request processing +4. Validation workflows +5. Data processing workflows +6. Error handling paths +7. Shutdown behavior + +### Deliverables + +Generate: + +* Execution flow documentation +* Sequence diagrams +* Lifecycle documentation +* Error handling documentation + +Include Mermaid sequence diagrams where useful. + +--- + +## Phase 4 — Module Documentation + +### Objectives + +Document implementation details for each major module. + +### Tasks + +For every major module: + +1. Identify purpose. +2. Identify key classes. +3. Identify public interfaces. +4. Identify internal workflows. +5. Identify dependencies. +6. Identify extension points. +7. Identify design patterns. + +### Deliverables + +For each module generate: + +* Purpose +* Responsibilities +* Public API +* Internal workflow +* Dependencies +* Extension points +* Usage examples + +--- + +## Phase 5 — Data and Configuration Documentation + +### Objectives + +Document data structures and configuration mechanisms. + +### Tasks + +1. Identify configuration sources. +2. Identify environment variables. +3. Identify configuration schemas. +4. Identify data models. +5. Identify serialization formats. +6. Identify validation rules. + +### Deliverables + +Generate: + +* Configuration guide +* Environment variable reference +* Data model documentation +* Schema documentation +* Configuration examples + +Include tables for all configuration settings. + +--- + +## Phase 6 — Security Documentation + +### Objectives + +Document security-relevant aspects of the system. + +### Tasks + +1. Identify trust boundaries. +2. Identify validation layers. +3. Identify authentication mechanisms. +4. Identify authorization mechanisms. +5. Identify security controls. +6. Identify attack surfaces. +7. Identify security assumptions. + +### Deliverables + +Generate: + +* Security architecture overview +* Trust boundary documentation +* Validation strategy +* Threat considerations +* Security assumptions +* Known limitations + +--- + +## Phase 7 — Testing Documentation + +### Objectives + +Document testing strategy and quality assurance processes. + +### Tasks + +1. Analyze test structure. +2. Identify testing frameworks. +3. Identify test categories. +4. Identify coverage areas. +5. Identify testing gaps. + +### Deliverables + +Generate: + +* Testing strategy +* Test organization +* Coverage overview +* Running tests guide +* Writing tests guide + +--- + +## Phase 8 — Contributor Documentation + +### Objectives + +Enable efficient onboarding and contribution. + +### Tasks + +Document: + +1. Local development setup. +2. Build process. +3. Development workflow. +4. Debugging workflow. +5. Coding standards. +6. Contribution process. + +### Deliverables + +Generate: + +* Contributor guide +* Development setup guide +* Debugging guide +* Coding conventions +* Contribution workflow + +--- + +## Phase 9 — Technical Debt Assessment + +### Objectives + +Identify maintainability and documentation improvement opportunities. + +### Tasks + +Analyze: + +1. Large modules. +2. Complex functions. +3. High-coupling areas. +4. Duplicate logic. +5. Missing tests. +6. Missing documentation. + +### Deliverables + +Generate: + +* Technical debt report +* Maintainability observations +* Refactoring opportunities +* Documentation improvement recommendations + +--- + +## Phase 10 — Documentation Package Generation + +### Objectives + +Generate a complete documentation set suitable for long-term maintenance. + +### Deliverables + +Generate the following artifacts when applicable: + +* PROJECT_OVERVIEW.md +* ARCHITECTURE.md +* EXECUTION_FLOW.md +* MODULE_REFERENCE.md +* CONFIGURATION_GUIDE.md +* SECURITY_ARCHITECTURE.md +* TESTING_GUIDE.md +* CONTRIBUTOR_GUIDE.md +* TECHNICAL_DEBT_REPORT.md + +Include: + +* Architecture diagrams +* Sequence diagrams +* Dependency diagrams +* Data flow diagrams +* Component relationship diagrams + +--- + +# Quality Requirements + +Documentation must: + +* Be derived from implementation. +* Reference actual modules, classes, and functions. +* Distinguish facts from assumptions. +* Explain design rationale when discoverable. +* Highlight extension points. +* Document dependencies and interactions. +* Be maintainable and contributor-friendly. +* Remain technology-agnostic where practical. + +Before generating documentation: + +1. Verify all referenced components exist. +2. Verify diagrams match implementation. +3. Verify dependencies are accurate. +4. Verify workflows match source code. +5. Explicitly identify any assumptions or uncertainties. + +Never invent architecture. + +Always validate findings against the repository. diff --git a/AI_REVIEW_WORKFLOW_PROMPTS.md b/AI_REVIEW_WORKFLOW_PROMPTS.md new file mode 100644 index 0000000..49f1c62 --- /dev/null +++ b/AI_REVIEW_WORKFLOW_PROMPTS.md @@ -0,0 +1,179 @@ + + +## PROMPT 1 — Analyze the project & create/update the Memory Bank (Cline format) + +``` +ROLE: Senior software architect doing brownfield analysis. +GOAL: Produce a source-verified Cline-format Memory Bank so any agent can rebuild context fast. +INPUTS: The repository. Optional: if a skill/workflow "architecture-discovery" or + "technical-documentation-workflow" exists (.agents/skills/ or .clinerules/), use it. +METHOD: + 1. Map repo: READMEs, layout, entry points, public APIs/CLIs, config, build/deploy, deps, tests. + 2. Read core source end-to-end; trace the main execution flow(s); mark security-critical + components, trust boundaries, design patterns. + 3. VERIFY empirically — don't trust docs/comments: run the test suite, grep to confirm + features exist, record exact tool/library versions, note advertised-vs-actual gaps. + 4. Write ./memory-bank/ with six files: projectbrief.md, productContext.md, systemPatterns.md, + techContext.md, activeContext.md, progress.md (purpose / flows / decisions / known issues). +CONSTRAINTS: Prefer grep/glob over full reads; cite file:line, don't paste code; for broad + exploration dispatch parallel read-only sub-agents returning conclusions only. Distinguish + FACTS (verified) from ASSUMPTIONS. No code changes. No commits. +OUTPUT: the six files + a <=15-line chat summary (architecture + top 3-5 risks/gaps). +STOP WHEN: the six files exist and the summary is posted. +``` + +--- + +## PROMPT 2 — Find correctness & security issues + +``` +ROLE: Security researcher + secure-code reviewer + penetration tester. +GOAL: A prioritized, de-duplicated backlog of real correctness & security issues. +INPUTS: ./memory-bank/ if present (else skim code). Optional skills: "sql-injection-researcher", + "refactoring-advisor", or a security-persona/"Security-Rule" file — use if present. +METHOD: + 1. Read code before judging; trace each security decision end-to-end; reason over the + AST/parse tree, not string/regex matching. + 2. SECURITY: injection bypasses (UNION/EXCEPT/INTERSECT, sub-queries, stacked/multi-statement, + comment & encoding evasion, parser/dialect confusion), authz bypasses (row/column/allow-list), + untrusted input reaching a sink unsanitized. + 3. CORRECTNESS/ROBUSTNESS: validation that crashes the caller (uncaught exceptions/500s), + wrong comparisons (string vs numeric), non-deterministic output, dead code, import-time + failures, build/packaging defects. + 4. Per issue: stable ID, Title, Severity, Component(file:line), Description, Root Cause, + concrete repro/attack example, Risk/Impact, Recommended Fix direction. + 5. De-duplicate; order by priority (crashes/security first); tag "code-confirmed" vs "suspected". +CONSTRAINTS: Targeted search over full reads; cite file:line; don't fabricate. No code changes yet. +OUTPUT: write the backlog table to ./ISSUES_AND_FINDINGS.md (do NOT commit) + a <=15-line summary + (counts by severity, the top fixes). +STOP WHEN: the backlog file is written and summarized. +``` + +--- + +## PROMPT 3 — Replicate findings against the running local app (real vs false positive) + +``` +ROLE: Verification engineer. +GOAL: Empirically classify each finding as VALIDATED / FALSE POSITIVE / NEEDS-INVESTIGATION. +INPUTS: ./ISSUES_AND_FINDINGS.md, ./memory-bank/. +CRITICAL: run LOCAL source, never a published/installed package + (Python: `pip install -e .` or `PYTHONPATH=src ...`; launch the API/CLI on local code and + confirm a local-only marker/fix is observable). +METHOD (per finding): + 1. Build the MINIMAL reproducing input (API payload, function call, or test). + 2. Run it; capture the EXACT actual output (status, returned object, error). + 3. Compare to expected; classify with the captured evidence. + 4. Be precise about version-/dialect-specific behavior; if a bug is only blocked incidentally + (not by an explicit control), say so. Don't over-claim. +CONSTRAINTS: Reference payloads by ID; don't paste large outputs verbatim — trim to the deciding + lines. No code changes. +OUTPUT: write ./REPLICATION_REPORT.md (do NOT commit): per finding — payload, observed output, + verdict, one-line root-cause confirmation; plus validated/false-positive counts in chat. +STOP WHEN: every finding has a verdict with evidence. +``` + +--- + +## PROMPT 4 — Propose a fix plan; implement fixes + unit tests after approval + +``` +ROLE: Senior engineer (security-aware), TDD-minded. +GOAL: A reviewable fix plan, then — only after approval — the fixes plus regression tests. +INPUTS: VALIDATED findings in ./REPLICATION_REPORT.md. Optional skills: "security-test-generator", + "refactoring-advisor". +PHASE A — PLAN (no code changes): per issue — fix approach, exact files/functions, backward-compat + & risk notes, the regression test(s) to add; group + order; flag any test files whose + expectations will change. Then STOP and wait for approval. +PHASE B — IMPLEMENT (only after approval): + 1. Read the affected tests BEFORE editing so you don't silently break expectations. + 2. Incremental, backward-compatible changes; AST-based over regex; keep the public contract stable. + 3. Every fix gets a regression test (positive + negative) that fails before / passes after. + 4. After each batch run the FULL suite; fix regressions before continuing. + 5. Per fix follow: Finding -> Root Cause -> Fix -> Impact -> Tests -> Docs (update + ./memory-bank/progress.md). +CONSTRAINTS: Minimal diffs; cite file:line in the plan, don't paste whole files. Don't commit. +OUTPUT: Phase A = the plan (chat). Phase B = diffs + new/updated test file + a green test run summary. +STOP WHEN: Phase A awaits approval; Phase B ends on a green suite. +``` + +--- + +## PROMPT 5 — Re-validate the fixes by building & running the app + +``` +ROLE: QA / release validator. +GOAL: Prove the fixes work end-to-end and nothing regressed. +INPUTS: ./REPLICATION_REPORT.md, the test suite, the local app. +METHOD: + 1. Run the FULL suite on local code (e.g. `PYTHONPATH=src python -m pytest + --ignore=`); report pass/skip/fail counts faithfully. + 2. Re-launch the app on local code (as Stage 3) and re-run the EXACT replication payloads; + build a before->after table proving each VALIDATED issue is fixed AND legit inputs still + work (no new false positives). + 3. Spot-check integration surfaces (REST/CLI/plugin) for the headline fixes. +CONSTRAINTS: Report real numbers; never paper over a failure — stop and show the output. +OUTPUT: test summary + before/after verification table (chat). No commits. +STOP WHEN: suite is green and the before/after table is complete. +``` + +--- + +## PROMPT 6 — Commit, push, and raise the MR/PR + +``` +ROLE: Release engineer. +GOAL: Ship the work on a branch with a complete PR/MR. +INPUTS: the working tree, Stage-5 evidence, ./REPLICATION_REPORT.md. +METHOD: + 1. Never commit to the default branch — create `fix/`. + 2. Stage ONLY source + test changes (list paths explicitly; exclude *.md and memory-bank/ + unless I say otherwise). Show `git status` so I can confirm nothing unintended is staged. + 3. Commit: one-line summary + body grouping every issue fixed (by ID) with its one-line fix + + the test result + any repo-required trailer/sign-off. + 4. Push + set upstream. Prefer `gh`/`glab` if available, else `git push`; if auth needs an + interactive prompt you can't complete, STOP and hand me the exact command. + 5. Open the PR/MR with: Summary, Issues-fixed tables by category, mapping to analysis docs, + the Stage-5 before/after evidence, Tests, reviewer notes, out-of-scope items. Return the URL. +CONSTRAINTS: If you obtain a token (e.g. `git credential fill`) NEVER print it. Code + tests only. +OUTPUT: branch name, commit hash, PR/MR URL. +STOP WHEN: the PR/MR is open and its URL is returned. +``` + +--- + +## Will the existing skills/workflows help? + +Yes — as **optional accelerators**. The prompts already embed the method, so these are speed-ups, not requirements. + +| Stage | Skills / workflows that help | Reusability | +|------|------------------------------|-------------| +| 1 Analyze + Memory Bank | `architecture-discovery`, `technical-documentation-workflow`, `documentation-engineer` | `technical-documentation-workflow` is generic; others lightly project-tuned | +| 2 Find issues / security | `sql-injection-researcher`, `refactoring-advisor`, `Security-Rule.md` (persona + Finding→Root Cause→Fix→Impact→Tests→Docs deliverable shape) | `refactoring-advisor` generic; injection one is SQL-specific | +| 3 Replicate locally | `benchmark-suite-builder` (attack-corpus harness) | project-tuned | +| 4 Plan + implement + tests | `security-test-generator`, `refactoring-advisor`, `Security-Rule.md` | test-gen SQL-tuned; advisor generic | +| 5 Re-validate | `benchmark-suite-builder`, `security-test-generator` | as above | +| 6 Commit / push / MR | none (handled by the prompt) | n/a | + +**How they're invoked:** Cline auto-loads `.clinerules/` and can run `.agents/skills/*`; Claude Code invokes skills via its Skill tool / subagents. Either way the prompts say *"if such a skill exists, use it; otherwise follow the steps below,"* so they degrade gracefully. + +**Housekeeping (worth a separate cleanup):** +- `.agents/skills/sql-injection-reseacher/` is a **misspelled duplicate** of `sql-injection-researcher/` — delete it. +- The correctly-spelled `sql-injection-researcher/SKILL.md` has trailing junk on its last line (`-sta-console-nakul.`) — trim it. + +## Will these prompts work on a clean project (no md / clean branch)? + +**Yes**, by design — with three things to know: +1. **Self-contained.** Stage 1 *creates* the Memory Bank from scratch (no pre-existing `.md` needed); Stages 2–3 generate their own `ISSUES_AND_FINDINGS.md` / `REPLICATION_REPORT.md`. The optional-skill lines are no-ops when `.agents/.clinerules` are absent. To get the accelerators on a new project, copy the `.agents/` and `.clinerules/` folders across. +2. **Stages 3 & 5 need a runnable app.** Dependencies must be installed, and the prompts force running **local code, not the published package** (the `PYTHONPATH=src` / `pip install -e .` gotcha — otherwise you'll test the released version and miss your own fixes). For non-Python stacks, replace the run/test commands. +3. **Clean branch / no-md preference honored.** Stage 6 commits **code + tests only** and branches off the default branch; the Memory Bank and analysis artifacts stay uncommitted (add them to `.git/info/exclude` if you want them ignored locally). + +## Appendix — stage → artifact map +| Stage | Reads | Writes | +|------|-------|--------| +| 1 | repo | `memory-bank/` (6 files) | +| 2 | `memory-bank/` | `ISSUES_AND_FINDINGS.md` | +| 3 | `ISSUES_AND_FINDINGS.md` | `REPLICATION_REPORT.md` | +| 4 | `REPLICATION_REPORT.md` | code + tests; updates `memory-bank/progress.md` | +| 5 | `REPLICATION_REPORT.md`, tests, app | before/after evidence (chat) | +| 6 | working tree, Stage-5 evidence | branch, commit, PR/MR URL | diff --git a/Dockerfile b/Dockerfile index c4b49e6..87d68e5 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,6 @@ FROM python:3.12-alpine COPY requirements.txt . -RUN pip install flask sql_data_guard +RUN pip install flask flasgger sql_data_guard WORKDIR /app/ COPY src/sql_data_guard/rest/sql_data_guard_rest.py . COPY src/sql_data_guard/rest/logging.conf . diff --git a/ONBOARDING.md b/ONBOARDING.md new file mode 100644 index 0000000..4b6dbf2 --- /dev/null +++ b/ONBOARDING.md @@ -0,0 +1,237 @@ +

    🛡️ Welcome to sql-data-guard

    +

    From zero to your first guarded SQL query — in under 5 minutes.

    + +

    + Setup time ~5 minutes + Python 3.8+ + Beginner friendly + MIT License +

    + +> [!NOTE] +> **New here?** This is your map. Follow it top-to-bottom and you'll have the project installed, a query verified, and the tests passing — before your coffee gets cold. ☕ + +--- + +## 🗺️ Your onboarding journey + +```mermaid +flowchart LR + A([👋 Start]) --> B[📦 Install] + B --> C[✅ Verify setup] + C --> D{🎯 Pick your path} + D -->|Code| E[🐍 Python library] + D -->|HTTP| F[🌐 REST API] + D -->|LLM tools| G[🔌 MCP / Dify] + E --> H([🎉 First guarded query!]) + F --> H + G --> H + + classDef start fill:#22c55e,stroke:#16a34a,color:#fff,font-weight:bold + classDef step fill:#3b82f6,stroke:#2563eb,color:#fff + classDef choice fill:#f59e0b,stroke:#d97706,color:#fff,font-weight:bold + classDef path fill:#8b5cf6,stroke:#7c3aed,color:#fff + classDef done fill:#ec4899,stroke:#db2777,color:#fff,font-weight:bold + + class A start + class B,C step + class D choice + class E,F,G path + class H done +``` + +--- + +## 1️⃣ What is this, in one breath? + +> [!TIP] +> **sql-data-guard checks an SQL query against an allow-list policy *before* it hits your database — and rewrites it if it breaks the rules.** Perfect for taming SQL that LLMs generate. 🤖→🛡️→🗄️ + +--- + +## 2️⃣ Prerequisites + +Tick these off before you start: + +- [ ] 🐍 **Python ≥ 3.8** installed → check with `python --version` +- [ ] 📥 **pip** available → check with `pip --version` +- [ ] 🌿 **git** (only if installing from source) +- [ ] 🐳 **Docker** *(optional — only for the REST API container)* + +> [!NOTE] +> The only runtime dependency is [`sqlglot`](https://github.com/tobymao/sqlglot). No database, no heavy stack. 🪶 + +--- + +## 3️⃣ Install & verify (the 60-second setup) + +```bash +# Install from PyPI +pip install sql-data-guard + +# Verify it imported correctly +python -c "from sql_data_guard import verify_sql; print('✅ sql-data-guard is ready!')" +``` + +
    +🛠️ Prefer installing from source? (for contributors) + +```bash +git clone https://github.com/ThalesGroup/sql-data-guard.git +cd sql-data-guard +pip install -e . +pip install -r test/test.requirements.txt +``` + +
    + +--- + +## 4️⃣ Your first win 🎉 + +Paste this into a file (`hello_guard.py`) and run it: + +```python +from sql_data_guard import verify_sql + +# 🎯 The policy: only these columns, only this account's rows +config = { + "tables": [ + { + "table_name": "orders", + "columns": ["id", "product_name", "account_id"], + "restrictions": [{"column": "account_id", "value": 123}], + } + ] +} + +# 🚨 A sketchy LLM-generated query (restricted column + always-true injection) +query = "SELECT id, name FROM orders WHERE 1 = 1" + +result = verify_sql(query, config) +print(result) +``` + +You'll see the guard **catch the problems and hand you a safe rewrite**: + +```json +{ + "allowed": false, + "errors": [ + "Column name not allowed. Column removed from SELECT clause", + "Always-True expression is not allowed", + "Missing restriction for table: orders column: account_id value: 123" + ], + "fixed": "SELECT id, product_name, account_id FROM orders WHERE account_id = 123", + "risk": 0.7 +} +``` + +> [!IMPORTANT] +> 🧠 **The mental model:** you give it `sql` + `config`, it gives you back `allowed`, `errors`, a `fixed` query, and a `risk` score (0 = safe → 1 = dangerous). That's the whole API. + +--- + +## 5️⃣ Pick your path 🎯 + +Choose the integration that matches how *you* build: + +| Path | You want to… | Go here | +|---|---|---| +| 🐍 **Python library** | Embed checks directly in app code | [`src/sql_data_guard/`](src/sql_data_guard/) · [README](README.md#usage-and-examples) | +| 🌐 **REST API** | Call it from any language / container | [`src/sql_data_guard/rest/`](src/sql_data_guard/rest/) · see below | +| 🔌 **MCP wrapper** | Guard an MCP database server | [`examples/mcp-wrapper-sqlite/`](examples/mcp-wrapper-sqlite/) | +| 🧩 **Dify plugin** | Add it to a Dify LLM workflow | [`plugins/dify/`](plugins/dify/README.md) | + +
    +🌐 Spin up the REST API + Swagger UI + +```bash +pip install flask flasgger sql-data-guard +APP_PORT=5050 PYTHONPATH=src python src/sql_data_guard/rest/sql_data_guard_rest.py +``` + +Then open the interactive playground 👉 **http://localhost:5050/apidocs** + +Or run it as a container: + +```bash +docker run -d -p 5000:5000 ghcr.io/thalesgroup/sql-data-guard +``` + +
    + +--- + +## 6️⃣ Run the tests ✅ + +Confirm everything works on your machine: + +```bash +PYTHONPATH=src python -m pytest --color=yes test/*_unit.py +``` + +
    +🪟 On Windows (cmd)? + +```bat +set PYTHONPATH=src +python -m pytest --color=yes test\*_unit.py +``` + +
    + +Green output means you're fully set up. 🟢 + +--- + +## 7️⃣ Where things live 🧭 + +```text +sql-data-guard/ +├── 🧠 src/sql_data_guard/ → the core engine (verify_sql lives here) +│ ├── rest/ → 🌐 Flask REST API + Swagger +│ └── mcpwrapper/ → 🔌 MCP server guard +├── 🧩 plugins/dify/ → Dify LLM-workflow plugin +├── 📚 docs/manual.md → restriction rules & operations reference +├── 🧪 test/ → the test suite +└── 📖 README.md → the full project docs +``` + +> [!TIP] +> Want the deep dive on restriction rules (`BETWEEN`, `IN`, comparisons)? → [`docs/manual.md`](docs/manual.md) + +--- + +## 8️⃣ Common gotchas 🩹 + +> [!WARNING] +> **Local REST calls return `503` / a Squid error page?** +> A corporate proxy is intercepting localhost. Bypass it: add `localhost, 127.0.0.1` to your proxy exceptions, or use `curl --noproxy "*"`. + +> [!WARNING] +> **`ModuleNotFoundError` when running from source?** +> You forgot `PYTHONPATH=src`. Prefix your command with it (or `set PYTHONPATH=src` on Windows). + +> [!WARNING] +> **Need a non-default port for the REST API?** +> Set the `APP_PORT` environment variable (defaults to `5000`). + +--- + +## 9️⃣ You're onboarded! What's next? 🚀 + +| I want to… | Go to | +|---|---| +| 📖 Read the full feature set | [README.md](README.md) | +| 📚 Understand every restriction rule | [docs/manual.md](docs/manual.md) | +| 🤝 Contribute code | [CONTRIBUTING.md](CONTRIBUTING.md) | +| 🔒 Report a security issue | [SECURITY.md](SECURITY.md) · security@opensource.thalesgroup.com | +| 🐛 File a bug or idea | [GitHub Issues](https://github.com/ThalesGroup/sql-data-guard/issues) | + +--- + +

    + Welcome aboard — go guard some queries! 🛡️
    + Stuck for more than 15 minutes? Open an issue. A good question makes the docs better for the next person. 💚 +

    diff --git a/README.md b/README.md index 985029a..fd3d50c 100644 --- a/README.md +++ b/README.md @@ -1,47 +1,170 @@ +

    sql-data-guard

    +

    A safety layer that verifies and rewrites SQL queries before they touch your database — built for the LLM era.

    -# sql-data-guard: Safety Layer for LLM Database Interactions +

    + SQL Data Guard logo +

    -
    - SQL Data Guard logo -
    +

    + PyPI + Python versions + Docker image + License: MIT +

    + +> [!TIP] +> 🚀 **New to sql-data-guard? Start here → [ONBOARDING.md](ONBOARDING.md)** — a colorful, step-by-step quick guide that gets you installed, runs your first guarded query, and points you to the right integration path in **under 5 minutes**. + +--- + +## Table of contents + +- [Overview](#overview) +- [Key features](#key-features) +- [How it works](#how-it-works) +- [Architecture](#architecture) +- [Quick start](#quick-start) +- [Installation](#installation) +- [Usage and examples](#usage-and-examples) +- [Configuration](#configuration) +- [Policy and security controls](#policy-and-security-controls) +- [REST API reference](#rest-api-reference) +- [MCP wrapper](#mcp-wrapper) +- [Dify plugin](#dify-plugin) +- [Project structure](#project-structure) +- [Development and contributing](#development-and-contributing) +- [AI-assisted development](#ai-assisted-development) +- [Testing](#testing) +- [Deployment and release](#deployment-and-release) +- [Security](#security) +- [FAQ and troubleshooting](#faq-and-troubleshooting) +- [License](#license) +- [Contact and support](#contact-and-support) + +--- + +## Overview + +**What.** `sql-data-guard` is an open-source Python library and service that inspects an SQL query against a declarative restriction configuration, decides whether the query is allowed to run, and — when it is not — rewrites it into a compliant form. + +**Why.** SQL is easy to use and just as easy to exploit. SQL injection remains one of the most targeted vulnerabilities, and the problem is amplified by *natural-language-to-SQL* features built on Large Language Models (LLMs). Prepared statements secure a query's *structure*, but LLM-generated queries are dynamic and have no fixed form, so they cannot be parameterized — leaving the door open to accidental data exposure and injection. `sql-data-guard` closes that gap by validating the query's *content* before execution. + +**Who it is for.** Teams whose applications build SQL dynamically — especially with LLMs — and who need fine-grained, column-level and row-level access control that the database permission model cannot express. Typical cases: + +- Applications that generate complex SQL queries. +- Applications that use LLMs to author SQL, where full query control is impractical. +- Per-user / per-role data access that must be correlated with fine-grained permissions. +- Multi-tenant apps needing row-level isolation that database permissions can't enforce. + +**Non-goals.** `sql-data-guard` does **not** replace the database permission model. It is an *additional* layer for the restrictions that are complex, vendor-specific, or otherwise impossible to express at the database level. + +--- + +## Key features + +**Access and query control** +- **Verify** any SQL query against an allow-list of tables, columns, and restrictions. +- **Rewrite** non-compliant queries automatically into a safe, allowed form. +- **Enforce row-level security** by injecting missing restrictions (e.g. `account_id = 123`). +- **Deny columns** explicitly (`denied_columns`) even when otherwise allowed, and expand `SELECT *` to the allow-list. +- **Cap result rows** with `force_limit` — inject or clamp the outermost `LIMIT`. + +**Data protection** +- **Mask sensitive columns** (`redact`, `hash`, `partial`) instead of dropping them, preserving the result shape for downstream apps. + +**Threat detection** +- **Detect injection** — stacked statements, dangerous functions, system-catalog probing, and always-true (`1 = 1`) expressions; opt-in comment-evasion scanning. +- **Allow/Block SQL functions** by policy (`allowed_functions` / `blocked_functions`). +- **Block** disallowed statements (`INSERT`, `UPDATE`, `DELETE`, `CREATE`, DDL/commands). +- **Score risk** for every query (`0.0` → `1.0`) and **hard-block** above an optional `max_risk` threshold. + +**Integration** +- **Integrate anywhere** — Python API, REST service with Swagger UI, MCP wrapper, or Dify plugin. +- **Multi-dialect** parsing via [sqlglot](https://github.com/tobymao/sqlglot) (SQLite, PostgreSQL, and more). + +--- + +## How it works + +1. **Input** — a SQL query string plus a restriction configuration (allowed tables, columns, restrictions, and optional policy controls). +2. **Validation** — the configuration itself is validated (supported operations, well-formed restrictions and column masks). +3. **Threat scan** — the raw string and parsed AST are scanned for injection patterns (stacked statements, dangerous functions, system-catalog probing, opt-in comment evasion). +4. **Verification** — the query is checked against the configuration: disallowed statements, unknown tables/columns, denied columns, `SELECT *`, static/always-true expressions, function policy, and missing row restrictions. +5. **Modification** — where possible, the query is rewritten to comply: remove denied/disallowed columns, expand `*`, mask sensitive columns, drop injection expressions, append required restrictions, and enforce the row cap. +6. **Output** — a result object with `allowed`, `errors`, `fixed`, and `risk`. + +--- + +## Architecture + +*High-level flow — a query and a policy go in; an allow/deny decision and an optional rewritten query come out.* + +```mermaid +flowchart LR + App["Application / LLM"] -->|"SQL + config"| Guard + subgraph Guard["sql-data-guard"] + V["validate config
    (restrictions, masks)"] --> S["scan raw + AST
    (injection detection)"] + S --> P["sqlglot parse"] + P --> C["verify
    (tables, columns, rows,
    functions, denials)"] + C --> R["rewrite / mask / limit"] + end + Guard -->|"allowed? errors, fixed, risk"| App + App -->|"only compliant SQL"| DB[(Database)] +``` -SQL is the go-to language for performing queries on databases, and for a good reason - it’s well known, easy to use and pretty simple. However, it seems that it’s as easy to use as it is to exploit, and SQL injection is still one of the most targeted vulnerabilities - especially nowadays with the proliferation of “natural language queries” harnessing Large Language Models (LLMs) power to generate and run SQL queries. +Integration surfaces, all sharing the same core `verify_sql`: +| Surface | Module / location | Use when | +|---|---|---| +| Python library | [src/sql_data_guard/](src/sql_data_guard/) | Embedding directly in app code | +| REST API (Flask + Swagger) | [src/sql_data_guard/rest/](src/sql_data_guard/rest/) | Language-agnostic / containerized use | +| MCP wrapper | [src/sql_data_guard/mcpwrapper/](src/sql_data_guard/mcpwrapper/) | Guarding an MCP database server | +| Dify plugin | [plugins/dify/](plugins/dify/) | Dify LLM workflows | -To help solve this problem, we developed sql-data-guard, an open-source project designed to verify that SQL queries access only the data they are allowed to. It takes a query and a restriction configuration, and returns whether the query is allowed to run or not. Additionally, it can modify the query to ensure it complies with the restrictions. sql-data-guard has also a built-in module for detection of malicious payloads, allowing it to report on and remove malicious expressions before query execution. +--- +## Quick start -sql-data-guard is particularly useful when constructing SQL queries with LLMs, as such queries can’t run as prepared statements. Prepared statements secure a query’s structure, but LLM-generated queries are dynamic and lack this fixed form, increasing SQL injection risk. sql-data-guard mitigates this by inspecting and validating the query's content. +```bash +# Install +pip install sql-data-guard +``` +```python +from sql_data_guard import verify_sql -By verifying and modifying queries before they are executed, sql-data-guard helps prevent unauthorized data access and accidental data exposure. Adding sql-data-guard to your application can prevent or minimize data breaches and the impact of SQL injection attacks, ensuring that only permitted data is accessed. +config = {"tables": [{"table_name": "orders", "columns": ["id"], + "restrictions": [{"column": "account_id", "value": 123}]}]} +print(verify_sql("SELECT id FROM orders WHERE account_id = 123", config)) +# {'allowed': True, 'errors': [], 'fixed': None, 'risk': 0.0} +``` -Connecting LLMs to SQL databases without strict controls can risk accidental data exposure, as models may generate SQL queries that access sensitive information. OWASP highlights cases of poor sandboxing leading to unauthorized disclosures, emphasizing the need for clear access controls and prompt validation. Businesses should adopt rigorous access restrictions, regular audits, and robust API security, especially to comply with privacy laws and regulations like GDPR and CCPA, which penalize unauthorized data exposure. +--- -## Why Use sql-data-guard? +## Installation -Consider using sql-guard if your application constructs SQL queries, and you need to ensure that only permitted data is accessed. This is particularly beneficial if: -- Your application generates complex SQL queries. -- Your application employs LLM (Large Language Models) to create SQL queries, making it difficult to fully control the queries. -- Different application users and roles should have different permissions, and you need to correlate an application user or role with fine-grained data access permission. -- In multi-tenant applications, you need to ensure that each tenant can access only their data, which requires row-level security and often cannot be done using the database permissions model. +**Prerequisites:** Python ≥ 3.8. The only runtime dependency is [`sqlglot`](https://github.com/tobymao/sqlglot). -sql-guard does not replace the database permissions model. Instead, it adds an extra layer of security, which is crucial when implementing fine-grained, column-level, and row-level security is challenging or impossible. -Data restrictions are often complex and cannot be expressed by the database permissions model. For instance, you may need to restrict access to specific columns or rows based on intricate business logic, which many database implementations do not support. Instead of relying on the database to enforce these restrictions, sql-guard helps you overcome vendor-specific limitations by verifying and modifying queries before they are executed. +| Method | Command | +|---|---| +| pip | `pip install sql-data-guard` | +| Docker (REST API) | `docker run -d -p 5000:5000 ghcr.io/thalesgroup/sql-data-guard` | +| From source | `git clone https://github.com/ThalesGroup/sql-data-guard.git && cd sql-data-guard && pip install -e .` | -## How It Works +Verify the library import: -1. **Input**: sql-data-guard takes an SQL query and a restriction configuration as input. -2. **Verification**: It verifies whether the query complies with the restrictions specified in the configuration. -3. **Modification**: If the query does not comply, sql-data-guard can modify the query to ensure it meets the restrictions. -4. **Output**: It returns whether the query is allowed to run or not, and if necessary, the modified query. +```bash +python -c "from sql_data_guard import verify_sql; print('ok')" +``` -sql-data-guard is designed to be easy to integrate into your application. It provides a simple API that you can call to verify and modify SQL queries before they are executed. You can integrate it using REST API or directly in your application code. +--- -## Example +## Usage and examples -Below you can find a Python snippet with allowed data access configuration, and usage of sql-data-guard. sql-data-guard finds a restricted column and an “always-true” possible injection and removes them both. It also adds a missing data restriction: +### Library + +The example below shows `sql-data-guard` finding a restricted column and an always-true injection, removing both, and injecting a missing row restriction: ```python from sql_data_guard import verify_sql @@ -51,61 +174,188 @@ config = { { "table_name": "orders", "columns": ["id", "product_name", "account_id"], - "restrictions": [{"column": "account_id", "value": 123}] + "restrictions": [{"column": "account_id", "value": 123}], } - ] + ] } query = "SELECT id, name FROM orders WHERE 1 = 1" result = verify_sql(query, config) print(result) ``` + Output: + ```json { - "allowed": false, - "errors": ["Column name not allowed. Column removed from SELECT clause", - "Always-True expression is not allowed", "Missing restriction for table: orders column: account_id value: 123"], - "fixed": "SELECT id, product_name, account_id FROM orders WHERE account_id = 123" + "allowed": false, + "errors": [ + "Column name not allowed. Column removed from SELECT clause", + "Always-True expression is not allowed", + "Missing restriction for table: orders column: account_id value: 123" + ], + "fixed": "SELECT id, product_name, account_id FROM orders WHERE account_id = 123", + "risk": 0.7 } ``` -For more details on restriction rules and validation, see the [manual.](docs/manual.md) +`verify_sql(sql, config, dialect=None)` returns a dict with: -Here is a table with more examples of SQL queries and their corresponding JSON outputs: +| Field | Type | Description | +|---|---|---| +| `allowed` | bool | Whether the query passes all restrictions as-is. | +| `errors` | list[str] | Human-readable violations found. | +| `fixed` | str \| null | A rewritten compliant query, or `null` if no fix was needed/possible. | +| `risk` | float | Risk score, `0.0` (safe) → `1.0` (high risk). | -| SQL Query | JSON Output | -|---------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| SELECT id, product_name FROM orders WHERE account_id = 123 | { "allowed": true, "errors": [], "fixed": null } | -| SELECT id FROM orders WHERE account_id = 456 | { "allowed": false, "errors": ["Missing restriction for table: orders column: account_id value: 123"], "fixed": "SELECT id FROM orders WHERE account_id = 456 AND account_id = 123" } | -| SELECT id, col FROM orders WHERE account_id = 123 | { "allowed": false, "errors": ["Column col is not allowed. Column removed from SELECT clause"], "fixed": "SELECT id FROM orders WHERE account_id = 123" } ``` | -| SELECT id FROM orders WHERE account_id = 123 OR 1 = 1 | { "allowed": false, "errors": ["Always-True expression is not allowed"], "fixed": "SELECT id FROM orders WHERE account_id = 123" } | -|SELECT * FROM orders WHERE account_id = 123| {"allowed": false, "errors": ["SELECT * is not allowed"], "fixed": "SELECT id, product_name, account_id FROM orders WHERE account_id = 123"} | +### More examples -This table provides a variety of SQL queries and their corresponding JSON outputs, demonstrating how `sql-data-guard` handles different scenarios. +| SQL query | Result (summary) | +|---|---| +| `SELECT id, product_name FROM orders WHERE account_id = 123` | `allowed: true`, no fix needed | +| `SELECT id FROM orders WHERE account_id = 456` | `allowed: false` — appends `AND account_id = 123` | +| `SELECT id, col FROM orders WHERE account_id = 123` | `allowed: false` — removes disallowed `col` | +| `SELECT id FROM orders WHERE account_id = 123 OR 1 = 1` | `allowed: false` — strips always-true expression | +| `SELECT * FROM orders WHERE account_id = 123` | `allowed: false` — expands `*` to allowed columns | -## Installation -To install sql-data-guard, use pip: +See the [restriction manual](docs/manual.md) for the full set of rules and validation behavior. -```bash -pip install sql-data-guard +--- + +## Configuration + +The configuration is a dict with a top-level `tables` list. Each table declares its allowed `columns` and optional `restrictions`. A top-level `max_length` (default `10000`) caps the accepted SQL string length. + +```json +{ + "max_length": 10000, + "tables": [ + { + "table_name": "orders", + "columns": ["id", "product_name", "account_id"], + "restrictions": [ + { "column": "account_id", "value": 123 }, + { "column": "price", "operation": "BETWEEN", "values": [100, 200] } + ] + } + ] +} ``` -## Docker Repository +### Restriction operations + +| Operation | Field | Meaning | +|---|---|---| +| `=` (default) | `value` | Column must equal the value | +| `>` `<` `>=` `<=` | `value` | Numeric comparison against a single value | +| `BETWEEN` | `values` (two numbers, low < high) | Column within an inclusive range | +| `IN` | `values` (list, consistent type) | Column matches one of the listed values | + +Validation errors are raised for unsupported operations (`UnsupportedRestrictionError`), missing `table_name`/`columns`, and invalid value types. Full details in [docs/manual.md](docs/manual.md). + +--- + +## Policy and security controls + +Beyond table/column allow-listing, `sql-data-guard` offers a set of **optional, declarative policy controls**. Everything except `tables` is optional, so existing configurations keep working unchanged. + +### Top-level keys + +| Key | Type | Default | Effect | +|---|---|---|---| +| `tables` | list | — (required) | Allowed tables and their `columns`, `restrictions`, etc. | +| `max_length` | int | `10000` | Reject SQL longer than this many characters. | +| `force_limit` | int | — | Inject or clamp the outermost `LIMIT` to this row cap. | +| `allowed_functions` | list[str] | — | If set, **only** these functions may be called (case-insensitive). | +| `blocked_functions` | list[str] | — | These functions are always blocked (case-insensitive). | +| `detect_comments` | bool | `false` | Opt-in: flag comment-based evasion (`--`, `/* */`, `#`). Also via `detect_injection.comments`. | +| `max_risk` | float | — | Hard-block (no auto-fix) when a query's risk exceeds this threshold. | -sql-data-guard is also available as a Docker image, which can be used to run the application in a containerized environment. This is particularly useful for deployment in cloud environments or for maintaining consistency across different development setups. +### Per-table keys -### Running the Docker Container +| Key | Type | Effect | +|---|---|---| +| `columns` | list[str] | Allow-list of selectable columns. | +| `restrictions` | list | Row filters (see operations above). | +| `denied_columns` | list[str] | Columns excluded from results even if present in `columns`. | +| `column_masks` | list | Column-masking rules (below). | -To run the sql-data-guard Docker container, use the following command: +### Column masking (`column_masks`) + +Masking rewrites a sensitive column into a masking expression instead of removing it — the query keeps returning a column with the same output name. + +| Policy | Behavior | Options | +|---|---|---| +| `redact` | Replace the value with a constant | `replacement` (default `****`) | +| `hash` | Replace with a one-way `MD5(col)` | — | +| `partial` | Keep the last N characters, mask the rest | `show_last` (default `4`) | + +### Always-on threat detection + +No configuration is required for these — they have no legitimate use in an application query: + +- **Stacked-statement rejection** (`a; b`). +- **Dangerous-function deny-list** — e.g. `xp_cmdshell`, `load_file`, `pg_read_file`, `pg_sleep`, `benchmark`, `waitfor`, `sys_exec`. +- **System-catalog probing** — e.g. `information_schema`, `sqlite_master`, `pg_catalog`. + +### Example combining several controls + +```json +{ + "force_limit": 1000, + "blocked_functions": ["pg_sleep", "load_file"], + "max_risk": 0.8, + "tables": [ + { + "table_name": "users", + "columns": ["id", "email", "ssn", "tenant_id"], + "denied_columns": ["ssn"], + "column_masks": [ + { "column": "email", "policy": "hash" } + ], + "restrictions": [{ "column": "tenant_id", "value": 42 }] + } + ] +} +``` + +--- + +## REST API reference + +A Flask service exposes `verify_sql` over HTTP, with an interactive Swagger UI powered by [flasgger](https://github.com/flasgger/flasgger). + +### Run with Docker ```bash docker run -d --name sql-data-guard -p 5000:5000 ghcr.io/thalesgroup/sql-data-guard ``` -### Calling the Docker Container Using REST API +### Run from source + +```bash +pip install flask flasgger sql-data-guard +# APP_PORT defaults to 5000 +APP_PORT=5050 PYTHONPATH=src python src/sql_data_guard/rest/sql_data_guard_rest.py +``` + +On Windows (cmd): + +```bat +set APP_PORT=5050 +set PYTHONPATH=src +python src\sql_data_guard\rest\sql_data_guard_rest.py +``` + +### `POST /verify-sql` + +Request body (`application/json`): -Once the `sql-data-guard` Docker container is running, you can interact with it using its REST API. Below is an example of how to verify an SQL query using `curl`: +| Field | Type | Required | Description | +|---|---|---|---| +| `sql` | string | yes | The SQL query to verify. | +| `config` | object | yes | The restriction configuration. | +| `dialect` | string | no | SQL dialect (e.g. `sqlite`, `postgres`). | ```bash curl -X POST http://localhost:5000/verify-sql \ @@ -114,18 +364,156 @@ curl -X POST http://localhost:5000/verify-sql \ "sql": "SELECT * FROM orders WHERE account_id = 123", "config": { "tables": [ - { - "table_name": "orders", + { "table_name": "orders", "columns": ["id", "product_name", "account_id"], - "restrictions": [{"column": "account_id", "value": 123}] - } - ] + "restrictions": [{"column": "account_id", "value": 123}] } + ] } }' ``` -## Contributing -We welcome contributions! Please see our [CONTRIBUTING.md](CONTRIBUTING.md) for more details. +The response mirrors the library result (`allowed`, `errors`, `fixed`, `risk`). + +### Swagger UI + +With the server running, open `http://localhost:5050/apidocs`, expand **POST /verify-sql**, click **Try it out**, edit the pre-filled example, and **Execute**. + +--- + +## MCP wrapper + +The MCP wrapper transparently sits in front of an MCP database server: it intercepts `tools/call` requests, runs the SQL argument through `verify_sql`, and either rewrites the query or blocks it before it reaches the inner server. Configuration ([example](examples/mcp-wrapper-sqlite/config.json)) declares the inner `mcp-server` image, which `mcp-tools` carry SQL, and the `sql-data-guard` policy (including `dialect` and `inject-response`). + +Runnable examples live in [examples/mcp-wrapper-sqlite/](examples/mcp-wrapper-sqlite/) and [examples/mcp-wrapper-postgres/](examples/mcp-wrapper-postgres/). The wrapper image is built from [wrapper.Dockerfile](wrapper.Dockerfile). + +--- + +## Dify plugin + +A [Dify](https://dify.ai/) plugin wraps `sql-data-guard` so LLM workflows can validate generated SQL inline. It accepts `sql`, `config`, and optional `dialect`, and returns `allowed`, `errors`, `fixed`, `verified_sql`, and `risk`. See [plugins/dify/README.md](plugins/dify/README.md) and [plugins/dify/DEV.md](plugins/dify/DEV.md). + +--- + +## Project structure + +```text +. +├── src/sql_data_guard/ # Library source +│ ├── sql_data_guard.py # verify_sql — entry point and query verification +│ ├── restriction_validation.py # Config/restriction validation +│ ├── restriction_verification.py # Row/column restriction enforcement +│ ├── column_masking.py # Column masking (redact / hash / partial) +│ ├── injection_detection.py # Malicious-payload / injection scanning +│ ├── verification_context.py # Shared verification state, risk, errors +│ ├── verification_utils.py # sqlglot expression helpers +│ ├── rest/ # Flask REST API + Swagger UI +│ └── mcpwrapper/ # MCP server guard wrapper +├── plugins/dify/ # Dify LLM-workflow plugin +├── examples/ # MCP wrapper examples (SQLite, PostgreSQL) +├── docs/manual.md # Restriction schema and validation reference +├── test/ # Unit, REST, join, update, and LLM tests +├── Dockerfile # REST API image +├── wrapper.Dockerfile # MCP wrapper image +└── pyproject.toml # Build config and metadata +``` + +--- + +## Development and contributing + +Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full guide and [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). New here? Start with the [ONBOARDING.md](ONBOARDING.md) guide. + +Local setup: + +```bash +git clone https://github.com/ThalesGroup/sql-data-guard.git +cd sql-data-guard +pip install -e . +pip install -r test/test.requirements.txt +``` + +Workflow: fork → branch → change → add tests → open a pull request. Every PR must follow the coding style, include tests, pass the full suite, and update docs when behavior changes. + +--- + +## AI-assisted development + +This project is developed with AI coding assistants under explicit, version-controlled guardrails kept in the repo: + +- **Reusable skills** in [.agents/skills/](.agents/skills/) encode repeatable engineering tasks — e.g. `architecture-discovery`, `security-test-generator`, `sql-injection-researcher`, `benchmark-suite-builder`, `documentation-engineer`, and `refactoring-advisor`. +- **Assistant rules and workflows** in [.clinerules/](.clinerules/) — a security rule and a technical-documentation workflow that constrain how AI assistants operate on this codebase. + +--- + +## Testing + +Tests use `pytest`. Run the unit suite from the repo root: + +```bash +PYTHONPATH=src python -m pytest --color=yes test/*_unit.py +``` + +On Windows (cmd): + +```bat +set PYTHONPATH=src +python -m pytest --color=yes test/*_unit.py +``` + +The suite spans **13 unit test files** covering core verification, validation, joins, updates, the REST API, and DuckDB integration, plus dedicated suites for the security features: `test_column_masking_unit.py`, `test_denied_columns_unit.py`, `test_function_policy_unit.py`, `test_injection_detection_unit.py`, `test_limit_enforcement_unit.py`, and `test_security_fixes_unit.py`. A separate LLM test ([test/test_sql_guard_llm.py](test/test_sql_guard_llm.py)) runs in its own CI workflow. Unit tests also run automatically on every push. + +--- + +## Deployment and release + +- **Versioning:** SemVer, managed via git tags. +- **Release:** push a tag (e.g. `git tag -a v0.1.0 -m "Release 0.1.0" && git push --tags`). CI then publishes the package to **PyPI** and the Docker image to the **GitHub Container Registry** automatically. +- **CI workflows** live in [.github/workflows/](.github/workflows/) — unit tests, Python compatibility, LLM tests, PyPI publish, REST/MCP Docker image builds, and Dify plugin publish. + +--- + +## Security + +`sql-data-guard` is itself a defensive security control. Connecting LLMs to SQL databases without strict controls risks unauthorized disclosure (a pattern OWASP highlights) and can breach privacy regulations such as GDPR and CCPA. This project adds verification and query rewriting as a guardrail — but it complements, not replaces, database permissions, auditing, and API security. + +To report a vulnerability, contact **security@opensource.thalesgroup.com**. Never store credentials in source or config. See [SECURITY.md](SECURITY.md) for the full policy and supported versions. + +--- + +## FAQ and troubleshooting + +### Does this replace my database's permission system? + +No. It is an additional, application-layer guardrail for restrictions that are complex, vendor-specific, or impossible to express in the database — especially row- and column-level rules. + +### Why not just use prepared statements? + +Prepared statements fix a query's *structure*. LLM-generated SQL is dynamic and has no fixed structure to parameterize, so `sql-data-guard` validates the query's *content* instead. + +### My local REST requests return 503 / a Squid error page. + +A corporate proxy is intercepting localhost traffic. Bypass the proxy for local addresses — add `localhost, 127.0.0.1` to the browser's proxy exceptions, or use `--noproxy "*"` with curl: + +```bash +curl --noproxy "*" -X POST http://127.0.0.1:5050/verify-sql \ + -H "Content-Type: application/json" \ + -d '{"sql":"SELECT id FROM orders WHERE account_id = 123","config":{"tables":[{"table_name":"orders","columns":["id","account_id"],"restrictions":[{"column":"account_id","value":123}]}]}}' +``` + +### How do I change the REST API port? + +Set the `APP_PORT` environment variable before starting the server (default `5000`). + +--- ## License -This project is licensed under the MIT License. See the [LICENSE.md](LICENSE.md) file for details. + +Released under the [MIT License](LICENSE.md). Copyright © 2025 Imperva. + +--- + +## Contact and support + +- **Issues & feature requests:** [GitHub Issues](https://github.com/ThalesGroup/sql-data-guard/issues) +- **Security reports:** security@opensource.thalesgroup.com (never via public issues) +- **Homepage:** https://github.com/ThalesGroup/sql-data-guard diff --git a/docs/manual.md b/docs/manual.md index d53c887..9841c45 100644 --- a/docs/manual.md +++ b/docs/manual.md @@ -78,4 +78,4 @@ The validation function checks that the restrictions adhere to the following rul "operation": "IN", "values": [100, "Electronics"] ``` - This would raise an error because the values are not consistently of the same data type. + This would raise an error because the values are not consistently of the same data type. \ No newline at end of file diff --git a/docs/technical-documentation/00-INDEX.md b/docs/technical-documentation/00-INDEX.md new file mode 100644 index 0000000..36b218a --- /dev/null +++ b/docs/technical-documentation/00-INDEX.md @@ -0,0 +1,102 @@ +# sql-data-guard — Technical Documentation Set + +> A multi-document technical analysis of the **sql-data-guard** project, generated for architecture reviews, knowledge transfer, onboarding, code reviews, and refactoring planning. + +`sql-data-guard` is a **safety layer for LLM ↔ database interactions**. It accepts an SQL query plus a JSON restriction configuration and returns whether the query is *allowed*, a list of *errors*, an optional auto-*fixed* query, and a numeric *risk* score. It also detects malicious payloads (stacked queries, dangerous functions, system-catalog probing, comment evasion). + +--- + +## Document map + +| # | Document | Audience | What it covers | +|---|----------|----------|----------------| +| 00 | **[00-INDEX.md](00-INDEX.md)** | Everyone | This page — navigation, project summary, glossary | +| 01 | **[01-CODE_EXPLANATION.md](01-CODE_EXPLANATION.md)** | New developers, reviewers | Module-by-module, function-by-function walkthrough with reasoning | +| 02 | **[02-HLD.md](02-HLD.md)** | Architects, stakeholders | High-Level Design: purpose, components, data flow, deployment surfaces | +| 03 | **[03-LLD.md](03-LLD.md)** | Implementers | Low-Level Design: classes, signatures, algorithms, sequence diagrams | +| 04 | **[04-ARCHITECTURE.md](04-ARCHITECTURE.md)** | Architects | Architecture, design patterns, security model, performance, refactoring | +| 05 | **[05-CODING_GUIDELINES.md](05-CODING_GUIDELINES.md)** | Contributors | Conventions, security mindset, testing, PR checklist | + +--- + +## Project at a glance + +| Attribute | Value | +|-----------|-------| +| **Project type** | Library + Flask REST microservice + MCP (Model Context Protocol) stdio wrapper | +| **Language** | Python (>= 3.8) | +| **Core dependency** | [`sqlglot`](https://github.com/tobymao/sqlglot) (SQL parser / AST) | +| **Public API** | `verify_sql(sql: str, config: dict, dialect: str = None) -> dict` | +| **Distribution** | PyPI (`pip install sql-data-guard`) + Docker image (`ghcr.io/thalesgroup/sql-data-guard`) | +| **License** | MIT | +| **Validation strategy** | AST-based (sqlglot) + a thin pre-parse regex scan for comment evasion | + +### Source layout + +```text +src/sql_data_guard/ +├── __init__.py # exports verify_sql +├── sql_data_guard.py # orchestrator: verify_sql() + query traversal +├── injection_detection.py # F1: malicious-payload scans (raw + AST) +├── restriction_validation.py # config-shape validation (fail fast) +├── restriction_verification.py # row-level restriction enforcement on the AST +├── column_masking.py # column redact / hash / partial masking +├── verification_context.py # mutable per-query result accumulator +├── verification_utils.py # small AST traversal helpers +├── rest/ +│ ├── sql_data_guard_rest.py # Flask app + Swagger, POST /verify-sql +│ └── logging.conf +└── mcpwrapper/ + └── mcp_wrapper.py # MCP stdio proxy around a containerised MCP server +``` + +--- + +## The core contract + +```python +from sql_data_guard import verify_sql + +config = { + "tables": [{ + "table_name": "orders", + "columns": ["id", "product_name", "account_id"], + "restrictions": [{"column": "account_id", "value": 123}], + }] +} + +verify_sql("SELECT id, name FROM orders WHERE 1 = 1", config) +# { +# "allowed": False, +# "errors": [ +# "Column name is not allowed. Column removed from SELECT clause", +# "Static expression is not allowed: 1 = 1", +# "Missing restriction for table: orders column: account_id value: 123" +# ], +# "fixed": "SELECT id, product_name, account_id FROM orders WHERE account_id = 123", +# "risk": +# } +``` + +Return dictionary keys: + +| Key | Type | Meaning | +|-----|------|---------| +| `allowed` | `bool` | `True` only when zero errors remain (and risk ≤ `max_risk` if set) | +| `errors` | `List[str]` | Ordered, de-duplicated human-readable findings | +| `fixed` | `Optional[str]` | A compliant rewrite, when all findings were auto-fixable | +| `risk` | `float` | Mean of per-finding risk weights, `0` (safe) .. `1` (high) | + +--- + +## Glossary + +| Term | Definition | +|------|-----------| +| **Restriction** | A row-level rule (`column`, `operation`, `value(s)`) that *must* appear in the query's `WHERE` clause; injected if missing. | +| **Fixable finding** | A violation the verifier can auto-correct (drop a column, add a restriction, clamp a limit). Surfaces in `fixed`. | +| **Hard block** | A non-fixable finding (stacked query, disallowed table, dangerous function). Forces `allowed = False`, `fixed = None`. | +| **Dynamic table / column** | Columns exposed by sub-queries, CTEs, lateral joins, `UNNEST` — resolved at verification time. | +| **Risk score** | Average of every finding's weight; compared against the optional `max_risk` config threshold. | +| **AST** | Abstract Syntax Tree produced by sqlglot from the raw SQL string. | +| **MCP** | Model Context Protocol — the stdio wrapper proxies tool calls and validates embedded SQL. | diff --git a/docs/technical-documentation/01-CODE_EXPLANATION.md b/docs/technical-documentation/01-CODE_EXPLANATION.md new file mode 100644 index 0000000..3b9b697 --- /dev/null +++ b/docs/technical-documentation/01-CODE_EXPLANATION.md @@ -0,0 +1,221 @@ +# 01 · Code Explanation + +> Module-by-module, function-by-function walkthrough of `src/sql_data_guard`. The goal is not to repeat the code, but to explain **what it does, why it exists, and how data moves through it**. + +See also: [00-INDEX](00-INDEX.md) · [02-HLD](02-HLD.md) · [03-LLD](03-LLD.md) + +--- + +## 1. `__init__.py` — the public surface + +```python +from .sql_data_guard import verify_sql +``` + +The entire package exposes exactly **one** public symbol, `verify_sql`. Everything else is an implementation detail. This is a deliberate *facade*: callers depend on a stable single entry point while the internals are free to evolve. + +--- + +## 2. `sql_data_guard.py` — the orchestrator + +This is the heart of the system. It owns `verify_sql()` plus the recursive query-tree traversal that enforces table/column/restriction rules. + +### 2.1 `verify_sql(sql, config, dialect=None) -> dict` + +The end-to-end pipeline. Reading top to bottom: + +1. **Config sanity gate.** If `config` is falsy, not a `dict`, or lacks `"tables"`, it returns a hard block immediately (`risk = 1.0`). *Why:* a missing config must never fail open. +2. **Length gate.** `max_length` (default `10_000`) caps the raw SQL string. *Why:* bounds parser cost and denial-of-service via gigantic inputs. +3. **Config-shape validation.** `validate_restrictions(config)` and `validate_column_masks(config)` run before any parsing. Errors here are hard blocks. *Why:* fail fast on operator misconfiguration rather than producing misleading query errors later. +4. **Context creation.** A fresh `VerificationContext` accumulates findings for this one query. +5. **Pre-parse raw scan.** `scan_raw_sql(sql, result)` looks for comment-evasion **before** sqlglot can discard the comment tail (the key bypass F1 closes). +6. **Parse.** `sqlglot.parse(sql, dialect)` returns a list of statements. + - **> 1 statement** → explicit *stacked query* hard block. + - **1 statement** → proceed. + - **0 statements with no prior errors** → "Could not find a query statement". + - **ParseError** → recorded as a `risk = 0.9` finding. +7. **AST scans + statement-type gating** (only when a single statement parsed): + - `scan_parsed_sql` — always-on malicious-payload AST scan. + - `_verify_functions` — config-driven function allow/deny (F2). + - `Command` / `Delete` / `Insert` / `Update` / `Create` → not allowed (read-only posture). + - `Query` (SELECT / set operations) → `_verify_query_statement` (the deep traversal). +8. **Force-limit enforcement.** `_enforce_force_limit` runs only if the query is still fixable and is a `Query` (F3). +9. **Auto-fix materialisation.** If the context is still fixable and there are findings, `result.fixed = parsed.sql(dialect)` renders the (mutated) AST back to SQL. +10. **Risk threshold (F5).** If `max_risk` is set and exceeded, the result is hard-blocked and `fixed` is suppressed — refusing to "auto-fix" a query the policy deems too risky. + +> **Key insight:** the AST is *mutated in place* throughout traversal (columns removed, restrictions appended, masks substituted, limits set). The final `fixed` string is simply that mutated tree serialised back to SQL. + +### 2.2 `_verify_functions(parsed, context)` — F2 function policy + +Reads two optional config keys, matched case-insensitively: + +- `blocked_functions` — any call is blocked. +- `allowed_functions` — if present, *only* these may be called. + +Every `expr.Func` node is inspected via `function_name()`. A violation is a **hard block** (not auto-fixable) — stripping a function from an arbitrary position could silently change query semantics or produce invalid SQL. Composes with the always-on dangerous-function deny-list in `injection_detection`. + +### 2.3 `_enforce_force_limit(parsed, context)` — F3 row cap + +If `force_limit` is a positive `int` (and not a `bool`): + +- No `LIMIT` → inject `LIMIT force_limit`. +- `LIMIT` above the cap → clamp down. +- `LIMIT` at/below the cap → leave unchanged. + +Only the **outermost** statement is touched — sub-queries and CTEs are intentionally left alone, because the cap protects the final result set returned to the caller. This is a *fixable* finding (`risk = 0.3`). + +### 2.4 The recursive traversal: `_verify_query_statement` + +This is the recursion backbone. For a given `Query` node: + +- **Set operations** (`UNION` / `EXCEPT` / `INTERSECT`, all `SetOperation` subclasses) → recurse into `left` and `right`. *Why:* each arm is its own SELECT that must independently satisfy table/column rules (S4). +- **CTEs** → register the alias *before* and *after* verifying the CTE body. The double registration captures columns expanded from `SELECT *` and lets forward/recursive references resolve (S3). +- **FROM tables** → `_verify_from_tables` confirms every table is allow-listed (or dynamic). +- When still fixable: verify the SELECT clause, the WHERE clause, then any sub-queries in ORDER/GROUP/HAVING/LIMIT/OFFSET. + +### 2.5 SELECT-clause verification + +`_verify_select_clause` / `_verify_select_clause_element` classify each projected expression: + +| Element | Handling | +|---------|----------| +| `Star` (`*`) | `_expand_star` — replace with the allow-listed columns (minus `denied_columns`); records "SELECT * is not allowed" (fixable). | +| `Column` `t.*` | Qualified wildcard — same expansion, scoped to one table (F10). | +| `Column` | `_verify_col` (allow-list check) then `_apply_column_mask` (F-mask). | +| `Tuple` | Recurse over members. | +| Function / expression | Recurse into the inner `Column` nodes (so `SUM(col)` is checked on `col`). | + +If the SELECT clause becomes empty after stripping, "No legal elements in SELECT clause" is a hard block. + +### 2.6 `_verify_col` — column authorisation + +A column is allowed if **any** of these hold: + +1. It belongs to an allow-listed config table. +2. Its table is a dynamic (sub-select / CTE) source, and the column is one that source actually exposes. +3. It is un-prefixed but exposed by some dynamic source. +4. It is in `dynamic_columns`. + +A column is **denied** outright (deny wins over allow) if it appears in `denied_columns` — stripped from SELECT with a fixable finding (F10). Disallowed columns are also stripped, fixable. + +> **Security note (S3):** when *all* FROM tables are dynamic, the code does **not** blindly allow every column. It only allows columns the dynamic sources expose — closing a wildcard-trust bypass. + +### 2.7 `_apply_column_mask` — F-mask rewrite + +If a column maps unambiguously to exactly one masked table, its expression is rewritten (`build_mask_expression`) into the masking form while preserving the output column name. Ambiguous or table-qualified mismatches are skipped. The rewrite is fixable (`risk = 0.2`). + +### 2.8 WHERE-clause + static-expression handling + +- `_verify_where_clause` recurses into `Subquery`/`Exists` inside the WHERE, then enforces restrictions. +- `_verify_static_expression` / `_has_static_expression` detect **always-true** style payloads (`OR 1=1`): an OR-branch with no column reference is flagged "Static expression is not allowed" and replaced with `FALSE`, then `simplify()` collapses it. This is the classic injection-neutralising rewrite. + +### 2.9 FROM-clause table discovery: `_get_from_clause_tables` + +Walks the FROM clause and each JOIN. For sub-queries it **verifies first, then records exposed columns**, so the alias maps to *real allowed columns* rather than `*`. Handles `Lateral` and `Unnest` join sources too. `_add_table_alias` / `_register_dynamic_columns` populate the context's dynamic maps. + +--- + +## 3. `verification_context.py` — the result accumulator + +`VerificationContext` is a small mutable object threaded through the whole traversal. + +| Member | Role | +|--------|------| +| `_errors: List[str]` | Ordered + **de-duplicated** findings (a list, not a set, so the API response is deterministic). | +| `_can_fix: bool` | Flips to `False` the moment any non-fixable finding is added. | +| `_risk: List[float]` | Per-finding weights; `risk` property returns their **mean**. | +| `_dynamic_tables` / `_dynamic_columns` | Columns exposed by sub-queries / CTEs / lateral / unnest. | +| `_column_masks` | `{table: {column: mask_spec}}` lookup built once from config. | + +`add_error(error, can_fix, risk)` is the single mutation entry point: de-dups the message, lowers `can_fix`, and appends the risk weight. + +> **Design caveat:** risk is an *average*. Adding several low-risk findings can dilute a single high-risk one. (See [04-ARCHITECTURE](04-ARCHITECTURE.md#risk-scoring) for the recommended max-weight refactor.) + +--- + +## 4. `injection_detection.py` — F1 malicious-payload scans + +Two complementary scans that **only report** (never mutate): + +- **`scan_raw_sql`** (pre-parse, opt-in via `detect_comments` / `detect_injection.comments`): regex for `--`, `/* */`, `#`. Catches comment-evasion that hides a payload tail from sqlglot. +- **`scan_parsed_sql`** (AST, always-on): + - `_scan_stacked_statements` — newer sqlglot wraps `a; b` in a `Block` node. + - `_scan_dangerous_functions` — `_DANGEROUS_FUNCTIONS` frozenset (`load_file`, `xp_cmdshell`, `pg_sleep`, `sleep`, `benchmark`, `waitfor`, …). + - `_scan_system_catalogs` — `information_schema`, `sqlite_master`, `pg_catalog`, etc. + +`function_name(func)` is a cross-version helper: dialect-specific functions parse as `Anonymous` nodes whose real name is on `.name`, while built-ins expose it via `sql_names()`. + +Risk weights are named constants (stacked `0.9`, dangerous fn `0.9`, comment `0.8`, catalog `0.8`). + +--- + +## 5. `restriction_validation.py` — fail-fast config validation + +Runs *before* parsing. Validates: + +- `force_limit` is a positive non-bool int (F3). +- `allowed_functions` / `blocked_functions` are lists of strings (F2). +- Each table has `table_name` and non-empty `columns`. +- `denied_columns` is a list of strings (F10). +- Each restriction's operation is supported (`=`, `>`, `<`, `>=`, `<=`, `BETWEEN`, `IN`), with structural checks per operator. + +`_validate_restriction` also **normalises the operation to upper case in place**, so configs can use `"between"` while downstream AST matching only compares upper-case operators (case-insensitive UX, single internal representation). + +--- + +## 6. `restriction_verification.py` — enforcing row-level restrictions + +`verify_restrictions` is where the WHERE clause is checked against configured restrictions. + +1. Split the WHERE clause into its AND-ed sub-expressions. +2. For each `(config_table, from_table, restriction)`, look for a sub-expression that *satisfies* the restriction (`_verify_restriction`). +3. If none satisfies it → record "Missing restriction…" (fixable) and **inject** the condition (`_create_new_condition`), ANDing it onto the existing WHERE (or creating one). + +`_verify_restriction` matches the query expression to the restriction: + +- `IN` → all query values must be a subset of allowed values. +- `EQ` → the value must be in the allowed set. +- `BETWEEN` → query range must be *inside* the allowed range. +- `<`, `<=`, `>`, `>=` → operator must match the restriction's operator, and `_compare_values` compares **numerically when possible** (fixes the "9" < "18" string-sort bug, S2). + +`_format_value` renders Python values as **safe SQL literals**, escaping embedded single quotes via sqlglot — preventing the fix itself from being an injection vector (S1). + +--- + +## 7. `column_masking.py` — redact / hash / partial + +- `validate_column_masks` — masked column must be in the table's `columns` allow-list; policy must be supported; `show_last` must be a non-negative int. +- `build_mask_lookup` — `{table: {column: mask_spec}}`. +- `build_mask_expression` — produces the dialect-portable masking expression, aliased back to the original output name: + - **redact** → constant string (default `****`). + - **hash** → `MD5(col)`. + - **partial** → `CONCAT('****', SUBSTRING(CAST(col AS VARCHAR), -show_last))`. + +> Masking keeps the **result shape** intact (same output column name), unlike dropping the column — so downstream consumers don't break. + +--- + +## 8. `verification_utils.py` — tiny AST helpers + +- `split_to_expressions(exp, type)` — flatten a binary AND/OR chain into its leaves. +- `find_direct(exp, type)` — yield only the *direct* children of a node matching a type (non-recursive), used to avoid pulling nested clauses out of context. + +--- + +## 9. `rest/sql_data_guard_rest.py` — Flask REST API + +- Single endpoint: `POST /verify-sql`, JSON body `{ sql, config, dialect? }`. +- Swagger UI via flasgger at `/apidocs`. +- Optional API-key auth (S6): if `SQL_GUARD_API_KEY` env var is set, requests must carry a matching `X-API-Key` header; otherwise open (dev default). +- Thin shim: validates request shape, calls `verify_sql`, JSON-ifies the result. + +--- + +## 10. `mcpwrapper/mcp_wrapper.py` — MCP stdio proxy + +Wraps a containerised MCP server (e.g. `mcp/sqlite`). On every `tools/call` it extracts the SQL argument, runs `verify_sql`, and: + +- **Blocked + fixable** → forwards the `fixed` SQL. +- **Blocked + not fixable** → replaces with a `SELECT 'Blocked by SQL Data Guard'` message (optionally `UNION`-ing the error strings, or injecting them into the response when `inject-response` is on). + +This lets an LLM agent talk to a real database server through a transparent validation layer, with no change to the agent. diff --git a/docs/technical-documentation/02-HLD.md b/docs/technical-documentation/02-HLD.md new file mode 100644 index 0000000..017a487 --- /dev/null +++ b/docs/technical-documentation/02-HLD.md @@ -0,0 +1,186 @@ +# 02 · High-Level Design (HLD) + +> The "30,000-foot" view: what the system is for, the major components, how data flows, and the deployment surfaces. For implementation detail see [03-LLD](03-LLD.md). + +See also: [00-INDEX](00-INDEX.md) · [01-CODE_EXPLANATION](01-CODE_EXPLANATION.md) · [04-ARCHITECTURE](04-ARCHITECTURE.md) + +--- + +## 1. Executive summary + +**Purpose.** `sql-data-guard` is a *defence-in-depth* layer that sits between an SQL-generating client (often an LLM) and a database. It inspects each query against a declarative restriction configuration and either **allows**, **rewrites** (auto-fix), or **blocks** it. + +**Business objective.** Prevent unauthorised data access and SQL-injection-style data exposure for dynamic, LLM-generated queries that *cannot* be expressed as prepared statements. It supports fine-grained, column-level and row-level (multi-tenant) security that the database permission model often cannot express — and helps meet GDPR/CCPA obligations. + +**Main workflows.** + +1. **Verify** — given `(sql, config)`, decide allowed/blocked and compute a risk score. +2. **Auto-fix** — rewrite a non-compliant-but-fixable query to a compliant one. +3. **Detect attacks** — flag stacked queries, dangerous functions, system-catalog probing, comment evasion. +4. **Proxy** — (MCP wrapper) transparently validate SQL flowing from an agent to a DB server. + +**Key responsibilities.** + +| Responsibility | Owner module | +|----------------|--------------| +| Orchestration & query traversal | `sql_data_guard.py` | +| Config validation (fail-fast) | `restriction_validation.py`, `column_masking.py` | +| Row-level restriction enforcement | `restriction_verification.py` | +| Column allow/deny & masking | `sql_data_guard.py`, `column_masking.py` | +| Attack detection | `injection_detection.py` | +| Result accumulation & risk | `verification_context.py` | +| Delivery surfaces | `rest/`, `mcpwrapper/` | + +--- + +## 2. System context + +```mermaid +flowchart LR + LLM["LLM / Application
    (generates SQL)"] + subgraph GUARD["sql-data-guard"] + API["REST API
    POST /verify-sql"] + MCP["MCP stdio wrapper"] + LIB["verify_sql() library"] + end + DB[("Database
    SQLite / Postgres / Trino / DuckDB …")] + + LLM -->|sql + config| API + LLM -->|tools/call| MCP + LLM -->|import| LIB + API --> LIB + MCP --> LIB + MCP -->|validated / fixed SQL| DB + LIB -->|allowed · errors · fixed · risk| LLM +``` + +> `sql-data-guard` does **not** execute SQL itself (except in tests). It is a validator/rewriter. The caller remains responsible for executing the (possibly fixed) query. + +--- + +## 3. Component overview + +```mermaid +flowchart TD + subgraph Delivery + REST["rest/sql_data_guard_rest.py
    Flask + Swagger"] + MCPW["mcpwrapper/mcp_wrapper.py
    stdio proxy"] + end + + subgraph Core + ORCH["sql_data_guard.py
    verify_sql + traversal"] + CTX["verification_context.py
    VerificationContext"] + end + + subgraph Rules + RV["restriction_validation.py
    config shape"] + RVER["restriction_verification.py
    row restrictions"] + CM["column_masking.py
    mask policies"] + ID["injection_detection.py
    attack scans"] + UTIL["verification_utils.py
    AST helpers"] + end + + SQLGLOT["sqlglot
    (parser / AST)"] + + REST --> ORCH + MCPW --> ORCH + ORCH --> CTX + ORCH --> RV + ORCH --> RVER + ORCH --> CM + ORCH --> ID + ORCH --> UTIL + ORCH --> SQLGLOT + RVER --> SQLGLOT + CM --> SQLGLOT +``` + +--- + +## 4. End-to-end data flow + +```mermaid +flowchart TD + IN["Input: sql, config, dialect"] + G1{"config valid?"} + G2{"len(sql) ≤ max_length?"} + VAL["validate_restrictions
    validate_column_masks"] + RAW["scan_raw_sql
    (comment evasion, opt-in)"] + PARSE["sqlglot.parse"] + G3{"statement count"} + SCAN["scan_parsed_sql
    (stacked / dangerous fn / catalog)"] + FN["_verify_functions (F2)"] + TYPE{"statement type"} + TRAV["_verify_query_statement
    (tables · columns · masks · restrictions)"] + LIM["_enforce_force_limit (F3)"] + FIX["materialise fixed = AST.sql()"] + RISK{"risk > max_risk?"} + OUT["Output: allowed · errors · fixed · risk"] + + IN --> G1 + G1 -- no --> OUT + G1 -- yes --> G2 + G2 -- no --> OUT + G2 -- yes --> VAL + VAL -- error --> OUT + VAL -- ok --> RAW --> PARSE --> G3 + G3 -- ">1 (stacked)" --> OUT + G3 -- "1" --> SCAN --> FN --> TYPE + TYPE -- "DML/DDL/Command" --> OUT + TYPE -- "Query" --> TRAV --> LIM --> FIX --> RISK + RISK -- yes --> OUT + RISK -- no --> OUT +``` + +--- + +## 5. Configuration model (high level) + +A config is a JSON/dict document. Top-level keys: + +| Key | Required | Purpose | Feature | +|-----|----------|---------|---------| +| `tables` | ✅ | Allow-listed tables, each with `columns`, optional `restrictions`, `denied_columns`, `column_masks` | core | +| `max_length` | optional | Max raw SQL length (default 10000) | core | +| `max_risk` | optional | Hard-block threshold on the risk score | F5 | +| `force_limit` | optional | Mandatory row cap on the outer query | F3 | +| `allowed_functions` / `blocked_functions` | optional | Function allow/deny list | F2 | +| `detect_comments` / `detect_injection.comments` | optional | Opt-in comment-evasion scan | F1 | + +Per-table keys: `table_name`, `columns`, `restrictions[]`, `denied_columns[]`, `column_masks[]`. + +--- + +## 6. Deployment surfaces + +| Surface | How to run | Best for | +|---------|-----------|----------| +| **Library** | `pip install sql-data-guard`; `from sql_data_guard import verify_sql` | In-process integration in a Python app | +| **REST** | `docker run -p 5000:5000 ghcr.io/thalesgroup/sql-data-guard` | Language-agnostic microservice; Swagger at `/apidocs` | +| **MCP wrapper** | Run `mcp_wrapper.py` with a `/conf/config.json`; it spawns the inner MCP server container | Transparent guard for MCP-based LLM agents | + +```mermaid +flowchart LR + subgraph Host + WRAP["mcp_wrapper.py"] + subgraph Docker + INNER["inner MCP server
    (e.g. mcp/sqlite)"] + end + end + AGENT["LLM agent (stdin/stdout)"] <-->|JSON-RPC| WRAP + WRAP <-->|stdin/stdout socket| INNER + INNER --> DB[("DB / data volume")] +``` + +--- + +## 7. Quality attributes + +| Attribute | How the design addresses it | +|-----------|------------------------------| +| **Security** | Default-deny tables/columns; read-only posture (DML/DDL blocked); injection neutralisation; safe literal escaping; optional API key. | +| **Backward compatibility** | All new policies (F2/F3/F5/F10/masking, comment detection) are opt-in; benign queries are unaffected. | +| **Portability** | sqlglot abstracts dialects (sqlite, postgres, mysql, trino, duckdb tested). | +| **Determinism** | Errors are an ordered, de-duplicated list. | +| **Testability** | Pure function core (`verify_sql`) with no I/O; large pytest suite incl. real DB execution. | +| **Performance** | Single parse pass; bounded input length; in-place AST mutation avoids re-parsing. | diff --git a/docs/technical-documentation/03-LLD.md b/docs/technical-documentation/03-LLD.md new file mode 100644 index 0000000..2ee3949 --- /dev/null +++ b/docs/technical-documentation/03-LLD.md @@ -0,0 +1,291 @@ +# 03 · Low-Level Design (LLD) + +> Implementation-level detail: classes, signatures, algorithms, and sequence diagrams. Pairs with [01-CODE_EXPLANATION](01-CODE_EXPLANATION.md) (prose) and [02-HLD](02-HLD.md) (overview). + +--- + +## 1. Public API contract + +```python +def verify_sql(sql: str, config: dict, dialect: str = None) -> dict: ... +``` + +| Param | Type | Notes | +|-------|------|-------| +| `sql` | `str` | Raw query; length-capped by `config["max_length"]` (default 10000). | +| `config` | `dict` | Must contain `"tables"`; otherwise hard-blocked. | +| `dialect` | `str?` | Passed through to sqlglot (`"sqlite"`, `"postgres"`, `"mysql"`, `"trino"`, `"duckdb"`, …). | + +**Return:** `{"allowed": bool, "errors": List[str], "fixed": Optional[str], "risk": float}`. + +--- + +## 2. Class & function inventory + +### 2.1 `VerificationContext` (`verification_context.py`) + +```python +class VerificationContext: + def __init__(self, config: dict, dialect: str) + @property can_fix: bool + @property errors: List[str] + @property fixed: Optional[str] # settable + @property config: dict + @property dynamic_tables: Dict[str, Set[str]] + @property dynamic_columns: Set[str] + @property dialect: str + @property column_masks: Dict[str, Dict[str, dict]] + @property risk: float # mean of _risk, 0 if empty + def add_error(self, error: str, can_fix: bool, risk: float) -> None +``` + +**`add_error` algorithm:** +``` +if error not in _errors: _errors.append(error) # ordered de-dup +if not can_fix: _can_fix = False # latch +_risk.append(risk) # always recorded +``` + +### 2.2 Orchestrator functions (`sql_data_guard.py`) + +| Function | Signature | Mutates AST? | Emits | +|----------|-----------|--------------|-------| +| `verify_sql` | `(sql, config, dialect) -> dict` | drives | final dict | +| `_verify_functions` | `(parsed, context)` | no | hard block on disallowed fn | +| `_enforce_force_limit` | `(parsed, context)` | yes (sets LIMIT) | fixable | +| `_verify_query_statement` | `(query, context)` | yes (recurses) | various | +| `_verify_from_tables` | `(context, query) -> List[Table]` | no | hard block on bad table | +| `_verify_select_clause` | `(context, clause, from_tables)` | yes (strips cols) | fixable / hard | +| `_verify_select_clause_element` | `(from_tables, context, e) -> bool` | yes | — | +| `_expand_star` | `(e, from_tables, context, table_filter="")` | yes (expands `*`) | fixable | +| `_verify_col` | `(col, from_tables, context) -> bool` | no | fixable on deny/disallow | +| `_apply_column_mask` | `(col, from_tables, context)` | yes (rewrite) | fixable | +| `_verify_where_clause` | `(context, query, from_tables)` | yes | — | +| `_verify_static_expression` | `(query, context) -> bool` | yes (→ FALSE) | fixable | +| `_get_from_clause_tables` | `(query, context) -> List[Table]` | yes (verifies subqueries) | — | +| `_add_table_alias` / `_register_dynamic_columns` | `(exp, context)` | no | populates dynamic maps | + +### 2.3 Restriction modules + +```python +# restriction_validation.py +def validate_restrictions(config: dict) -> None # raises UnsupportedRestrictionError | ValueError +SUPPORTED_OPERATIONS = {"=", ">", "<", ">=", "<=", "BETWEEN", "IN"} + +# restriction_verification.py +def verify_restrictions(select, context, from_tables) -> None +def _verify_restriction(restriction, from_table, exp) -> bool +def _compare_values(query_value, restriction_value, operation) -> bool +def _create_new_condition(context, restriction, table_prefix) -> Expression +def _format_value(value) -> str # safe SQL literal +``` + +### 2.4 Masking (`column_masking.py`) + +```python +SUPPORTED_MASK_POLICIES = {"redact", "hash", "partial"} +def validate_column_masks(config: dict) -> None +def build_mask_lookup(config: dict) -> Dict[str, Dict[str, dict]] +def build_mask_expression(mask: dict, column: Column, dialect: Optional[str]) -> Expression +``` + +### 2.5 Injection detection (`injection_detection.py`) + +```python +def scan_raw_sql(sql: str, context) -> None # opt-in +def scan_parsed_sql(parsed, context) -> None # always-on +def comment_detection_enabled(config: dict) -> bool +def is_stacked_statement(parsed) -> bool +def function_name(func: Func) -> str +_DANGEROUS_FUNCTIONS: frozenset +_SYSTEM_CATALOGS: frozenset +``` + +--- + +## 3. Core algorithms + +### 3.1 Restriction satisfaction (`_verify_restriction`) + +```text +if exp is NOT -> False (negation can't satisfy a positive restriction) +if exp is Paren -> recurse into exp.this +if exp.this not Column or name != restriction.column -> False +if table-qualified and table doesn't match from_table -> False +values = restriction values (as strings) +match exp: + IN -> every query member ∈ values + EQ -> query value ∈ values + BETWEEN -> restriction.low ≤ query.low AND query.high ≤ restriction.high + LT/LE/GT/GE -> operator matches restriction.operation + AND _compare_values(query_value, values[0], op) +else -> False +``` + +### 3.2 Numeric-aware comparison (`_compare_values`, fix S2) + +```text +try: left, right = float(query_value), float(restriction_value) +except: left, right = query_value, restriction_value # lexicographic fallback +apply <, <=, >, >= accordingly +``` +Prevents `"9" < "18"` evaluating false due to string ordering. + +### 3.3 Static-expression neutralisation (`OR 1=1`) + +```text +for each OR-branch of a WHERE AND-term: + if branch has no Column node: + add_error("Static expression is not allowed", fixable, 0.8) + if branch's (un-parenthesised) parent is an OR: + replace branch with Boolean(False) +simplify(where_clause) # collapses `... OR FALSE` to `...` +``` + +### 3.4 Force-limit (F3) + +```text +if force_limit not positive int (or is bool): return +limit = outer query LIMIT +if limit exists and current ≤ force_limit: return # leave alone +set LIMIT = force_limit +add_error("Row limit enforced: ...", fixable, 0.3) +``` + +--- + +## 4. Sequence diagrams + +### 4.1 Happy path — fully compliant query + +```mermaid +sequenceDiagram + participant Caller + participant verify_sql + participant Validator as restriction_validation + participant sqlglot + participant Scan as injection_detection + participant Ctx as VerificationContext + + Caller->>verify_sql: verify_sql(sql, config, dialect) + verify_sql->>Validator: validate_restrictions / column_masks + Validator-->>verify_sql: ok + verify_sql->>Ctx: new VerificationContext + verify_sql->>Scan: scan_raw_sql (opt-in) + verify_sql->>sqlglot: parse(sql) + sqlglot-->>verify_sql: [Select] + verify_sql->>Scan: scan_parsed_sql + verify_sql->>verify_sql: _verify_query_statement (tables/cols/restrictions OK) + verify_sql-->>Caller: {allowed:true, errors:[], fixed:null, risk:0} +``` + +### 4.2 Auto-fix path — disallowed column + missing restriction + +```mermaid +sequenceDiagram + participant Caller + participant verify_sql + participant Trav as _verify_query_statement + participant RVer as restriction_verification + participant Ctx as VerificationContext + + Caller->>verify_sql: "SELECT id, secret FROM orders" + verify_sql->>Trav: traverse + Trav->>Ctx: add_error("Column secret not allowed", fixable, 0.3) + Trav->>Trav: strip `secret` from SELECT + Trav->>RVer: verify_restrictions + RVer->>Ctx: add_error("Missing restriction ...", fixable, 0.5) + RVer->>RVer: inject `AND account_id = 123` + verify_sql->>verify_sql: fixed = AST.sql() + verify_sql-->>Caller: {allowed:false, errors:[...], fixed:"SELECT id FROM orders WHERE account_id = 123", risk:0.4} +``` + +### 4.3 Hard block — stacked query + +```mermaid +sequenceDiagram + participant Caller + participant verify_sql + participant sqlglot + participant Ctx as VerificationContext + + Caller->>verify_sql: "SELECT id FROM orders; DROP TABLE orders" + verify_sql->>sqlglot: parse(sql) + sqlglot-->>verify_sql: [Select, Drop] (len > 1) + verify_sql->>Ctx: add_error("Stacked query detected", can_fix=false, 0.9) + verify_sql-->>Caller: {allowed:false, fixed:null, risk:0.9} +``` + +### 4.4 MCP proxy interception + +```mermaid +sequenceDiagram + participant Agent + participant Wrapper as mcp_wrapper + participant Guard as verify_sql + participant Inner as MCP server + participant DB + + Agent->>Wrapper: tools/call {query: SQL} + Wrapper->>Guard: verify_sql(SQL, config) + alt allowed or fixable + Guard-->>Wrapper: fixed / allowed SQL + Wrapper->>Inner: forward (possibly fixed) SQL + Inner->>DB: execute + DB-->>Inner: rows + Inner-->>Wrapper: result + Wrapper-->>Agent: result (+ injected note if configured) + else hard block + Wrapper->>Inner: SELECT 'Blocked by SQL Data Guard' + Inner-->>Wrapper: blocked message + Wrapper-->>Agent: blocked message + end +``` + +--- + +## 5. State: the dynamic-source maps + +`VerificationContext` carries two maps that make sub-queries / CTEs safe: + +| Map | Populated by | Used by | +|-----|--------------|---------| +| `dynamic_tables: {alias -> {columns}}` | `_add_table_alias` (TableAlias columns or `named_selects`) | `_verify_col` to allow qualified refs to a sub-source's real columns | +| `dynamic_columns: {column}` | `_register_dynamic_columns` (un-aliased sources) | `_verify_col` to allow un-prefixed refs | + +**Ordering rule (S3):** a sub-query must be *verified and SELECT-`*` expanded* **before** its exposed columns are recorded, so the alias maps to real allowed columns, not `*`. + +--- + +## 6. Error taxonomy & risk weights + +| Finding | Fixable? | Risk | +|---------|----------|------| +| Invalid config / over max_length | no | 1.0 | +| Table not allowed | no | 1.0 | +| Stacked query / dangerous fn | no | 0.9 | +| DML/DDL/Command | no | 0.9 | +| Parse error | no | 0.9 | +| Static (always-true) expression | yes | 0.8 | +| Comment evasion / system catalog | no | 0.8 | +| Could not find query statement | no | 0.7 | +| Missing restriction | yes | 0.5 | +| No legal SELECT elements | no | 0.5 | +| Disallowed / denied column | yes | 0.3 | +| Force-limit enforced | yes | 0.3 | +| Column masked | yes | 0.2 | +| SELECT * expansion | yes | 0.1 | + +> `risk` = mean of the weights recorded for *all* findings on the query. + +--- + +## 7. Error-handling strategy + +| Layer | Strategy | +|-------|----------| +| Config validation | Raises `UnsupportedRestrictionError` / `ValueError`, caught in `verify_sql` and converted to a hard-block dict. | +| Parsing | `sqlglot.errors.ParseError` caught; logged; recorded as a finding (does not raise to caller). | +| Value coercion | `_compare_values` / limit parsing wrap `ValueError`/`TypeError`/`AttributeError` with safe fallbacks. | +| REST | Returns `400` for non-JSON / missing `sql`/`config`; `401` for bad API key. | +| MCP | `KeyboardInterrupt`/`EOFError` caught to stop the inner container cleanly. | diff --git a/docs/technical-documentation/04-ARCHITECTURE.md b/docs/technical-documentation/04-ARCHITECTURE.md new file mode 100644 index 0000000..a82371a --- /dev/null +++ b/docs/technical-documentation/04-ARCHITECTURE.md @@ -0,0 +1,210 @@ +# 04 · Architecture, Security, Performance & Refactoring + +> Architectural decisions, design patterns, the security model, performance considerations, and a prioritised refactoring/technical-debt register. + +See also: [02-HLD](02-HLD.md) · [03-LLD](03-LLD.md) · [05-CODING_GUIDELINES](05-CODING_GUIDELINES.md) + +--- + +## 1. Architectural style + +`sql-data-guard` is a **layered, pipeline-oriented validator** built around a single pure function (`verify_sql`) with thin delivery adapters (REST, MCP). The processing core is an **AST-rewriting pipeline**: parse → scan → traverse/mutate → serialise. + +```mermaid +flowchart TD + subgraph Adapters + REST[REST adapter] + MCP[MCP adapter] + end + subgraph Application + ORCH["verify_sql (orchestrator)"] + end + subgraph Domain + RULES["restriction / column / mask / injection rules"] + CTX["VerificationContext (state)"] + end + subgraph Infra + SG[sqlglot] + end + REST --> ORCH + MCP --> ORCH + ORCH --> RULES + ORCH --> CTX + RULES --> CTX + ORCH --> SG + RULES --> SG +``` + +--- + +## 2. Design patterns in use + +| Pattern | Where | Benefit / drawback | +|---------|-------|--------------------| +| **Facade** | `__init__.py` exposes only `verify_sql` | Stable public surface; internals free to change. | +| **Pipeline / Chain of stages** | `verify_sql` orchestration | Clear, ordered stages; easy to insert new checks. Drawback: the function is long. | +| **Strategy** | Mask policies (`redact`/`hash`/`partial`); restriction operators | New policies/operators added without touching callers. | +| **Visitor-ish traversal** | `_verify_query_statement` recursion over sqlglot nodes | Handles arbitrary nesting (CTEs, sub-queries, set ops). | +| **Accumulator / Context object** | `VerificationContext` | Threads mutable result state without global state; testable. | +| **Adapter** | `rest/` and `mcpwrapper/` wrap the core | Multiple delivery surfaces share one engine. | +| **Fail-fast guard clauses** | Config + length gates at the top of `verify_sql` | Cheap rejection before expensive parsing. | + +--- + +## 3. Security model + +### 3.1 Trust boundaries + +```mermaid +flowchart LR + UNTRUSTED["UNTRUSTED
    LLM-generated SQL"] --> GUARD["sql-data-guard
    (trust boundary)"] + GUARD --> TRUSTED["TRUSTED
    Database"] +``` + +The guard treats *all* incoming SQL as hostile. The configuration is the *trusted policy*. + +### 3.2 Controls (mapped to attack classes) + +| Attack class | Control | Module | +|--------------|---------|--------| +| Column over-exposure | Column allow-list + deny-list (F10) + masking | `sql_data_guard`, `column_masking` | +| Row over-exposure / multi-tenant leak | Mandatory restrictions injected into WHERE | `restriction_verification` | +| Always-true / boolean injection (`OR 1=1`) | Static-expression detection → `FALSE` + simplify | `sql_data_guard` | +| Stacked / multi-statement (`; DROP`) | Statement-count + Block-node detection | `verify_sql`, `injection_detection` | +| Dangerous functions (`LOAD_FILE`, `xp_cmdshell`, `SLEEP`) | Always-on deny-list + config deny/allow (F2) | `injection_detection`, `sql_data_guard` | +| System-catalog exfiltration | Catalog table deny-list | `injection_detection` | +| Comment evasion (`-- payload`) | Pre-parse raw scan (opt-in) | `injection_detection` | +| UNION/EXCEPT/INTERSECT smuggling | Each set-op arm verified independently (S4) | `sql_data_guard` | +| Sub-query column smuggling | Dynamic columns resolved, not blanket-trusted (S3) | `sql_data_guard` | +| Injection via the *fix itself* | Safe literal escaping (S1) | `restriction_verification` | +| Oversized payload DoS | `max_length` gate | `verify_sql` | +| Excessive data return | `force_limit` row cap (F3) | `sql_data_guard` | +| Risk accumulation | `max_risk` hard-block threshold (F5) | `verify_sql` | +| Unauthorised API access | Optional `X-API-Key` (S6) | `rest` | + +### 3.3 Posture + +- **Read-only by default:** `Delete`/`Insert`/`Update`/`Create`/`Command` are hard-blocked. +- **Default-deny:** unknown tables/columns are rejected/stripped. +- **Defence in depth:** complements, does not replace, DB permissions. + +### 3.4 Residual risks (from F1 doc + code review) + +- **Comment detection is coarse** — flags `--`/`/* */` anywhere when enabled, including inside string literals (potential false positives). A precise version would strip string literals first. +- **Risk is an average**, diluting a single high-risk finding among many low-risk ones (see §6). +- **MCP wrapper non-fixable fallback** builds a `UNION ALL SELECT ''` string from error messages; error text is developer-controlled, but the pattern is worth auditing for injection if messages ever incorporate user input. + +--- + +## 4. Performance considerations + +### 4.1 Python / engine + +| Aspect | Observation | Note | +|--------|-------------|------| +| Parse cost | Single `sqlglot.parse` per call | Dominant cost; bounded by `max_length`. | +| Traversal | One recursive pass; `find_all` used several times per query | For very large queries, repeated `find_all` could be consolidated. | +| In-place mutation | Avoids re-parsing to produce `fixed` | Efficient — `fixed = AST.sql()` is one serialisation. | +| Allocation | New `VerificationContext` per call | Cheap; stateless across calls (thread-safe). | + +### 4.2 Database + +`sql-data-guard` does not run SQL in production, but its **rewrites affect DB performance**: + +- Injected restrictions add `WHERE` predicates — generally *improve* selectivity (encourage index use). +- `SELECT *` expansion lists explicit columns — neutral-to-positive (avoids reading unneeded columns only if projection is pushed down). +- Masking wraps columns in `MD5`/`CONCAT`/`SUBSTRING` — these are **non-sargable**; masked columns won't use indexes (acceptable since they're output transforms, not filters). +- `force_limit` adds `LIMIT` — caps result transfer and can enable early termination. + +### 4.3 API + +- Flask dev server (`app.run`) is used directly — fine for dev; **production should front it with a WSGI server** (gunicorn/uWSGI) behind TLS. + +--- + +## 5. Concurrency & thread-safety + +- The core is **stateless between calls**; all per-query state lives in a fresh `VerificationContext`. `verify_sql` is therefore safe to call concurrently. +- The MCP wrapper uses a daemon thread to stream the inner container's stdout and mutates a module-level `errors` dict keyed by request id — **the only shared mutable state**; under high concurrency this dict access is not explicitly locked (low risk for stdio single-agent use, but noted). + +--- + +## 6. Technical debt & risk register + +### High + +| ID | Item | Rationale | Suggested fix | +|----|------|-----------|---------------| +| H1 | **Risk = mean** dilutes severity | Many low-risk fixes can mask one 0.9 finding; weakens `max_risk` policy | Switch to `max(weights)` or a weighted scheme; keep mean as secondary metric. | +| H2 | **Coarse comment scan** | False positives on comments inside string literals | Strip string literals before regex; or tokenise. | +| H3 | **`verify_sql` length/complexity** | The orchestrator function does config-gate + parse + scan + dispatch (cyclomatic > target) | Extract `_parse_statements`, `_dispatch_statement`, `_finalise_result`. | + +### Medium + +| ID | Item | Suggested fix | +|----|------|---------------| +| M1 | `_verify_col` boolean chain is large/hard to read | Extract named predicates (`_is_dynamic_match`, `_is_config_column`). | +| M2 | Repeated `for config_t in context.config["tables"]` scans | Precompute `{table_name: config_table}` once in the context. | +| M3 | REST runs Flask dev server | Document/ship a gunicorn entrypoint for production. | +| M4 | MCP shared `errors` dict unlocked | Guard with a lock or use per-request correlation that doesn't share state. | + +### Low + +| ID | Item | Suggested fix | +|----|------|---------------| +| L1 | Magic literals (`"sub_select"`, mask defaults) | Promote to named constants. | +| L2 | `_format_value` numeric branch returns `str(value)` unquoted | Fine for numerics; document the invariant. | +| L3 | Comment patterns recompiled at import | Acceptable; already module-level constants. | + +--- + +## 7. Selected refactoring opportunities (with examples) + +### 7.1 Risk aggregation (H1) + +**Current** +```python +@property +def risk(self) -> float: + return sum(self._risk) / len(self._risk) if self._risk else 0 +``` + +**Proposed** (severity-preserving) +```python +@property +def risk(self) -> float: + return max(self._risk) if self._risk else 0.0 +``` +*Benefit:* a single dangerous-function finding (0.9) is no longer diluted by several 0.1 SELECT-* findings, so `max_risk` thresholds behave intuitively. *Impact:* would change some existing risk assertions — coordinate with tests (documented change, not silent). + +### 7.2 Table lookup index (M2) + +**Current:** nested loops over `config["tables"]` in `_verify_from_tables`, `_verify_col`, `_expand_star`, `verify_restrictions`. + +**Proposed:** build `context._tables_by_name = {t["table_name"]: t for t in config["tables"]}` once; O(1) lookups thereafter. + +### 7.3 Decompose `verify_sql` (H3) + +Split into: +```python +def verify_sql(sql, config, dialect=None): + guard = _pre_checks(sql, config) # config + length gates + if guard: return guard + ctx = VerificationContext(config, dialect) + parsed = _scan_and_parse(sql, ctx, dialect) + _dispatch_statement(parsed, ctx, dialect) + return _finalise(parsed, ctx, config, dialect) +``` +*Benefit:* each helper is independently testable and under the complexity limit. + +--- + +## 8. Architecture decision summary + +| Decision | Why | Trade-off | +|----------|-----|-----------| +| AST-based (sqlglot) over regex | Robust to whitespace/case/dialect tricks; enables safe rewriting | Adds a parse dependency; limited by sqlglot's grammar coverage. | +| In-place AST mutation + re-serialise | Single-pass auto-fix | Mutating shared nodes requires careful ordering (CTEs registered twice). | +| Opt-in for all new policies | Backward compatibility (280+ tests stayed green) | Secure defaults rely on the operator enabling features. | +| Read-only posture | LLM queries should never write | Write use-cases need explicit future support. | +| Mean risk (current) | Simple | Dilution problem (H1). | diff --git a/docs/technical-documentation/05-CODING_GUIDELINES.md b/docs/technical-documentation/05-CODING_GUIDELINES.md new file mode 100644 index 0000000..6a5e3d3 --- /dev/null +++ b/docs/technical-documentation/05-CODING_GUIDELINES.md @@ -0,0 +1,139 @@ +# 05 · Coding Guidelines & Contribution Standards + +> Conventions, security mindset, testing expectations, and a PR checklist tailored to `sql-data-guard`. These build on the project's existing `.clinerules` (PEP 8 + code-quality + security) and `CONTRIBUTING.md`. + +See also: [01-CODE_EXPLANATION](01-CODE_EXPLANATION.md) · [04-ARCHITECTURE](04-ARCHITECTURE.md) + +--- + +## 1. Golden rules (project-specific) + +1. **Read before you edit.** Trace execution end-to-end; security logic is subtle. +2. **Prefer AST-based validation over regex.** Regex is a last resort (only the pre-parse comment scan uses it, by necessity). +3. **Never break backward compatibility silently.** New policies must be **opt-in** and documented. +4. **Every change ships with:** rationale · security impact · tests · doc update. +5. **Fail closed.** When in doubt, block (`can_fix=False`) rather than allow. +6. **The fix must be safe.** Any SQL the guard *generates* must use escaped literals (`_format_value`). + +--- + +## 2. Style & formatting + +| Topic | Rule | +|-------|------| +| Formatter | `black` (project default); 4-space indent, ≤ 99-char lines (PEP 8). | +| Imports | stdlib → third-party → local, alphabetical within groups. | +| Naming | `snake_case` functions/vars, `PascalCase` classes, `UPPER_SNAKE_CASE` constants, `_prefix` for private. | +| Type hints | Required on all function signatures (`def verify_sql(sql: str, config: dict, dialect: str = None) -> dict`). | +| Docstrings | Required on public functions/modules; explain **why**, not just what. | +| f-strings | Preferred for interpolation. | +| Logging | Use `logging`, never `print()` in library code. | + +--- + +## 3. Complexity & size limits (from code-quality rules) + +| Metric | Limit | +|--------|-------| +| Cyclomatic / cognitive complexity | < 15 | +| Nesting depth | ≤ 3 | +| Function length | < 50 lines | +| Parameters | ≤ 4 | +| Duplication | none ≥ 3 lines | + +> `verify_sql` and `_verify_col` currently push these limits — see [04-ARCHITECTURE §6](04-ARCHITECTURE.md#6-technical-debt--risk-register) (H3, M1). New code must stay within limits; touch those hotspots only with accompanying tests. + +--- + +## 4. Security mindset (review like an attacker) + +When adding or reviewing rule code, actively look for: + +- **Allow-list bypasses:** can a column/table reach the output via a sub-query, CTE, lateral, `UNNEST`, or set-operation arm without being checked? +- **Restriction bypasses:** can a `WHERE` predicate *appear* to satisfy a restriction while being neutralised by `NOT`, `OR`, parentheses, or type confusion? +- **Comment / encoding tricks:** does the payload survive a `--`, `/* */`, `#`, or unusual whitespace? +- **Parser confusion:** does a dialect parse the statement differently than expected (test across dialects)? +- **Fix-as-vector:** does an injected restriction or mask use raw string concatenation? It must not. + +Map every new control to an attack class (see [04-ARCHITECTURE §3.2](04-ARCHITECTURE.md#32-controls-mapped-to-attack-classes)). + +### Security rules quick reference (applies here) + +- Parameterised/escaped values only — never concatenate untrusted strings into SQL. +- Catch **specific** exceptions; never an empty `except:`. +- No secrets in code or logs; the optional API key comes from `SQL_GUARD_API_KEY` env var. +- Validate config **before** parsing (fail fast). + +--- + +## 5. Testing standards + +The suite is the project's safety net. Conventions observed in `test/`: + +| Convention | Detail | +|------------|--------| +| Framework | `pytest`; classes group related cases (`TestStackedQueries`, `TestColumnMaskRewrite`, …). | +| Helper | `conftest.py::verify_sql_test` asserts `errors`, `risk`, and `fixed`, and (optionally) executes against a real DB connection. | +| Multi-dialect | Parametrise across `sqlite` / `mysql` / `postgres` / `trino` / `duckdb` where behaviour can differ. | +| Real execution | Many tests run the `fixed` SQL against SQLite/DuckDB to prove it's valid and returns the expected rows. | +| Risk assertions | `errors present ⇒ risk > 0`; `no errors ⇒ risk == 0`. | +| Determinism | `errors` is a list — order-insensitive comparisons via `set(...)`. | + +### What every new feature/fix needs + +1. **Positive test** — the legitimate case still passes (non-breaking). +2. **Negative test** — the attack/violation is caught with the right message and risk. +3. **Fix test** — if fixable, assert the exact `fixed` string *and* (ideally) execute it. +4. **Validation test** — bad config produces a graceful error, not a crash. +5. **False-positive guard** — a benign look-alike (e.g. a column named `sys`) is *not* flagged. + +```bash +# Run the suite (from repo root) +pip install -r test/test.requirements.txt +PYTHONPATH=src pytest test -q +``` + +--- + +## 6. Adding a new policy — recipe + +To add, say, a new restriction operator or mask policy: + +1. **Validate** the config shape in `restriction_validation.py` / `column_masking.py` (fail fast). +2. **Enforce/rewrite** in the relevant module, reporting via `context.add_error(msg, can_fix, risk)`. +3. **Choose a risk weight** consistent with the [taxonomy](03-LLD.md#6-error-taxonomy--risk-weights); use a named constant. +4. **Keep it opt-in** — default behaviour unchanged. +5. **Document** in a `docs/FEATURE_Fx_*.md` and update [00-INDEX](00-INDEX.md). +6. **Test** per §5. + +--- + +## 7. Deliverable format for findings (from `.clinerules`) + +For every security/quality task, produce: + +1. **Findings** — what's wrong, with evidence (run the code, don't assume). +2. **Root Cause** — why it happens. +3. **Recommended Fix** — implementable, with code. +4. **Impact** — security + backward-compatibility. +5. **Test Cases** — positive, negative, fix, validation. +6. **Documentation Updates** — which docs change. + +Never stop at identifying an issue — always propose an implementable improvement. + +--- + +## 8. Pull-request checklist + +- [ ] Change is **minimal and focused**; no unrelated refactoring. +- [ ] New behaviour is **opt-in**; existing tests stay green. +- [ ] **Rationale + security impact** described in the PR. +- [ ] **Tests** added (positive / negative / fix / validation / false-positive). +- [ ] Multi-dialect tested where relevant. +- [ ] Any guard-generated SQL uses **escaped literals**. +- [ ] **Specific** exceptions caught; no empty `except`. +- [ ] Complexity/size limits respected (§3). +- [ ] Public APIs have **type hints + docstrings**. +- [ ] Risk weight chosen via **named constant**, consistent with taxonomy. +- [ ] Relevant **docs updated** (feature doc + this analysis set + README/manual if user-facing). +- [ ] `black` clean; imports sorted; no `print()` / unused imports. diff --git a/pyproject.toml b/pyproject.toml index 33b23ea..30a6204 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -9,7 +9,8 @@ package-dir = {"" = "src"} extra-index-url = ["https://pypi.org/simple"] [project] name = "sql-data-guard" -version = "UPDATED-BY-WORKFLOW" +# Local/dev placeholder; overwritten by the publish workflow before release. +version = "0.0.0.dev0" dependencies = [ "sqlglot" ] diff --git a/requirements.txt b/requirements.txt index bb0c364..4c2287d 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1 +1,3 @@ -sqlglot \ No newline at end of file +sqlglot +flask +flasgger \ No newline at end of file diff --git a/src/sql_data_guard/column_masking.py b/src/sql_data_guard/column_masking.py new file mode 100644 index 0000000..3524a7c --- /dev/null +++ b/src/sql_data_guard/column_masking.py @@ -0,0 +1,126 @@ +"""Column-level masking / redaction. + +Instead of *removing* a sensitive column from the SELECT clause (which changes the +result shape and breaks downstream applications), sql-data-guard can rewrite the +column expression into a masking function. The query keeps returning a column with +the same output name, but the sensitive value never leaves the trust boundary. + +Masking is declared per table via a ``column_masks`` list in the config:: + + { + "table_name": "users", + "columns": ["id", "email", "credit_card"], + "column_masks": [ + {"column": "credit_card", "policy": "partial", "show_last": 4}, + {"column": "email", "policy": "hash"}, + {"column": "ssn", "policy": "redact"} + ] + } + +Supported policies: + +* ``redact`` -> replace the value with a constant string (default ``'****'``). + Configure the constant via ``"replacement"``. +* ``hash`` -> replace the value with a one-way hash (``MD5(col)``), portable + across dialects via sqlglot. +* ``partial`` -> keep the last ``show_last`` characters, mask the rest + (e.g. ``****1234``). ``show_last`` defaults to 4. +""" + +from typing import Dict, Optional + +import sqlglot +import sqlglot.expressions as expr + +SUPPORTED_MASK_POLICIES = {"redact", "hash", "partial"} + +_DEFAULT_REPLACEMENT = "****" +_DEFAULT_SHOW_LAST = 4 + + +def validate_column_masks(config: dict) -> None: + """Validate the ``column_masks`` section of every table in the config. + + Raises: + ValueError: if a mask references an unknown policy, targets a column that + is not in the table's ``columns`` allow-list, or uses an invalid + ``show_last`` value. + """ + for table in config.get("tables", []): + masks = table.get("column_masks") + if not masks: + continue + if not isinstance(masks, list): + raise ValueError( + f"'column_masks' for table '{table.get('table_name')}' must be a list." + ) + allowed_columns = set(table.get("columns", [])) + for mask in masks: + column = mask.get("column") + if not column: + raise ValueError( + f"Each column mask in table '{table.get('table_name')}' must have a 'column'." + ) + if column not in allowed_columns: + raise ValueError( + f"Masked column '{column}' must also appear in the 'columns' " + f"allow-list of table '{table.get('table_name')}'." + ) + policy = mask.get("policy") + if policy not in SUPPORTED_MASK_POLICIES: + raise ValueError( + f"Unsupported mask policy '{policy}' for column '{column}'. " + f"Supported policies: {sorted(SUPPORTED_MASK_POLICIES)}." + ) + if policy == "partial": + show_last = mask.get("show_last", _DEFAULT_SHOW_LAST) + if not isinstance(show_last, int) or show_last < 0: + raise ValueError( + f"'show_last' for column '{column}' must be a non-negative integer." + ) + + +def build_mask_lookup(config: dict) -> Dict[str, Dict[str, dict]]: + """Build a ``{table_name: {column_name: mask_spec}}`` lookup from the config.""" + lookup: Dict[str, Dict[str, dict]] = {} + for table in config.get("tables", []): + masks = table.get("column_masks") + if not masks: + continue + lookup[table["table_name"]] = {m["column"]: m for m in masks} + return lookup + + +def build_mask_expression( + mask: dict, column: expr.Column, dialect: Optional[str] +) -> expr.Expression: + """Build the masking expression for ``column`` according to ``mask``. + + The returned expression preserves the original output name via an alias so the + rewritten query yields a column with the same name the caller asked for. + """ + output_name = column.alias_or_name + policy = mask["policy"] + + if policy == "redact": + replacement = mask.get("replacement", _DEFAULT_REPLACEMENT) + masked: expr.Expression = expr.Literal.string(replacement) + elif policy == "hash": + masked = sqlglot.parse_one( + f"MD5({column.sql(dialect=dialect)})", dialect=dialect + ) + elif policy == "partial": + show_last = mask.get("show_last", _DEFAULT_SHOW_LAST) + replacement = mask.get("replacement", _DEFAULT_REPLACEMENT) + col_sql = column.sql(dialect=dialect) + # '****' || SUBSTRING(col FROM -show_last) — keep the last `show_last` chars. + # SUBSTRING with a negative start is normalized by sqlglot per dialect on + # output (e.g. SUBSTR for sqlite), which keeps the mask portable. + masked = sqlglot.parse_one( + f"CONCAT('{replacement}', SUBSTRING(CAST({col_sql} AS VARCHAR), -{show_last}))", + dialect=dialect, + ) + else: # pragma: no cover - guarded by validate_column_masks + raise ValueError(f"Unsupported mask policy '{policy}'.") + + return expr.alias_(masked, output_name) diff --git a/src/sql_data_guard/injection_detection.py b/src/sql_data_guard/injection_detection.py new file mode 100644 index 0000000..f54a0fe --- /dev/null +++ b/src/sql_data_guard/injection_detection.py @@ -0,0 +1,205 @@ +"""Malicious-payload / SQL-injection detection module. + +This module implements the "built-in module for detection of malicious payloads" +that the project README advertises. It performs two complementary scans: + +1. A *pre-parse* raw-string scan that catches attacks which the AST traversal + cannot see, most importantly comment-based evasion (``--``, ``/* */``, ``#``) + that hides a payload tail from sqlglot. +2. An *AST* scan that flags known attack constructs (stacked / multiple + statements, dangerous functions, system-catalog probing) with + *intent-revealing* error messages and risk weights, instead of relying on + incidental rules such as "static expression is not allowed". + +The functions here only *report* findings through +:class:`~sql_data_guard.verification_context.VerificationContext`. They never +mutate the query. Stripping/auto-fix remains the responsibility of the core +verifier. +""" + +from __future__ import annotations + +import re +from typing import List, Pattern, Tuple + +import sqlglot.expressions as expr + +from .verification_context import VerificationContext + +# Risk weights. Kept in named constants so the magic numbers are documented +# in one place and easy to tune. +_RISK_STACKED_QUERY = 0.9 +_RISK_COMMENT_EVASION = 0.8 +_RISK_DANGEROUS_FUNCTION = 0.9 +_RISK_SYSTEM_CATALOG = 0.8 + +# Functions that are dangerous regardless of the configured allow-list: +# file access, command execution and time-based blind-probe primitives. +_DANGEROUS_FUNCTIONS = frozenset( + { + "load_file", + "load_data", + "xp_cmdshell", + "pg_read_file", + "pg_sleep", + "sleep", + "benchmark", + "waitfor", + "dbms_pipe", + "sys_exec", + "sys_eval", + } +) + +# System catalogs / metadata tables that legitimate application queries should +# never need. Probing them is a classic information-schema exfiltration step. +_SYSTEM_CATALOGS = frozenset( + { + "information_schema", + "sqlite_master", + "sqlite_temp_master", + "pg_catalog", + } +) + + +# Pre-parse patterns. Each tuple is (compiled_regex, error_message, risk). +# These run on the raw SQL string *before* sqlglot parsing. +_COMMENT_PATTERNS: List[Tuple[Pattern[str], str, float]] = [ + ( + re.compile(r"--"), + "Comment-based evasion detected: '--' line comment is not allowed", + _RISK_COMMENT_EVASION, + ), + ( + re.compile(r"/\*.*?\*/", re.DOTALL), + "Comment-based evasion detected: '/* */' block comment is not allowed", + _RISK_COMMENT_EVASION, + ), + ( + re.compile(r"#"), + "Comment-based evasion detected: '#' comment is not allowed", + _RISK_COMMENT_EVASION, + ), +] + + +def scan_raw_sql(sql: str, context: VerificationContext) -> None: + """Scan the raw SQL string for attacks invisible to the AST. + + Comment-based evasion is *opt-in* because legitimate queries may contain + comments. It is only scanned when ``detect_comments`` is enabled in the + config (see :func:`comment_detection_enabled`). + + Args: + sql: The raw, unparsed SQL query. + context: The verification context to report findings to. + """ + if not comment_detection_enabled(context.config): + return + for pattern, message, risk in _COMMENT_PATTERNS: + if pattern.search(sql): + context.add_error(message, False, risk) + + +def scan_parsed_sql(parsed: expr.Expression, context: VerificationContext) -> None: + """Scan the parsed AST for known malicious constructs. + + Stacked statements, dangerous functions and system-catalog probing are + *always on* (they have no legitimate use in an application query). Only + comment detection (in :func:`scan_raw_sql`) is opt-in. + + Args: + parsed: The root sqlglot expression produced by ``parse_one``. + context: The verification context to report findings to. + """ + _scan_stacked_statements(parsed, context) + _scan_dangerous_functions(parsed, context) + _scan_system_catalogs(parsed, context) + + +def comment_detection_enabled(config: dict) -> bool: + """Return whether opt-in comment-evasion detection is enabled. + + Accepts either ``{"detect_injection": {"comments": True}}`` or the simple + shorthand ``{"detect_comments": True}``. Defaults to ``False`` so existing + queries containing benign comments are unaffected. + """ + if config.get("detect_comments"): + return True + detect = config.get("detect_injection") + if isinstance(detect, dict): + return bool(detect.get("comments")) + return False + + + +def is_stacked_statement(parsed: expr.Expression) -> bool: + """Return ``True`` if ``parsed`` represents more than one SQL statement. + + Newer sqlglot versions wrap stacked statements (``a; b``) in a ``Block`` + node. Detecting this explicitly lets the caller emit an intent-revealing + error instead of the incidental "could not find a query statement". + """ + block_cls = getattr(expr, "Block", None) + if block_cls is not None and isinstance(parsed, block_cls): + return len(parsed.expressions) > 1 + return False + + +def _scan_stacked_statements( + parsed: expr.Expression, context: VerificationContext +) -> None: + if is_stacked_statement(parsed): + context.add_error( + "Stacked query detected: multiple statements are not allowed", + False, + _RISK_STACKED_QUERY, + ) + + +def _scan_dangerous_functions( + parsed: expr.Expression, context: VerificationContext +) -> None: + for func in parsed.find_all(expr.Func): + name = function_name(func) + if name and name.lower() in _DANGEROUS_FUNCTIONS: + context.add_error( + f"Dangerous function detected: {name} is not allowed", + False, + _RISK_DANGEROUS_FUNCTION, + ) + + +def _scan_system_catalogs( + parsed: expr.Expression, context: VerificationContext +) -> None: + reported = set() + for table in parsed.find_all(expr.Table): + for part in (table.name, table.db, table.catalog): + if part and part.lower() in _SYSTEM_CATALOGS and part not in reported: + reported.add(part) + context.add_error( + f"System catalog probing detected: {part} is not allowed", + False, + _RISK_SYSTEM_CATALOG, + ) + + +def function_name(func: expr.Func) -> str: + """Best-effort extraction of a function's name across sqlglot versions. + + Unknown/dialect-specific functions (e.g. ``SLEEP``, ``LOAD_FILE``, + ``pg_sleep``) are parsed as ``Anonymous`` nodes whose ``sql_names()`` is + ``["ANONYMOUS"]`` and whose real name is on ``.name``. Built-in functions + expose the real name through ``sql_names()``. We prefer ``.name`` when the + only ``sql_names()`` entry is the placeholder ``ANONYMOUS``. + """ + sql_names = func.sql_names() + if func.name: + return func.name + if sql_names and sql_names[0] != "ANONYMOUS": + return sql_names[0] + return type(func).__name__ + + diff --git a/src/sql_data_guard/mcpwrapper/mcp_wrapper.py b/src/sql_data_guard/mcpwrapper/mcp_wrapper.py index 144c60a..9fcc46b 100644 --- a/src/sql_data_guard/mcpwrapper/mcp_wrapper.py +++ b/src/sql_data_guard/mcpwrapper/mcp_wrapper.py @@ -8,6 +8,12 @@ from sql_data_guard import verify_sql +# Module-level defaults so the module is import-safe; the real values are assigned +# in the __main__ block when run as a process (Q3). +config: dict = {} +inject_response: bool = False +errors: Dict[int, dict] = {} + def load_config() -> dict: return json.load(open("/conf/config.json")) @@ -125,5 +131,4 @@ def input_line(line: str) -> str: if __name__ == "__main__": config = load_config() inject_response = config["sql-data-guard"]["inject-response"] - errors: Dict[int, dict] = {} main() diff --git a/src/sql_data_guard/rest/sql_data_guard_rest.py b/src/sql_data_guard/rest/sql_data_guard_rest.py index a46bf9a..e28ed32 100644 --- a/src/sql_data_guard/rest/sql_data_guard_rest.py +++ b/src/sql_data_guard/rest/sql_data_guard_rest.py @@ -2,15 +2,111 @@ import os from logging.config import fileConfig +from flasgger import Swagger from flask import Flask, jsonify, request from sql_data_guard import verify_sql app = Flask(__name__) +swagger = Swagger( + app, + template={ + "info": { + "title": "sql-data-guard API", + "description": "Safety Layer for LLM Database Interactions. " + "Verifies and optionally rewrites SQL queries against a " + "restriction configuration.", + "version": "1.0", + } + }, +) + +# Optional API-key authentication (S6). Disabled unless SQL_GUARD_API_KEY is set, +# preserving the default open behavior for local/dev use. +_API_KEY = os.environ.get("SQL_GUARD_API_KEY") + + +@app.before_request +def _require_api_key(): + if _API_KEY and request.headers.get("X-API-Key") != _API_KEY: + return jsonify({"error": "Unauthorized"}), 401 @app.route("/verify-sql", methods=["POST"]) def _verify_sql(): + """Verify an SQL query against a restriction configuration. + --- + consumes: + - application/json + produces: + - application/json + parameters: + - in: body + name: body + required: true + schema: + type: object + required: + - sql + - config + properties: + sql: + type: string + description: The SQL query to verify. + example: SELECT * FROM orders WHERE account_id = 123 + config: + type: object + description: >- + The restriction configuration: allowed tables, columns, + row restrictions, and optional column_masks (redact | hash | + partial) that rewrite sensitive columns instead of dropping them. + example: + tables: + - table_name: orders + columns: + - id + - product_name + - account_id + - credit_card + restrictions: + - column: account_id + value: 123 + column_masks: + - column: credit_card + policy: partial + show_last: 4 + dialect: + type: string + description: Optional SQL dialect (e.g. "sqlite", "postgres"). + example: sqlite + responses: + 200: + description: Verification result. + schema: + type: object + properties: + allowed: + type: boolean + description: Whether the query is allowed to run. + errors: + type: array + items: + type: string + description: List of restriction violations found. + fixed: + type: string + description: A rewritten, compliant query (null if none needed). + risk: + type: number + description: Risk score of the query. + 400: + description: Bad request (missing or invalid input). + schema: + type: object + properties: + error: + type: string + """ if not request.is_json: return jsonify({"error": "Request must be JSON"}), 400 data = request.get_json() diff --git a/src/sql_data_guard/restriction_validation.py b/src/sql_data_guard/restriction_validation.py index d0cd6c9..529397d 100644 --- a/src/sql_data_guard/restriction_validation.py +++ b/src/sql_data_guard/restriction_validation.py @@ -2,6 +2,11 @@ class UnsupportedRestrictionError(Exception): pass +# Allowed restriction operations (comparison + range/membership operators). +SUPPORTED_OPERATIONS = {"=", ">", "<", ">=", "<=", "BETWEEN", "IN"} +_COMPARISON_OPERATIONS = {">", "<", ">=", "<="} + + def validate_restrictions(config: dict): """ Validates the restrictions in the configuration to ensure only supported operations are used. @@ -11,71 +16,127 @@ def validate_restrictions(config: dict): Raises: UnsupportedRestrictionError: If an unsupported restriction operation is found. - ValueError: If there are no tables in the configuration. + ValueError: If the configuration or a restriction value is structurally invalid. """ - supported_operations = [ - "=", - ">", - "<", - ">=", - "<=", - "BETWEEN", - "IN", - ] # Allowed operations - # Ensure 'tables' exists in config and is not empty + # Validate the optional mandatory-row-cap option (feature F3). + if "force_limit" in config: + force_limit = config["force_limit"] + if ( + not isinstance(force_limit, int) + or isinstance(force_limit, bool) + or force_limit <= 0 + ): + raise UnsupportedRestrictionError( + f"Invalid 'force_limit': expected a positive integer. Received: {force_limit}" + ) + + # Validate the optional function allow-list / deny-list (feature F2). + for key in ("allowed_functions", "blocked_functions"): + if key in config: + value = config[key] + if not isinstance(value, list) or not all( + isinstance(v, str) for v in value + ): + raise UnsupportedRestrictionError( + f"Invalid '{key}': expected a list of function-name strings. Received: {value}" + ) + tables = config.get("tables", []) - # Check if tables are empty if not tables: raise ValueError("Configuration must contain at least one table.") for table in tables: - # Ensure that 'table_name' exists in each table if "table_name" not in table: raise ValueError("Each table must have a 'table_name' key.") - # Ensure that 'columns' exists and is not empty in each table if "columns" not in table or not table["columns"]: raise ValueError( "Each table must have a 'columns' key with valid column definitions." ) - restrictions = table.get("restrictions", []) - if not restrictions: - continue # Skip if no restrictions are provided - - for restriction in restrictions: - operation = restriction.get("operation") - if operation == "BETWEEN": - values = restriction.get("values") - if not ( - isinstance(values, list) - and len(values) == 2 - and all(isinstance(v, (int, float)) for v in values) - and values[0] < values[1] - ): - raise ValueError( - f"Invalid 'BETWEEN' format. Expected list of two numeric values where min < max. Received: {values}" - ) - - elif operation == "IN": - values = restriction.get("values") - if not ( - isinstance(values, list) - and len(values) == 2 - and all(isinstance(v, (int, float)) for v in values) - ): - raise ValueError( - f"Invalid 'IN' format. Expected list of two numeric values. Received: {values}" - ) - - elif operation == ">=": - # You may want to ensure the value provided is numeric for >= - value = restriction.get("value") - if not isinstance(value, (int, float)): - raise ValueError( - f"Invalid restriction value type for column '{restriction['column']}' in table '{table['table_name']}'. Expected a numeric value." - ) - - elif operation and operation.lower() not in supported_operations: + # Validate the optional negative (deny-list) column rules (feature F10). + if "denied_columns" in table: + denied = table["denied_columns"] + if not isinstance(denied, list) or not all( + isinstance(c, str) for c in denied + ): raise UnsupportedRestrictionError( - f"Invalid restriction: 'operation={operation}' is not supported." + f"Invalid 'denied_columns' for table '{table['table_name']}': " + f"expected a list of column-name strings. Received: {denied}" ) + + for restriction in table.get("restrictions", []): + _validate_restriction(restriction, table) + + +def _validate_restriction(restriction: dict, table: dict): + """ + Validates a single restriction and normalizes its operation to upper case. + + Normalizing in place lets callers use case-insensitive operations (e.g. "between") + while the downstream AST matching logic continues to compare against upper-case + operators only. + """ + raw_operation = restriction.get("operation") + # Normalize so operation matching is case-insensitive (C3). Symbol operators + # such as "=" are unaffected by upper(). + if isinstance(raw_operation, str): + operation = raw_operation.upper() + restriction["operation"] = operation + else: + # A missing operation defaults to scalar equality. + operation = "=" if raw_operation is None else raw_operation + + if operation == "BETWEEN": + _validate_between(restriction) + elif operation == "IN": + _validate_in(restriction) + elif operation in _COMPARISON_OPERATIONS: + _validate_comparison_value(restriction, table) + elif operation == "=": + if "value" not in restriction and "values" not in restriction: + raise ValueError( + f"Restriction for column '{restriction.get('column')}' with operation " + f"'=' must have a 'value' (or 'values')." + ) + else: + raise UnsupportedRestrictionError( + f"Invalid restriction: 'operation={raw_operation}' is not supported." + ) + + +def _validate_between(restriction: dict): + values = restriction.get("values") + if not ( + isinstance(values, list) + and len(values) == 2 + and all(isinstance(v, (int, float)) for v in values) + and values[0] < values[1] + ): + raise ValueError( + f"Invalid 'BETWEEN' format. Expected list of two numeric values " + f"where min < max. Received: {values}" + ) + + +def _validate_in(restriction: dict): + values = restriction.get("values") + if not (isinstance(values, list) and len(values) >= 1): + raise ValueError( + f"Invalid 'IN' format. Expected a non-empty list of values. Received: {values}" + ) + all_numeric = all(isinstance(v, (int, float)) for v in values) + all_strings = all(isinstance(v, str) for v in values) + if not (all_numeric or all_strings): + raise ValueError( + f"Invalid 'IN' format. All values must be of the same type " + f"(all numeric or all strings). Received: {values}" + ) + + +def _validate_comparison_value(restriction: dict, table: dict): + value = restriction.get("value") + if not isinstance(value, (int, float)): + raise ValueError( + f"Invalid restriction value type for column '{restriction.get('column')}' " + f"in table '{table['table_name']}'. Expected a numeric value." + ) diff --git a/src/sql_data_guard/restriction_verification.py b/src/sql_data_guard/restriction_verification.py index 1d6d83f..cf6ae12 100644 --- a/src/sql_data_guard/restriction_verification.py +++ b/src/sql_data_guard/restriction_verification.py @@ -6,6 +6,14 @@ from .verification_context import VerificationContext from .verification_utils import split_to_expressions +_COMPARISON_OPERATORS = (">", "<", ">=", "<=") +_COMPARISON_EXPR_TO_OP = { + expr.LT: "<", + expr.LTE: "<=", + expr.GT: ">", + expr.GTE: ">=", +} + def verify_restrictions( select_statement: expr.Query, @@ -14,7 +22,6 @@ def verify_restrictions( ): where_clause = select_statement.find(expr.Where) if where_clause is None: - where_clause = select_statement.find(expr.Where) and_exps = [] else: and_exps = list(split_to_expressions(where_clause.this, expr.And)) @@ -68,20 +75,23 @@ def _create_new_condition( Returns: condition expression """ - if restriction.get("operation") == "BETWEEN": + operation = restriction.get("operation") + if operation == "BETWEEN": operator = "BETWEEN" operand = f"{_format_value(restriction['values'][0])} AND {_format_value(restriction['values'][1])}" - elif restriction.get("operation") == "IN": + elif operation == "IN": operator = "IN" values = restriction.get("values", [restriction.get("value")]) - operand = f"({', '.join(map(str, values))})" + # Format each value so string members are safely quoted/escaped (S1). + operand = f"({', '.join(_format_value(v) for v in values)})" else: - operator = "=" - operand = ( - _format_value(restriction["value"]) - if "value" in restriction - else str(restriction["values"])[1:-1] - ) + # Preserve the restriction's comparison operator (e.g. '<') instead of + # forcing '='. Defaults to '=' for scalar equality restrictions. + operator = operation if operation in _COMPARISON_OPERATORS else "=" + if "value" in restriction: + operand = _format_value(restriction["value"]) + else: + operand = ", ".join(_format_value(v) for v in restriction["values"]) new_condition = sqlglot.parse_one( f"{table_prefix}{restriction['column']} {operator} {operand}", dialect=context.dialect, @@ -90,10 +100,11 @@ def _create_new_condition( def _format_value(value): + """Render a Python value as a safe SQL literal (escaping embedded quotes).""" if isinstance(value, str): - return f"'{value}'" - else: - return value + # sqlglot's string literal handles escaping of embedded single quotes. + return expr.Literal.string(value).sql() + return str(value) def _verify_restriction( @@ -145,19 +156,35 @@ def _verify_restriction( if isinstance(exp, (expr.LT, expr.LTE, expr.GT, expr.GTE)) and isinstance( exp.right, expr.Condition ): - if restriction.get("operation") not in [">=", ">", "<=", "<"]: + operation = restriction.get("operation") + if operation not in _COMPARISON_OPERATORS: return False - assert len(values) == 1 - if isinstance(exp, expr.LT) and restriction["operation"] == "<": - return str(exp.right.this) < values[0] - elif isinstance(exp, expr.LTE) and restriction["operation"] == "<=": - return str(exp.right.this) <= values[0] - elif isinstance(exp, expr.GT) and restriction["operation"] == ">": - return str(exp.right.this) > values[0] - elif isinstance(exp, expr.GTE) and restriction["operation"] == ">=": - return str(exp.right.this) >= values[0] - else: + if len(values) != 1: + return False + # The query's operator must match the restriction's operator. + if _COMPARISON_EXPR_TO_OP.get(type(exp)) != operation: return False + return _compare_values(str(exp.right.this), values[0], operation) + return False + + +def _compare_values(query_value: str, restriction_value: str, operation: str) -> bool: + """ + Compares two values numerically when possible, falling back to lexicographic + comparison for non-numeric strings. Prevents the "9" < "18" string-comparison bug. + """ + try: + left, right = float(query_value), float(restriction_value) + except (TypeError, ValueError): + left, right = query_value, restriction_value + if operation == "<": + return left < right + if operation == "<=": + return left <= right + if operation == ">": + return left > right + if operation == ">=": + return left >= right return False diff --git a/src/sql_data_guard/sql_data_guard.py b/src/sql_data_guard/sql_data_guard.py index 531ef64..88b8e46 100644 --- a/src/sql_data_guard/sql_data_guard.py +++ b/src/sql_data_guard/sql_data_guard.py @@ -5,11 +5,19 @@ import sqlglot.expressions as expr from sqlglot.optimizer.simplify import simplify +from .column_masking import build_mask_expression, validate_column_masks +from .injection_detection import ( + function_name, + is_stacked_statement, + scan_parsed_sql, + scan_raw_sql, +) from .restriction_validation import validate_restrictions, UnsupportedRestrictionError from .restriction_verification import verify_restrictions from .verification_context import VerificationContext from .verification_utils import split_to_expressions, find_direct + _DEFAULT_MAX_LENGTH = 10_000 @@ -48,20 +56,42 @@ def verify_sql(sql: str, config: dict, dialect: str = None) -> dict: "risk": 1.0, } - # First, validate restrictions + # First, validate restrictions and column masks try: validate_restrictions(config) + validate_column_masks(config) except UnsupportedRestrictionError as e: return {"allowed": False, "errors": [str(e)], "fixed": None, "risk": 1.0} + except ValueError as e: + return {"allowed": False, "errors": [str(e)], "fixed": None, "risk": 1.0} result = VerificationContext(config, dialect) + parsed = None + # Pre-parse scan for attacks invisible to the AST (e.g. comment evasion). + scan_raw_sql(sql, result) try: - parsed = sqlglot.parse_one(sql, dialect=dialect) + statements = [s for s in sqlglot.parse(sql, dialect=dialect) if s is not None] except sqlglot.errors.ParseError as e: logging.error(f"SQL: {sql}\nError parsing SQL: {e}") result.add_error(f"Error parsing sql: {e}", False, 0.9) - parsed = None + statements = [] + if len(statements) > 1: + # Reject stacked / multi-statement payloads explicitly rather than relying on + # sqlglot incidentally wrapping them in a non-query node (S7). Uses the same + # intent-revealing message as the injection_detection module's stacked scan. + result.add_error( + "Stacked query detected: multiple statements are not allowed", False, 0.9 + ) + elif len(statements) == 1: + parsed = statements[0] + elif len(result.errors) == 0: + result.add_error("Could not find a query statement", False, 0.7) if parsed: + # AST-based malicious-payload scan (stacked queries, dangerous + # functions, system-catalog probing). Always on. + scan_parsed_sql(parsed, result) + # Config-driven function allow-list / deny-list (feature F2). + _verify_functions(parsed, result) if isinstance(parsed, expr.Command): result.add_error(f"{parsed.name} statement is not allowed", False, 0.9) elif isinstance(parsed, (expr.Delete, expr.Insert, expr.Update, expr.Create)): @@ -70,18 +100,103 @@ def verify_sql(sql: str, config: dict, dialect: str = None) -> dict: ) elif isinstance(parsed, expr.Query): _verify_query_statement(parsed, result) + elif is_stacked_statement(parsed): + # Stacked statements are already reported by scan_parsed_sql with an + # intent-revealing message; avoid the misleading generic error. + pass else: result.add_error("Could not find a query statement", False, 0.7) - if result.can_fix and len(result.errors) > 0: + + if parsed is not None and isinstance(parsed, expr.Query) and result.can_fix: + _enforce_force_limit(parsed, result) + if result.can_fix and parsed is not None and len(result.errors) > 0: result.fixed = parsed.sql(dialect=dialect) + + allowed = len(result.errors) == 0 + fixed = result.fixed + max_risk = config.get("max_risk") + if max_risk is not None and result.risk > max_risk: + # Risk exceeds the configured threshold: refuse to auto-fix and hard-block (S5). + allowed = False + fixed = None return { - "allowed": len(result.errors) == 0, + "allowed": allowed, "errors": result.errors, - "fixed": result.fixed, + "fixed": fixed, "risk": result.risk, } +def _verify_functions(parsed: expr.Expression, context: VerificationContext): + """Enforce a config-driven function allow-list / deny-list (feature F2). + + Two optional, top-level config keys (function names are matched + case-insensitively): + + * ``blocked_functions`` -- any call to one of these functions is blocked. + * ``allowed_functions`` -- if present, *only* these functions may be called; + any other function is blocked. + + A violation is a hard block (not auto-fixable): stripping a function call + from an arbitrary position could silently change query semantics or produce + invalid SQL, so the query is rejected outright. This composes with the + always-on dangerous-function deny-list in :mod:`injection_detection`. + """ + blocked = context.config.get("blocked_functions") + allowed = context.config.get("allowed_functions") + if not blocked and not allowed: + return + blocked_lower = {f.lower() for f in blocked} if blocked else set() + allowed_lower = {f.lower() for f in allowed} if allowed else None + for func in parsed.find_all(expr.Func): + name = function_name(func) + if not name: + continue + lname = name.lower() + if lname in blocked_lower: + context.add_error(f"Function {name} is not allowed", False, 0.9) + elif allowed_lower is not None and lname not in allowed_lower: + context.add_error( + f"Function {name} is not in the allowed functions list", + False, + 0.9, + ) + + +def _enforce_force_limit(parsed: expr.Query, context: VerificationContext): + """Enforce a mandatory row cap on the outermost query (feature F3). + + If the config sets ``force_limit`` to a positive integer, the outermost + query must not return more rows than that cap: + + * No ``LIMIT`` present -> inject ``LIMIT force_limit``. + * ``LIMIT`` larger than the cap -> clamp it down to ``force_limit``. + * ``LIMIT`` at or below the cap -> left unchanged. + + Only the outermost statement is touched (sub-queries/CTEs are not), because + the cap protects the final result set returned to the caller. The rewrite is + fixable, so it surfaces in the ``fixed`` query just like other auto-fixes. + """ + force_limit = context.config.get("force_limit") + if not isinstance(force_limit, int) or isinstance(force_limit, bool): + return + if force_limit <= 0: + return + limit = parsed.args.get("limit") + if limit is not None: + try: + current = int(limit.expression.name) + except (AttributeError, ValueError, TypeError): + current = None + if current is not None and current <= force_limit: + return + action = f"clamped from {current} to" if current is not None else "set to" + else: + action = "set to" + parsed.set("limit", expr.Limit(expression=expr.Literal.number(force_limit))) + context.add_error(f"Row limit enforced: LIMIT {action} {force_limit}", True, 0.3) + + def _verify_where_clause( context: VerificationContext, select_statement: expr.Query, @@ -138,13 +253,17 @@ def _has_static_expression(context: VerificationContext, exp: expr.Expression) - def _verify_query_statement(query_statement: expr.Query, context: VerificationContext): - if isinstance(query_statement, expr.Union): + if isinstance(query_statement, expr.SetOperation): + # Covers UNION, EXCEPT and INTERSECT (all share the SetOperation base). _verify_query_statement(query_statement.left, context) _verify_query_statement(query_statement.right, context) return for cte in query_statement.ctes: + # Register early so recursive/forward CTE references resolve, then refresh + # after verification to capture columns expanded from SELECT * (S3). _add_table_alias(cte, context) _verify_query_statement(cte.this, context) + _add_table_alias(cte, context) from_tables = _verify_from_tables(context, query_statement) if context.can_fix: _verify_select_clause(context, query_statement, from_tables) @@ -194,18 +313,17 @@ def _verify_select_clause( def _verify_select_clause_element( from_tables: List[expr.Table], context: VerificationContext, e: expr.Expression ): - if isinstance(e, expr.Column): + if isinstance(e, expr.Column) and e.name == "*": + # Table-qualified wildcard (``t.*``) -- expand like ``*`` but scoped to + # the named table (feature F10). + _expand_star(e, from_tables, context, table_filter=e.table) + return False + elif isinstance(e, expr.Column): if not _verify_col(e, from_tables, context): return False + _apply_column_mask(e, from_tables, context) elif isinstance(e, expr.Star): - context.add_error("SELECT * is not allowed", True, 0.1) - for t in from_tables: - for config_t in context.config["tables"]: - if t.name == config_t["table_name"]: - for c in config_t["columns"]: - e.parent.set( - "expressions", e.parent.expressions + [sqlglot.parse_one(c)] - ) + _expand_star(e, from_tables, context) return False elif isinstance(e, expr.Tuple): result = True @@ -220,6 +338,45 @@ def _verify_select_clause_element( return True +def _expand_star( + e: expr.Expression, + from_tables: List[expr.Table], + context: VerificationContext, + table_filter: str = "", +): + """Replace a ``*`` / ``table.*`` wildcard with the allowed column list. + + Columns listed in a table's ``denied_columns`` are excluded from the + expansion (feature F10), so ``SELECT *`` never silently surfaces a denied + column. ``table_filter`` restricts expansion to a single table for the + qualified ``table.*`` form. + """ + context.add_error("SELECT * is not allowed", True, 0.1) + for t in from_tables: + if table_filter and table_filter not in (t.name, t.alias): + continue + for config_t in context.config["tables"]: + if t.name == config_t["table_name"]: + denied = set(config_t.get("denied_columns", [])) + for c in config_t["columns"]: + if c in denied: + continue + e.parent.set( + "expressions", e.parent.expressions + [sqlglot.parse_one(c)] + ) + + +def _denied_columns( + from_tables: List[expr.Table], context: VerificationContext +) -> set: + """Union of ``denied_columns`` across the config tables in this query.""" + denied = set() + for config_t in context.config["tables"]: + if any(t.name == config_t["table_name"] for t in from_tables): + denied.update(config_t.get("denied_columns", [])) + return denied + + def _verify_col( col: expr.Column, from_tables: List[expr.Table], context: VerificationContext ) -> bool: @@ -234,15 +391,34 @@ def _verify_col( Returns: bool: True if the column reference is allowed, False otherwise. """ + # A denied column (feature F10) is rejected even if it is otherwise + # allow-listed: deny wins, and the column is stripped from the SELECT. + if col.name in _denied_columns(from_tables, context): + context.add_error( + f"Column {col.name} is denied. Column should be removed from SELECT clause", + True, + 0.3, + ) + return False if ( col.table == "sub_select" or (col.table != "" and col.table in context.dynamic_tables) - or (all(t.name in context.dynamic_tables for t in from_tables)) + or ( + # All FROM tables are dynamic: allow only columns those dynamic tables + # actually expose, instead of blindly allowing everything (S3). + len(from_tables) > 0 + and all(t.name in context.dynamic_tables for t in from_tables) + and any( + col.name in context.dynamic_tables.get(t.name, set()) + for t in from_tables + ) + ) or ( col.table == "" and col.name - in [col for t_cols in context.dynamic_tables.values() for col in t_cols] + in [c for t_cols in context.dynamic_tables.values() for c in t_cols] ) + or (col.table == "" and col.name in context.dynamic_columns) or ( any( col.name in config_t["columns"] @@ -262,6 +438,42 @@ def _verify_col( return False +def _apply_column_mask( + col: expr.Column, from_tables: List[expr.Table], context: VerificationContext +): + """Rewrite a SELECT column into its masking expression, if one is configured. + + The column is only masked when it can be unambiguously attributed to a single + configured table that declares a mask for it. The rewrite preserves the output + column name, so the query shape is unchanged. + """ + if not context.column_masks: + return + + masked_tables = [ + t + for t in from_tables + if t.name in context.column_masks and col.name in context.column_masks[t.name] + ] + if len(masked_tables) != 1: + # No mask for this column, or ambiguous across multiple masked tables. + return + table = masked_tables[0] + + # If the column is table-qualified, it must refer to this table (by alias or name). + if col.table and col.table not in (table.alias, table.name): + return + + mask = context.column_masks[table.name][col.name] + masked_expr = build_mask_expression(mask, col, context.dialect) + col.replace(masked_expr) + context.add_error( + f"Column {col.name} is masked ({mask['policy']})", + True, + 0.2, + ) + + def _get_from_clause_tables( select_clause: expr.Query, context: VerificationContext ) -> List[expr.Table]: @@ -284,8 +496,11 @@ def _get_from_clause_tables( if isinstance(t, expr.Table): result.append(t) for l in find_direct(clause, expr.Subquery): - _add_table_alias(l, context) + # Verify (and expand SELECT *) before recording exposed columns so + # that the alias maps to the real, allowed columns rather than "*". _verify_query_statement(l.this, context) + _add_table_alias(l, context) + _register_dynamic_columns(l.this, context) for join_clause in join_clauses: for l in find_direct(join_clause, expr.Lateral): _add_table_alias(l, context) @@ -303,3 +518,15 @@ def _add_table_alias(exp: expr.Expression, context: VerificationContext): else: column_names = {c for c in exp.this.named_selects} context.dynamic_tables[table_alias.alias_or_name] = column_names + + +def _register_dynamic_columns(query: expr.Expression, context: VerificationContext): + """ + Record the columns a (possibly un-aliased) dynamic source exposes, so the outer + query may reference them un-prefixed without blindly allowing every column. + """ + if query is None: + return + for name in query.named_selects: + if name and name != "*": + context.dynamic_columns.add(name) diff --git a/src/sql_data_guard/verification_context.py b/src/sql_data_guard/verification_context.py index e290351..cf696e3 100644 --- a/src/sql_data_guard/verification_context.py +++ b/src/sql_data_guard/verification_context.py @@ -1,5 +1,7 @@ from typing import Set, Dict, List, Optional +from .column_masking import build_mask_lookup + class VerificationContext: """ @@ -7,35 +9,44 @@ class VerificationContext: Attributes: _can_fix (bool): Indicates if the query can be fixed. - _errors (List[str]): List of errors found during verification. + _errors (List[str]): Ordered, de-duplicated list of errors found during verification. _fixed (Optional[str]): The fixed query if modifications were made. _config (dict): The configuration used for verification. - _dynamic_tables (Set[str]): Set of dynamic tables found in the query, like sub select and WITH clauses. + _dynamic_tables (Dict[str, Set[str]]): Dynamic tables (sub selects, WITH clauses) mapped + to the column names they expose. + _dynamic_columns (Set[str]): Columns exposed by dynamic sources that have no usable alias + (e.g. un-aliased FROM sub-queries), accessible un-prefixed by the outer query. _dialect (str): The SQL dialect to use for parsing. + _risk (List[float]): Per-finding risk weights, averaged to produce the final risk score. """ def __init__(self, config: dict, dialect: str): super().__init__() self._can_fix = True - self._errors = set() + # Ordered + de-duplicated so the API response is deterministic (a set is not). + self._errors: List[str] = [] self._fixed = None self._config = config self._dynamic_tables: Dict[str, Set[str]] = {} + self._dynamic_columns: Set[str] = set() self._dialect = dialect self._risk: List[float] = [] + # {table_name: {column_name: mask_spec}} + self._column_masks: Dict[str, Dict[str, dict]] = build_mask_lookup(config) @property def can_fix(self) -> bool: return self._can_fix def add_error(self, error: str, can_fix: bool, risk: float): - self._errors.add(error) + if error not in self._errors: + self._errors.append(error) if not can_fix: self._can_fix = False self._risk.append(risk) @property - def errors(self) -> Set[str]: + def errors(self) -> List[str]: return self._errors @property @@ -54,10 +65,18 @@ def config(self) -> dict: def dynamic_tables(self) -> Dict[str, Set[str]]: return self._dynamic_tables + @property + def dynamic_columns(self) -> Set[str]: + return self._dynamic_columns + @property def dialect(self) -> str: return self._dialect + @property + def column_masks(self) -> Dict[str, Dict[str, dict]]: + return self._column_masks + @property def risk(self) -> float: return sum(self._risk) / len(self._risk) if len(self._risk) > 0 else 0 diff --git a/test/conftest.py b/test/conftest.py index a17e753..1298361 100644 --- a/test/conftest.py +++ b/test/conftest.py @@ -15,11 +15,11 @@ def verify_sql_test( ) -> str: result = verify_sql(sql, config, dialect) if errors is None: - assert result["errors"] == set() + assert not result["errors"] else: - expected_errors = list(errors) - actual_errors = list(result["errors"]) - assert actual_errors == expected_errors + # errors is now an ordered list; compare order-insensitively since callers + # pass an (unordered) set of expected messages. + assert set(result["errors"]) == set(errors) if len(result["errors"]) > 0: assert result["risk"] > 0 else: diff --git a/test/test.requirements.txt b/test/test.requirements.txt index 630d544..92f76c2 100644 --- a/test/test.requirements.txt +++ b/test/test.requirements.txt @@ -1,3 +1,4 @@ pytest duckdb -flask \ No newline at end of file +flask +flasgger diff --git a/test/test_column_masking_unit.py b/test/test_column_masking_unit.py new file mode 100644 index 0000000..83a3823 --- /dev/null +++ b/test/test_column_masking_unit.py @@ -0,0 +1,288 @@ +"""Tests for column-level masking / redaction.""" + +import sqlite3 + +import pytest + +from conftest import verify_sql_test +from sql_data_guard import verify_sql +from sql_data_guard.column_masking import validate_column_masks + + +class TestColumnMaskValidation: + """Validation of the column_masks section of the config.""" + + def test_valid_masks(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "email", "credit_card"], + "column_masks": [ + {"column": "credit_card", "policy": "partial", "show_last": 4}, + {"column": "email", "policy": "hash"}, + {"column": "id", "policy": "redact"}, + ], + } + ] + } + validate_column_masks(config) # should not raise + + def test_unsupported_policy(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id"], + "column_masks": [{"column": "id", "policy": "encrypt"}], + } + ] + } + with pytest.raises(ValueError, match="Unsupported mask policy"): + validate_column_masks(config) + + def test_mask_column_not_in_allow_list(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id"], + "column_masks": [{"column": "ssn", "policy": "redact"}], + } + ] + } + with pytest.raises(ValueError, match="must also appear in the 'columns'"): + validate_column_masks(config) + + def test_partial_invalid_show_last(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["card"], + "column_masks": [ + {"column": "card", "policy": "partial", "show_last": -1} + ], + } + ] + } + with pytest.raises(ValueError, match="show_last"): + validate_column_masks(config) + + def test_masks_must_be_list(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id"], + "column_masks": {"column": "id", "policy": "redact"}, + } + ] + } + with pytest.raises(ValueError, match="must be a list"): + validate_column_masks(config) + + def test_invalid_config_surfaces_through_verify_sql(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id"], + "column_masks": [{"column": "id", "policy": "encrypt"}], + } + ] + } + result = verify_sql("SELECT id FROM users", config, "sqlite") + assert result["allowed"] is False + assert result["risk"] == 1.0 + assert any("Unsupported mask policy" in e for e in result["errors"]) + + +class TestColumnMaskRewrite: + """Rewriting of SELECT columns into masking expressions.""" + + def test_redact_mask(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "ssn"], + "column_masks": [{"column": "ssn", "policy": "redact"}], + } + ] + } + verify_sql_test( + "SELECT id, ssn FROM users", + config, + errors={"Column ssn is masked (redact)"}, + fix="SELECT id, '****' AS ssn FROM users", + ) + + def test_redact_custom_replacement(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "ssn"], + "column_masks": [ + {"column": "ssn", "policy": "redact", "replacement": "REDACTED"} + ], + } + ] + } + verify_sql_test( + "SELECT ssn FROM users", + config, + errors={"Column ssn is masked (redact)"}, + fix="SELECT 'REDACTED' AS ssn FROM users", + ) + + def test_hash_mask(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "email"], + "column_masks": [{"column": "email", "policy": "hash"}], + } + ] + } + verify_sql_test( + "SELECT id, email FROM users", + config, + errors={"Column email is masked (hash)"}, + fix="SELECT id, MD5(email) AS email FROM users", + ) + + def test_partial_mask(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "credit_card"], + "column_masks": [ + {"column": "credit_card", "policy": "partial", "show_last": 4} + ], + } + ] + } + verify_sql_test( + "SELECT credit_card FROM users", + config, + errors={"Column credit_card is masked (partial)"}, + fix="SELECT '****' || SUBSTRING(CAST(credit_card AS TEXT), -4) AS credit_card FROM users", + ) + + def test_mask_with_restriction(self): + """Masking and row restrictions compose cleanly.""" + config = { + "tables": [ + { + "table_name": "orders", + "columns": ["id", "account_id", "card"], + "restrictions": [{"column": "account_id", "value": 123}], + "column_masks": [{"column": "card", "policy": "redact"}], + } + ] + } + verify_sql_test( + "SELECT id, card FROM orders WHERE account_id = 123", + config, + errors={"Column card is masked (redact)"}, + fix="SELECT id, '****' AS card FROM orders WHERE account_id = 123", + ) + + def test_unmasked_column_untouched(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "email"], + "column_masks": [{"column": "email", "policy": "redact"}], + } + ] + } + # Selecting only the unmasked column: nothing to fix. + verify_sql_test("SELECT id FROM users", config) + + def test_no_masks_configured(self): + config = { + "tables": [{"table_name": "users", "columns": ["id", "email"]}] + } + verify_sql_test("SELECT id, email FROM users", config) + + def test_table_qualified_column_is_masked(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "ssn"], + "column_masks": [{"column": "ssn", "policy": "redact"}], + } + ] + } + verify_sql_test( + "SELECT u.ssn FROM users AS u", + config, + errors={"Column ssn is masked (redact)"}, + fix="SELECT '****' AS ssn FROM users AS u", + ) + + +class TestColumnMaskExecution: + """The rewritten (fixed) query must actually run and hide the data.""" + + @pytest.fixture(scope="class") + def cnn(self): + conn = sqlite3.connect(":memory:") + conn.execute( + "CREATE TABLE users (id INT, email TEXT, credit_card TEXT)" + ) + conn.execute( + "INSERT INTO users VALUES (1, 'alice@example.com', '4111111111111234')" + ) + conn.commit() + yield conn + conn.close() + + def test_redact_executes_and_hides(self, cnn): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "credit_card"], + "column_masks": [{"column": "credit_card", "policy": "redact"}], + } + ] + } + verify_sql_test( + "SELECT id, credit_card FROM users", + config, + errors={"Column credit_card is masked (redact)"}, + fix="SELECT id, '****' AS credit_card FROM users", + cnn=cnn, + data=[[1, "****"]], + ) + + def test_partial_executes_and_keeps_last_four(self, cnn): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "credit_card"], + "column_masks": [ + {"column": "credit_card", "policy": "partial", "show_last": 4} + ], + } + ] + } + # sqlglot renders SUBSTRING(...) as sqlite-native SUBSTRING/|| so the masked + # query runs without any custom UDFs. + sql_to_use = verify_sql_test( + "SELECT id, credit_card FROM users", + config, + errors={"Column credit_card is masked (partial)"}, + fix="SELECT id, '****' || SUBSTRING(CAST(credit_card AS TEXT), -4) AS credit_card FROM users", + ) + row = cnn.execute(sql_to_use).fetchone() + assert row == (1, "****1234") diff --git a/test/test_denied_columns_unit.py b/test/test_denied_columns_unit.py new file mode 100644 index 0000000..4ba5fd1 --- /dev/null +++ b/test/test_denied_columns_unit.py @@ -0,0 +1,91 @@ +"""Unit tests for negative (deny-list) column rules & wildcard support (F10). + +Per-table ``denied_columns``: + +* an explicitly-referenced denied column is stripped from the SELECT, +* ``SELECT *`` expands to the allowed columns *minus* the denied ones, +* ``table.*`` expands the same way, scoped to that table, +* deny wins even when the column is otherwise allow-listed, +* the feature is a no-op when ``denied_columns`` is absent. +""" + +import pytest + +from sql_data_guard import verify_sql + + +def _config(denied=None) -> dict: + table = { + "table_name": "orders", + "columns": ["id", "product_name", "ssn", "account_id"], + } + if denied is not None: + table["denied_columns"] = denied + return {"tables": [table]} + + +class TestExplicitDeniedColumn: + def test_denied_column_is_stripped(self): + result = verify_sql( + "SELECT id, ssn FROM orders", _config(denied=["ssn"]), "sqlite" + ) + assert result["allowed"] is False + assert result["fixed"] == "SELECT id FROM orders" + assert ( + "Column ssn is denied. Column should be removed from SELECT clause" + in result["errors"] + ) + + def test_non_denied_column_passes(self): + result = verify_sql( + "SELECT id, product_name FROM orders", _config(denied=["ssn"]), "sqlite" + ) + assert result["allowed"] is True + assert result["fixed"] is None + + def test_qualified_denied_column_is_stripped(self): + result = verify_sql( + "SELECT orders.id, orders.ssn FROM orders", _config(denied=["ssn"]), "sqlite" + ) + assert result["allowed"] is False + assert ( + "Column ssn is denied. Column should be removed from SELECT clause" + in result["errors"] + ) + + +class TestWildcardExclusion: + def test_star_excludes_denied_columns(self): + result = verify_sql("SELECT * FROM orders", _config(denied=["ssn"]), "sqlite") + assert result["allowed"] is False + assert result["fixed"] == "SELECT id, product_name, account_id FROM orders" + assert "ssn" not in result["fixed"] + + def test_table_qualified_star_excludes_denied(self): + result = verify_sql( + "SELECT orders.* FROM orders", _config(denied=["ssn"]), "sqlite" + ) + assert result["allowed"] is False + assert "ssn" not in result["fixed"] + assert "product_name" in result["fixed"] + + +class TestNonBreaking: + def test_star_without_denied_expands_all(self): + result = verify_sql("SELECT * FROM orders", _config(), "sqlite") + assert result["fixed"] == ( + "SELECT id, product_name, ssn, account_id FROM orders" + ) + + def test_plain_query_without_denied_is_allowed(self): + result = verify_sql("SELECT id FROM orders", _config(), "sqlite") + assert result["allowed"] is True + assert result["fixed"] is None + + +class TestValidation: + @pytest.mark.parametrize("bad", ["ssn", [1, 2], {"a": 1}]) + def test_invalid_denied_columns_returns_graceful_error(self, bad): + result = verify_sql("SELECT id FROM orders", _config(denied=bad), "sqlite") + assert result["allowed"] is False + assert any("Invalid 'denied_columns'" in e for e in result["errors"]) diff --git a/test/test_function_policy_unit.py b/test/test_function_policy_unit.py new file mode 100644 index 0000000..6786297 --- /dev/null +++ b/test/test_function_policy_unit.py @@ -0,0 +1,114 @@ +"""Unit tests for the function allow-list / deny-list (feature F2). + +Two optional, top-level config keys (matched case-insensitively): + +* ``blocked_functions`` -- listed functions are blocked. +* ``allowed_functions`` -- only listed functions may be called. + +Violations are hard blocks (no auto-fix). The feature is a no-op when neither +key is present, and composes with the always-on dangerous-function deny-list +shipped by F1. +""" + +import pytest + +from sql_data_guard import verify_sql + + +def _config(allowed=None, blocked=None) -> dict: + config = { + "tables": [ + { + "table_name": "orders", + "columns": ["id", "product_name", "account_id"], + } + ] + } + if allowed is not None: + config["allowed_functions"] = allowed + if blocked is not None: + config["blocked_functions"] = blocked + return config + + +class TestBlockedFunctions: + @pytest.mark.parametrize( + "sql,dialect,fn", + [ + ("SELECT UPPER(product_name) FROM orders", "sqlite", "UPPER"), + ("SELECT id FROM orders WHERE LENGTH(product_name) > 3", "sqlite", "LENGTH"), + ], + ) + def test_blocked_function_is_rejected(self, sql, dialect, fn): + result = verify_sql(sql, _config(blocked=[fn]), dialect) + assert result["allowed"] is False + assert result["fixed"] is None # hard block, not auto-fixed + assert any(f"Function {fn} is not allowed" in e for e in result["errors"]) + + def test_block_is_case_insensitive(self): + # Lower-case call, upper-case deny-list entry: matching is + # case-insensitive even though sqlglot canonicalises the displayed name. + result = verify_sql( + "SELECT upper(product_name) FROM orders", _config(blocked=["UPPER"]), "sqlite" + ) + assert result["allowed"] is False + assert any("is not allowed" in e for e in result["errors"]) + + def test_unlisted_function_passes_with_deny_list(self): + result = verify_sql( + "SELECT COUNT(id) FROM orders", _config(blocked=["UPPER"]), "sqlite" + ) + assert result["allowed"] is True + assert result["errors"] == [] + + +class TestAllowedFunctions: + def test_allowed_function_passes(self): + result = verify_sql( + "SELECT COUNT(id) FROM orders", _config(allowed=["COUNT"]), "sqlite" + ) + assert result["allowed"] is True + + def test_non_allowed_function_is_blocked(self): + result = verify_sql( + "SELECT UPPER(product_name) FROM orders", + _config(allowed=["COUNT"]), + "sqlite", + ) + assert result["allowed"] is False + assert any( + "Function UPPER is not in the allowed functions list" in e + for e in result["errors"] + ) + + +class TestComposesWithF1: + def test_f1_dangerous_function_still_blocked_without_config(self): + # LOAD_FILE is always blocked by F1 even with no F2 policy. + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123 AND LOAD_FILE('/etc/passwd')", + _config(), + "mysql", + ) + assert result["allowed"] is False + assert any("Dangerous function detected: LOAD_FILE" in e for e in result["errors"]) + + +class TestNonBreaking: + def test_no_policy_is_noop(self): + result = verify_sql( + "SELECT COUNT(id) FROM orders", _config(), "sqlite" + ) + assert result["allowed"] is True + assert result["fixed"] is None + + +class TestValidation: + @pytest.mark.parametrize("key", ["allowed_functions", "blocked_functions"]) + @pytest.mark.parametrize("bad", ["UPPER", [1, 2], {"a": 1}]) + def test_invalid_function_list_returns_graceful_error(self, key, bad): + config = _config() + config[key] = bad + result = verify_sql("SELECT id FROM orders", config, "sqlite") + assert result["allowed"] is False + assert any(f"Invalid '{key}'" in e for e in result["errors"]) diff --git a/test/test_injection_detection_unit.py b/test/test_injection_detection_unit.py new file mode 100644 index 0000000..a05c2d5 --- /dev/null +++ b/test/test_injection_detection_unit.py @@ -0,0 +1,171 @@ +"""Unit tests for the malicious-payload / SQL-injection detection module (F1). + +These tests cover the new ``injection_detection`` module and its integration +with ``verify_sql``: + +* Stacked queries (always on) +* Dangerous functions (always on) +* System-catalog probing (always on) +* Comment-based evasion (opt-in via ``detect_comments`` / ``detect_injection``) + +They also lock in that the feature is *non-breaking*: a benign query with the +feature disabled is still allowed. +""" + +import pytest + +from sql_data_guard import verify_sql + + +def _config(detect_comments: bool = False) -> dict: + config = { + "tables": [ + { + "table_name": "orders", + "database_name": "orders_db", + "columns": ["id", "product_name", "account_id"], + "restrictions": [{"column": "account_id", "value": 123}], + } + ] + } + if detect_comments: + config["detect_comments"] = True + return config + + +def _errors(sql: str, config: dict, dialect: str = "sqlite") -> list: + return list(verify_sql(sql, config, dialect)["errors"]) + + +class TestStackedQueries: + def test_stacked_drop_is_blocked_with_intent_revealing_error(self): + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123; DROP TABLE orders", + _config(), + "sqlite", + ) + assert result["allowed"] is False + assert ( + "Stacked query detected: multiple statements are not allowed" + in result["errors"] + ) + + def test_stacked_select_select_is_blocked(self): + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123; " + "SELECT id FROM orders WHERE account_id = 123", + _config(), + "sqlite", + ) + assert result["allowed"] is False + assert ( + "Stacked query detected: multiple statements are not allowed" + in result["errors"] + ) + + def test_single_statement_is_not_flagged_as_stacked(self): + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123", _config(), "sqlite" + ) + assert result["allowed"] is True + assert result["errors"] == [] + + +class TestDangerousFunctions: + @pytest.mark.parametrize( + "sql,dialect,fragment", + [ + ( + "SELECT id FROM orders WHERE account_id = 123 AND SLEEP(5)", + "sqlite", + "SLEEP", + ), + ( + "SELECT id FROM orders WHERE account_id = 123 AND BENCHMARK(1000000, 1)", + "mysql", + "BENCHMARK", + ), + ( + "SELECT id FROM orders WHERE account_id = 123 AND LOAD_FILE('/etc/passwd')", + "mysql", + "LOAD_FILE", + ), + ("SELECT pg_sleep(5) FROM orders WHERE account_id = 123", "postgres", "pg_sleep"), + ], + ) + def test_dangerous_function_is_detected(self, sql, dialect, fragment): + errors = _errors(sql, _config(), dialect) + assert any( + e.startswith(f"Dangerous function detected: {fragment}") for e in errors + ), errors + + def test_benign_aggregate_function_is_allowed(self): + result = verify_sql( + "SELECT COUNT(id) FROM orders WHERE account_id = 123", _config(), "sqlite" + ) + assert result["allowed"] is True + + +class TestSystemCatalogProbing: + def test_information_schema_is_detected(self): + errors = _errors( + "SELECT id FROM information_schema.tables", _config(), "sqlite" + ) + assert any("System catalog probing detected: information_schema" in e for e in errors) + + def test_sqlite_master_is_detected(self): + errors = _errors( + "SELECT id FROM orders WHERE account_id = 123 " + "UNION SELECT name FROM sqlite_master", + _config(), + "sqlite", + ) + assert any("System catalog probing detected: sqlite_master" in e for e in errors) + + def test_common_name_is_not_a_false_positive(self): + # 'sys' is a plausible real table name and must NOT be flagged. + errors = _errors("SELECT id FROM sys", _config(), "sqlite") + assert not any("System catalog probing" in e for e in errors) + + +class TestCommentEvasion: + def test_comment_not_flagged_when_disabled(self): + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123 -- trailing", + _config(detect_comments=False), + "sqlite", + ) + # No injection error; comment detection is opt-in. + assert not any("Comment-based evasion" in e for e in result["errors"]) + + def test_line_comment_flagged_when_enabled(self): + errors = _errors( + "SELECT id FROM orders WHERE account_id = 123 -- ; DROP TABLE orders", + _config(detect_comments=True), + ) + assert any("Comment-based evasion detected: '--'" in e for e in errors) + + def test_block_comment_flagged_when_enabled(self): + errors = _errors( + "SELECT id FROM orders WHERE account_id = 123 /* hide */", + _config(detect_comments=True), + ) + assert any("Comment-based evasion detected: '/* */'" in e for e in errors) + + def test_detect_injection_nested_flag(self): + config = _config() + config["detect_injection"] = {"comments": True} + errors = _errors( + "SELECT id FROM orders WHERE account_id = 123 # hash", config + ) + assert any("Comment-based evasion detected: '#'" in e for e in errors) + + +class TestRiskScoring: + def test_injection_increases_risk(self): + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123 AND SLEEP(5)", + _config(), + "sqlite", + ) + assert result["risk"] > 0 diff --git a/test/test_limit_enforcement_unit.py b/test/test_limit_enforcement_unit.py new file mode 100644 index 0000000..97e5f89 --- /dev/null +++ b/test/test_limit_enforcement_unit.py @@ -0,0 +1,96 @@ +"""Unit tests for mandatory LIMIT / max-rows enforcement (feature F3). + +``force_limit`` caps the number of rows the outermost query may return: + +* injected when no ``LIMIT`` is present, +* clamped when an existing ``LIMIT`` exceeds the cap, +* left untouched when the existing ``LIMIT`` is within the cap, +* a no-op when the option is absent (non-breaking). +""" + +import pytest + +from sql_data_guard import verify_sql + + +def _config(force_limit=None) -> dict: + config = { + "tables": [ + { + "table_name": "orders", + "columns": ["id", "product_name", "account_id"], + } + ] + } + if force_limit is not None: + config["force_limit"] = force_limit + return config + + +class TestForceLimitInjection: + def test_limit_injected_when_absent(self): + result = verify_sql("SELECT id FROM orders", _config(1000), "sqlite") + assert result["allowed"] is False + assert result["fixed"] == "SELECT id FROM orders LIMIT 1000" + assert any("Row limit enforced" in e for e in result["errors"]) + + def test_limit_clamped_when_too_large(self): + result = verify_sql( + "SELECT id FROM orders LIMIT 5000", _config(1000), "sqlite" + ) + assert result["allowed"] is False + assert result["fixed"] == "SELECT id FROM orders LIMIT 1000" + assert any("clamped from 5000 to 1000" in e for e in result["errors"]) + + def test_limit_left_when_within_cap(self): + result = verify_sql( + "SELECT id FROM orders LIMIT 10", _config(1000), "sqlite" + ) + assert result["allowed"] is True + assert result["fixed"] is None + assert result["errors"] == [] + + def test_limit_equal_to_cap_is_allowed(self): + result = verify_sql( + "SELECT id FROM orders LIMIT 1000", _config(1000), "sqlite" + ) + assert result["allowed"] is True + assert result["fixed"] is None + + +class TestForceLimitUnion: + def test_union_limit_injected_on_outer_query(self): + result = verify_sql( + "SELECT id FROM orders UNION SELECT account_id FROM orders", + _config(1000), + "sqlite", + ) + assert result["allowed"] is False + assert result["fixed"].endswith("LIMIT 1000") + + +class TestForceLimitCombinedWithFix: + def test_limit_added_alongside_column_strip(self): + # 'secret' is not allowed -> stripped; force_limit also applied. + result = verify_sql( + "SELECT id, secret FROM orders", _config(1000), "sqlite" + ) + assert result["allowed"] is False + assert result["fixed"] == "SELECT id FROM orders LIMIT 1000" + assert any("Row limit enforced" in e for e in result["errors"]) + assert any("Column secret is not allowed" in e for e in result["errors"]) + + +class TestForceLimitNonBreaking: + def test_no_force_limit_is_noop(self): + result = verify_sql("SELECT id FROM orders", _config(), "sqlite") + assert result["allowed"] is True + assert result["fixed"] is None + + +class TestForceLimitValidation: + @pytest.mark.parametrize("bad", [0, -5, "1000", 10.5, True]) + def test_invalid_force_limit_returns_graceful_error(self, bad): + result = verify_sql("SELECT id FROM orders", _config(bad), "sqlite") + assert result["allowed"] is False + assert any("Invalid 'force_limit'" in e for e in result["errors"]) diff --git a/test/test_rest_api_unit.py b/test/test_rest_api_unit.py index c5b47a7..6eaf0f3 100644 --- a/test/test_rest_api_unit.py +++ b/test/test_rest_api_unit.py @@ -74,3 +74,31 @@ def test_verify_sql_error(self, config): "fixed": "SELECT id FROM orders WHERE id = 123", "risk": 0.3, } + + def test_verify_sql_column_masking(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "credit_card"], + "column_masks": [ + {"column": "credit_card", "policy": "partial", "show_last": 4} + ], + } + ] + } + result = app.test_client().post( + "/verify-sql", + json={ + "sql": "SELECT id, credit_card FROM users", + "config": config, + "dialect": "postgres", + }, + ) + assert result.status_code == 200 + assert result.json == { + "allowed": False, + "errors": ["Column credit_card is masked (partial)"], + "fixed": "SELECT id, CONCAT('****', SUBSTRING(CAST(credit_card AS VARCHAR) FROM -4)) AS credit_card FROM users", + "risk": 0.2, + } diff --git a/test/test_security_fixes_unit.py b/test/test_security_fixes_unit.py new file mode 100644 index 0000000..d2235b4 --- /dev/null +++ b/test/test_security_fixes_unit.py @@ -0,0 +1,395 @@ +""" +Regression tests for the security / robustness fixes tracked as C1-C4, S1-S7, Q1-Q4. + +Each test pins the corrected behavior for a previously validated bug so future +changes cannot silently reintroduce it. See memory-bank/progress.md for the catalogue. +""" + +import pytest + +from sql_data_guard import verify_sql +from sql_data_guard.restriction_validation import ( + validate_restrictions, + UnsupportedRestrictionError, +) + + +def _orders_config() -> dict: + return { + "tables": [ + { + "table_name": "orders", + "columns": ["id", "product_name", "account_id"], + "restrictions": [{"column": "account_id", "value": 123}], + } + ] + } + + +# --------------------------------------------------------------------------- # +# S7 - explicit multi-statement (stacked query) guard +# --------------------------------------------------------------------------- # +class TestS7MultiStatement: + def test_stacked_drop_is_rejected(self): + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123; DROP TABLE orders", + _orders_config(), + ) + assert result["allowed"] is False + assert ( + "Stacked query detected: multiple statements are not allowed" + in result["errors"] + ) + assert result["fixed"] is None + + def test_stacked_select_is_rejected(self): + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123; SELECT * FROM secret", + _orders_config(), + ) + assert result["allowed"] is False + assert ( + "Stacked query detected: multiple statements are not allowed" + in result["errors"] + ) + + def test_single_statement_with_trailing_semicolon_is_allowed(self): + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123;", _orders_config() + ) + assert result["allowed"] is True + assert result["errors"] == [] + + def test_single_statement_with_comment_tail_is_allowed(self): + # The "-- ..." tail is an inert comment, not a second statement. + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123 -- ; DROP TABLE orders", + _orders_config(), + ) + assert result["allowed"] is True + + +# --------------------------------------------------------------------------- # +# S1 - generated restriction values must be safely escaped / quoted +# --------------------------------------------------------------------------- # +class TestS1ValueEscaping: + def test_scalar_string_with_single_quote_is_escaped(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "name"], + "restrictions": [{"column": "name", "value": "O'Brien"}], + } + ] + } + result = verify_sql("SELECT id FROM users", config) + # No parse error / crash, and the embedded quote is doubled (escaped). + assert result["fixed"] == "SELECT id FROM users WHERE name = 'O''Brien'" + + def test_in_string_values_are_quoted(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "role"], + "restrictions": [ + { + "column": "role", + "operation": "IN", + "values": ["admin", "user", "guest"], + } + ], + } + ] + } + result = verify_sql("SELECT id FROM users", config) + assert result["fixed"] == ( + "SELECT id FROM users WHERE role IN ('admin', 'user', 'guest')" + ) + + +# --------------------------------------------------------------------------- # +# S2 - numeric comparison (not string comparison) for < > <= >= +# --------------------------------------------------------------------------- # +class TestS2NumericComparison: + def _age_config(self) -> dict: + return { + "tables": [ + { + "table_name": "users", + "columns": ["age"], + "restrictions": [{"column": "age", "operation": "<", "value": 18}], + } + ] + } + + def test_more_restrictive_query_satisfies_restriction(self): + # age < 9 is within age < 18; "9" < "18" lexicographically was the bug. + result = verify_sql("SELECT age FROM users WHERE age < 9", self._age_config()) + assert result["allowed"] is True + assert result["errors"] == [] + + def test_less_restrictive_query_injects_correct_operator(self): + result = verify_sql("SELECT age FROM users WHERE age < 99", self._age_config()) + assert result["allowed"] is False + # Injected condition uses the restriction's operator (<), not "=". + assert result["fixed"] == "SELECT age FROM users WHERE (age < 99) AND age < 18" + + +# --------------------------------------------------------------------------- # +# S3 - dynamic-table column allow-list bypass +# --------------------------------------------------------------------------- # +class TestS3DynamicTableColumns: + def _users_config(self) -> dict: + return {"tables": [{"table_name": "users", "columns": ["id", "public_data"]}]} + + def test_unauthorized_column_via_subquery_is_blocked(self): + result = verify_sql( + "SELECT secret_col FROM (SELECT * FROM users) AS sub", self._users_config() + ) + assert result["allowed"] is False + assert ( + "Column secret_col is not allowed. Column removed from SELECT clause" + in result["errors"] + ) + # The disallowed column must NOT survive into a fixed query. + assert result["fixed"] is None or "secret_col" not in result["fixed"] + + def test_authorized_column_via_subquery_is_allowed(self): + result = verify_sql( + "SELECT id FROM (SELECT * FROM users) AS sub", self._users_config() + ) + # Inner SELECT * is flagged, but the legitimate outer column survives. + assert result["fixed"] == "SELECT id FROM (SELECT id, public_data FROM users) AS sub" + + def test_unaliased_subquery_column_still_allowed(self): + config = _orders_config() + result = verify_sql("SELECT id FROM (SELECT id FROM orders)", config) + assert "Column id is not allowed. Column removed from SELECT clause" not in ( + result["errors"] + ) + + +# --------------------------------------------------------------------------- # +# S4 - EXCEPT / INTERSECT set operations are verified like UNION +# --------------------------------------------------------------------------- # +class TestS4SetOperations: + def test_valid_except_is_allowed(self): + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123 " + "EXCEPT SELECT id FROM orders WHERE account_id = 123", + _orders_config(), + ) + assert result["allowed"] is True, result + + def test_valid_intersect_is_allowed(self): + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123 " + "INTERSECT SELECT id FROM orders WHERE account_id = 123", + _orders_config(), + ) + assert result["allowed"] is True, result + + def test_except_with_disallowed_column_is_blocked(self): + result = verify_sql( + "SELECT id FROM orders WHERE account_id = 123 " + "EXCEPT SELECT secret FROM orders WHERE account_id = 123", + _orders_config(), + ) + assert result["allowed"] is False + assert any("secret" in e for e in result["errors"]) + + +# --------------------------------------------------------------------------- # +# S5 - opt-in max_risk hard-block +# --------------------------------------------------------------------------- # +class TestS5MaxRisk: + def test_query_above_threshold_is_hard_blocked(self): + config = _orders_config() + config["max_risk"] = 0.2 + result = verify_sql("SELECT id FROM orders WHERE 1 = 1", config) + assert result["allowed"] is False + assert result["fixed"] is None + assert result["risk"] > 0.2 + + def test_query_below_threshold_is_still_fixed(self): + config = _orders_config() + config["max_risk"] = 0.9 + result = verify_sql("SELECT id FROM orders WHERE 1 = 1", config) + assert result["fixed"] is not None + + def test_no_threshold_preserves_default_behavior(self): + result = verify_sql("SELECT id FROM orders WHERE 1 = 1", _orders_config()) + assert result["fixed"] is not None + + +# --------------------------------------------------------------------------- # +# C1-C4 - config validation robustness (no caller-facing crashes) +# --------------------------------------------------------------------------- # +class TestConfigValidation: + def test_in_accepts_many_values(self): + config = { + "tables": [ + { + "table_name": "orders", + "columns": ["id", "account_id"], + "restrictions": [ + {"column": "account_id", "operation": "IN", "values": [1, 2, 3]} + ], + } + ] + } + # Must not raise; previously crashed with len(values) == 2 requirement. + validate_restrictions(config) + + def test_in_accepts_string_values(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "role"], + "restrictions": [ + { + "column": "role", + "operation": "IN", + "values": ["admin", "user", "guest"], + } + ], + } + ] + } + validate_restrictions(config) + + def test_in_rejects_mixed_types_without_crashing_verify_sql(self): + config = { + "tables": [ + { + "table_name": "users", + "columns": ["id", "role"], + "restrictions": [ + {"column": "role", "operation": "IN", "values": [1, "a"]} + ], + } + ] + } + # C4: verify_sql converts the ValueError into a result dict, not a crash. + result = verify_sql("SELECT id FROM users", config) + assert result["allowed"] is False + assert any("IN" in e for e in result["errors"]) + + def test_missing_value_for_scalar_op_is_clean_error(self): + config = { + "tables": [ + { + "table_name": "orders", + "columns": ["id", "status"], + "restrictions": [{"column": "status", "operation": "="}], + } + ] + } + result = verify_sql("SELECT id FROM orders", config) + assert result["allowed"] is False + assert result["risk"] == 1.0 + + def test_lowercase_operations_are_accepted(self): + config = { + "tables": [ + { + "table_name": "orders", + "columns": ["id", "account_id"], + "restrictions": [ + {"column": "account_id", "operation": "between", "values": [1, 5]} + ], + } + ] + } + result = verify_sql( + "SELECT id FROM orders WHERE account_id BETWEEN 1 AND 5", config + ) + assert result["allowed"] is True, result + + def test_unsupported_operation_still_rejected(self): + config = { + "tables": [ + { + "table_name": "products", + "columns": ["price"], + "restrictions": [ + {"column": "price", "value": 100, "operation": "NotSupported"} + ], + } + ] + } + with pytest.raises(UnsupportedRestrictionError): + validate_restrictions(config) + + +# --------------------------------------------------------------------------- # +# Q1 - errors are an ordered, de-duplicated list +# --------------------------------------------------------------------------- # +class TestQ1ErrorsList: + def test_errors_is_a_list(self): + result = verify_sql("SELECT id FROM orders WHERE account_id = 123", _orders_config()) + assert isinstance(result["errors"], list) + + def test_errors_are_deduplicated(self): + result = verify_sql("SELECT secret FROM (SELECT * FROM users) AS s", { + "tables": [{"table_name": "users", "columns": ["id"]}] + }) + assert len(result["errors"]) == len(set(result["errors"])) + + +# --------------------------------------------------------------------------- # +# Q3 - the MCP wrapper module is import-safe +# --------------------------------------------------------------------------- # +class TestQ3McpImport: + def test_mcp_wrapper_is_importable(self): + # Previously raised NameError because module-level globals were only + # defined inside the __main__ block. (Requires the optional docker dep.) + pytest.importorskip("docker") + from sql_data_guard.mcpwrapper import mcp_wrapper + + assert isinstance(mcp_wrapper.errors, dict) + assert isinstance(mcp_wrapper.config, dict) + + +# --------------------------------------------------------------------------- # +# S6 - opt-in REST API key authentication +# --------------------------------------------------------------------------- # +class TestS6ApiKey: + def _request(self): + from sql_data_guard.rest import sql_data_guard_rest as rest_mod + + return rest_mod + + def test_no_key_configured_allows_request(self, monkeypatch): + rest_mod = self._request() + monkeypatch.setattr(rest_mod, "_API_KEY", None) + result = rest_mod.app.test_client().post( + "/verify-sql", + json={"sql": "SELECT id FROM orders WHERE id = 123", "config": { + "tables": [{"table_name": "orders", "columns": ["id"], + "restrictions": [{"column": "id", "value": 123}]}] + }}, + ) + assert result.status_code == 200 + + def test_missing_key_is_unauthorized(self, monkeypatch): + rest_mod = self._request() + monkeypatch.setattr(rest_mod, "_API_KEY", "secret") + result = rest_mod.app.test_client().post("/verify-sql", json={}) + assert result.status_code == 401 + + def test_correct_key_is_authorized(self, monkeypatch): + rest_mod = self._request() + monkeypatch.setattr(rest_mod, "_API_KEY", "secret") + result = rest_mod.app.test_client().post( + "/verify-sql", + json={"sql": "SELECT id FROM orders WHERE id = 123", "config": { + "tables": [{"table_name": "orders", "columns": ["id"], + "restrictions": [{"column": "id", "value": 123}]}] + }}, + headers={"X-API-Key": "secret"}, + ) + assert result.status_code == 200 diff --git a/test/test_sql_guard_curr_unit.py b/test/test_sql_guard_curr_unit.py index ea7119c..fbe5e5d 100644 --- a/test/test_sql_guard_curr_unit.py +++ b/test/test_sql_guard_curr_unit.py @@ -220,7 +220,7 @@ def test_inner_join_with_multiple_conditions(self, config): """ res = verify_sql(sql_query, config) assert res["allowed"] is True, res - assert res["errors"] == set(), res + assert not res["errors"], res def test_union_with_invalid_column(self, config): sql_query = """ @@ -240,7 +240,7 @@ def test_right_join_with_no_matching_prod_id(self, config): """ res = verify_sql(sql_query, config) assert res["allowed"] is True, res - assert res["errors"] == set(), res + assert not res["errors"], res class TestSQLJsonArrayQueries: