feat: complete AI analyzer + foundation fixes + test suite#1
Merged
Conversation
…re method, fix getRun handler - SaveSource: real tar.gz/zip extraction with Dockerfile validation and path traversal protection - GetRun: new store method querying Postgres with pgx.ErrNoRows handling - getRun handler: queries Postgres instead of returning hardcoded placeholder - go.sum: generated for reproducible builds - .gitignore: exclude compiled sample-engine binary
- Add redis/go-redis dependency to telemetry service - Initialize Redis client on boot (graceful degradation if unavailable) - ZADD leaderboard:scores with GT flag (only update if new score is higher) - HSET leaderboard:metrics for per-team metric detail display - Fix ctx variable ordering bug (was used before declaration) - Add REDIS_URL env var to telemetry service in docker-compose.yml - go.sum: generated for reproducible builds
New service: services/ai-analyzer/ (Go, Gemini API) Three specialized agents run concurrently: - Security: container escape vectors, memory safety, input validation - Performance: O(n) scans, lock contention, allocation patterns - Correctness: price-time priority, fill semantics, overflow protection Components: - internal/gemini/client.go: raw HTTP Gemini API client (no SDK dependency) - internal/agents/: security, performance, correctness agents + synthesizer - internal/report/: post-run performance report generator - cmd/main.go: HTTP API (POST /api/analyze, POST /api/report, GET /api/health) Features: - Concurrent agent execution with graceful error handling - Risk score computation (0-100) from severity-weighted findings - Structured JSON output via Gemini response_mime_type - Multipart file upload and JSON body support - 100KB source code cap to prevent token overflow - CORS enabled, graceful shutdown, 90s timeout for LLM calls
- analyze.html: full AI analysis UI with risk gauge, agent status cards, severity-colored findings, category filter, recommendation panel - layout.js: add AI Analysis nav item to sidebar navigation - Caddyfile: route /api/analyze and /api/report to ai-analyzer:7080
- sql/init.sql: add analysis_reports table for persisting AI review results - .env.example: add GEMINI_API_KEY and GEMINI_MODEL documentation - README.md: add AI analyzer to service table, repo layout, test coverage, AI setup docs - DESIGN.md: system design document for hackathon submission - services/botfleet/go.sum: generated for reproducible builds
Test suite in tests/ (standalone Go module, no external deps): sandbox_test.go (6 tests): - tar.gz/zip extraction, Dockerfile validation - Path traversal protection, nested dirs, magic bytes scoring_test.go (13 tests): - mathExpDecay and mathSat edge cases - Composite score: perfect, zero-tps, all-errors, realistic, high-latency agent_test.go (8 tests): - Risk score computation and capping - Recommendation dedup, strength detection, report breakdown All pass with -race flag.
There was a problem hiding this comment.
Pull request overview
This PR expands the platform with an AI-powered analyzer service, strengthens the submission sandbox upload path (archive extraction + Dockerfile validation), and upgrades telemetry to stream live metrics and maintain a Redis-backed leaderboard, alongside a new standalone unit test suite.
Changes:
- Added
ai-analyzerGo service (Gemini-backed multi-agent analysis + post-run performance report) and wired it into Caddy + docker-compose. - Implemented robust archive extraction in gateway sandbox (
tar.gz/tar/zip) with traversal protection + per-file size caps, plus addedGET /api/runs/{id}DB-backed handler. - Enhanced telemetry service with optional Redis connectivity for live run streaming and leaderboard updates; added standalone Go tests module covering sandbox extraction, scoring math, and agent synthesis behavior.
Reviewed changes
Copilot reviewed 30 out of 34 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
tests/scoring_test.go |
Adds unit tests for composite scoring math helpers. |
tests/sandbox_test.go |
Adds unit tests for archive extraction behavior and traversal protection. |
tests/go.mod |
Introduces a standalone Go module for the tests suite. |
tests/agent_test.go |
Adds unit tests for agent risk scoring, recommendation dedup, and strengths logic. |
sql/init.sql |
Adds analysis_reports table + seeds a demo submission; adjusts telemetry rollup view fields. |
services/telemetry/go.sum |
Adds dependency checksums for Redis client and transitive deps. |
services/telemetry/go.mod |
Adds Redis dependency for leaderboard/live updates. |
services/telemetry/cmd/main.go |
Adds Redis integration, live per-run publishing, and Redis leaderboard writes during finalize. |
services/gateway/internal/store/store.go |
Adds GetRun DB accessor for GET /api/runs/{id}. |
services/gateway/internal/sandbox/sandbox.go |
Implements real archive extraction + Dockerfile validation in SaveSource. |
services/gateway/internal/api/handlers.go |
Implements GET /api/runs/{id} returning DB run metadata. |
services/gateway/go.sum |
Updates dependency checksums (Redis + transitive). |
services/gateway/go.mod |
Removes unused Docker SDK dep; adds/normalizes indirect deps. |
services/botfleet/go.sum |
Adds checksums for new indirect deps. |
services/botfleet/go.mod |
Adds indirect deps required by botfleet build. |
services/ai-analyzer/internal/report/generator.go |
Adds post-run performance report generator using telemetry metrics + source. |
services/ai-analyzer/internal/gemini/client.go |
Adds a minimal raw HTTP Gemini client with JSON parsing helper. |
services/ai-analyzer/internal/agents/types.go |
Defines shared agent/report types and JSON schema. |
services/ai-analyzer/internal/agents/synthesizer.go |
Adds concurrent agent runner + synthesis (risk score, recommendations, summary). |
services/ai-analyzer/internal/agents/security.go |
Adds security agent prompt + JSON parsing wrapper. |
services/ai-analyzer/internal/agents/performance.go |
Adds performance agent prompt + JSON parsing wrapper. |
services/ai-analyzer/internal/agents/correctness.go |
Adds correctness agent prompt + JSON parsing wrapper. |
services/ai-analyzer/go.mod |
Introduces ai-analyzer Go module. |
services/ai-analyzer/Dockerfile |
Adds multi-stage Docker build for ai-analyzer service. |
services/ai-analyzer/cmd/main.go |
Implements ai-analyzer HTTP API, middleware, and graceful shutdown. |
README.md |
Documents new service, routes, and test suite usage. |
frontend/platform/assets/layout.js |
Adds navigation entry for the new AI Analysis page. |
frontend/platform/analyze.html |
Adds AI analysis UI page rendering findings, strengths, recommendations, risk gauge. |
examples/sample-engine-go/main.go |
Fixes price scaling and improves duplicate ID dedup to include clientId. |
docker-compose.yml |
Wires Redis into telemetry and adds the ai-analyzer service; pins network name. |
DESIGN.md |
Adds/updates end-to-end design documentation including AI analyzer and pipeline details. |
Caddyfile |
Routes /api/analyze and /api/report to ai-analyzer; keeps other /api/* to gateway. |
.gitignore |
Ignores the sample-engine binary artifact. |
.env.example |
Adds Gemini configuration keys and clarifies bot scaling wording. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+136
to
+139
| INSERT INTO submissions (id, team_id, name, lang, hash, image_tag, endpoint, status, size_bytes) | ||
| VALUES ('sub_sample', 't_demo', 'sample-engine', 'go', 'seed-sample', | ||
| 'iicpc/sample-engine:seed', 'http://sample-engine:9001', 'deployed', 0) | ||
| ON CONFLICT (team_id, hash) DO NOTHING; |
Comment on lines
+135
to
+137
| // Generate summary | ||
| summary := generateSummary(len(allFindings), riskScore, len(secFindings), len(perfFindings), len(corrFindings)) | ||
|
|
Comment on lines
+212
to
+215
| // CORS headers | ||
| w.Header().Set("Access-Control-Allow-Origin", "*") | ||
| w.Header().Set("Access-Control-Allow-Methods", "GET, POST, OPTIONS") | ||
| w.Header().Set("Access-Control-Allow-Headers", "Content-Type") |
Comment on lines
+5
to
+9
| Multi-agent AI pipeline: Security + Performance + Correctness. | ||
| Calls the ai-analyzer service (POST /api/analyze) with source code | ||
| and displays structured findings with risk score and recommendations. | ||
| Falls back to a local mock if no backend is available. | ||
| ===================================================================== --> |
Comment on lines
+1
to
+4
| // Unit tests for the composite scoring functions. | ||
| // Verifies mathExpDecay and mathSat produce correct scores. | ||
| // Run with: go test -v ./tests/ -run TestScoring | ||
| // No external dependencies required. |
Comment on lines
+294
to
+297
| clean := filepath.Clean(f.Name) | ||
| if strings.Contains(clean, "..") || filepath.IsAbs(clean) { | ||
| continue | ||
| } |
Comment on lines
+91
to
+94
| // Sort findings by severity (critical first) | ||
| sort.Slice(allFindings, func(i, j int) bool { | ||
| return severityWeight[allFindings[i].Severity] > severityWeight[allFindings[j].Severity] | ||
| }) |
Comment on lines
+96
to
+100
| // Compute risk score: sum of severity weights, capped at 100 | ||
| riskScore := 0 | ||
| for _, f := range allFindings { | ||
| riskScore += severityWeight[f.Severity] | ||
| } |
Comment on lines
+66
to
+67
| created_at TIMESTAMPTZ NOT NULL DEFAULT now() | ||
| ); |
Comment on lines
+348
to
352
| percentile_cont(0.50) WITHIN GROUP (ORDER BY latency_ns) AS p50, | ||
| percentile_cont(0.90) WITHIN GROUP (ORDER BY latency_ns) AS p90, | ||
| percentile_cont(0.99) WITHIN GROUP (ORDER BY latency_ns) AS p99, | ||
| count(*)::float / NULLIF(EXTRACT(EPOCH FROM (max(ts) - min(ts))),0) AS tps, | ||
| sum(CASE WHEN err IS NOT NULL THEN 1 ELSE 0 END)::float / NULLIF(count(*),0) AS err_rate |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
6 commits:
All 4 Go services compile and pass go vet. 27/27 tests pass with -race.