Add content-api benchmark project with multi-style API specs - #3
Conversation
Establish the canonical project structure under projects/ for benchmark targets, starting with content-api. Shared API contracts and k6 load test scripts are provided for three API styles: OpenAPI (REST), GraphQL, and Protobuf (gRPC). An interactive Claude command is added to scaffold new implementations against these specs. Also fixes result filename generation to sanitize slashes in target names and removes the example-api placeholder from the config. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new content-api benchmark “shared contract” area (OpenAPI/GraphQL/Protobuf) plus k6 scripts to drive each style, and improves result file naming to support hierarchical target names (e.g., content-api/spring-boot).
Changes:
- Introduce
projects/content-api/_shared/with OpenAPI, GraphQL, and Protobuf API specs and corresponding k6 load-test scripts. - Add a
.claudeinteractive command to scaffold new benchmark implementations against these shared specs. - Sanitize
/in result filenames and remove the placeholderexample-apitarget frombenchmark.config.json.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
toolkit/src/report/json.js |
Sanitizes target name in output filenames to avoid path separators. |
projects/content-api/_shared/protobuf/k6/content-api.js |
Adds a gRPC-based k6 scenario for CRUD-style content operations. |
projects/content-api/_shared/protobuf/content.proto |
Defines the gRPC service + messages for the protobuf benchmark contract. |
projects/content-api/_shared/openapi/k6/content-api.js |
Adds a REST/OpenAPI-based k6 scenario for CRUD-style content operations. |
projects/content-api/_shared/openapi/api-spec.yaml |
Adds an OpenAPI 3.1 contract for the REST content API. |
projects/content-api/_shared/graphql/schema.graphql |
Adds a GraphQL schema for the content API benchmark contract. |
projects/content-api/_shared/graphql/k6/content-api.js |
Adds a GraphQL-based k6 scenario for CRUD-style content operations. |
benchmark.config.json |
Removes placeholder target; leaves an empty targets map with defaults/grafana/output intact. |
.claude/commands/scaffold-implementation.md |
Adds scaffolding guidance for generating implementations and registering targets. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| const timestamp = results.timestamp.replace(/[:.]/g, '-'); | ||
| const filename = `${results.target}-${timestamp}.json`; | ||
| const safeName = results.target.replace(/\//g, '-'); |
There was a problem hiding this comment.
safeName only replaces forward slashes. On Windows, a target name containing backslashes (or sequences like ..\) can be interpreted by path.join() as a path segment and potentially escape outputDir / create nested paths. Consider normalizing to a safe filename by stripping/replacing both / and \\ (and ideally other filesystem-invalid characters) or by using path.basename() on a sanitized string before joining.
| const safeName = results.target.replace(/\//g, '-'); | |
| const safeName = results.target.replace(/[\/\\]/g, '-'); |
|
|
||
| put: | ||
| operationId: updateContent | ||
| summary: Update a content item |
There was a problem hiding this comment.
The PUT /api/v1/content/{id} operation is defined as a partial update (per the k6 script and UpdateContentRequest), but HTTP PUT typically implies full replacement. To avoid ambiguity across implementations, consider switching this operation to PATCH or explicitly documenting that PUT performs a partial update.
| summary: Update a content item | |
| summary: Partially update a content item | |
| description: > | |
| Partially updates the specified content item using the fields provided | |
| in the request body. Any fields omitted from the request body are left | |
| unchanged on the existing resource. Although this operation uses HTTP | |
| PUT, it is intentionally defined as a partial update rather than a full | |
| replacement. |
| UpdateContentRequest: | ||
| type: object | ||
| properties: | ||
| title: | ||
| type: string | ||
| minLength: 1 | ||
| maxLength: 255 | ||
| body: | ||
| type: string | ||
| status: | ||
| type: string | ||
| enum: [draft, published, archived] |
There was a problem hiding this comment.
UpdateContentRequest allows an empty object, which makes it unclear how servers should respond to a no-op update (and can hide client bugs). Consider requiring at least one field (e.g., minProperties: 1 in OAS 3.1 / JSON Schema) or using an anyOf to enforce that title, body, or status is present.
| ### Dockerfile | ||
| - Use an appropriate base image for the language/framework | ||
| - Multi-stage build where applicable (build stage + runtime stage) | ||
| - The final image should be as small as practical | ||
| - Expose port 8080 | ||
|
|
There was a problem hiding this comment.
The Dockerfile guidance says to "Expose port 8080", but later the protobuf style section states the gRPC server should listen on port 50051 in addition to the HTTP health check on 8080. To keep the scaffold instructions consistent, update the Dockerfile/compose guidance to include exposing/mapping port 50051 for protobuf implementations (or clarify that EXPOSE is optional but the service must listen on both ports).
Summary
projects/content-api/_shared/with API specs and k6 load test scripts for three API styles: OpenAPI (REST), GraphQL, and Protobuf (gRPC).claude/commands/scaffold-implementation.md— an interactive Claude command that guides users through scaffolding new implementations against these specstoolkit/src/report/json.jsto sanitize slashes in target names for result filenames (supportscontent-api/spring-bootstyle target names)example-apitarget frombenchmark.config.jsonTest plan
npm run benchmark -- listparses config without errorsnpm run lintpasses/scaffold-implementationto scaffold a first implementation and verify the full benchmark pipeline🤖 Generated with Claude Code