✨ feat(stress): add distributed stress testing infrastructure#306
Closed
✨ feat(stress): add distributed stress testing infrastructure#306
Conversation
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #306 +/- ##
==========================================
+ Coverage 96.96% 97.09% +0.13%
==========================================
Files 24 25 +1
Lines 5099 5343 +244
==========================================
+ Hits 4944 5188 +244
Misses 155 155
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: 01539fa | Previous: 38a0e68 | Ratio |
|---|---|---|---|
tests/benchmark/test_latency.py::TestLatencyComparison::test_two_limits |
235.00870426785562 iter/sec (stddev: 0.00850320509097727) |
288.3994402811384 iter/sec (stddev: 0.00014876579195594755) |
1.23 |
tests/benchmark/test_operations.py::TestTransactionOverheadBenchmarks::test_transactional_acquire |
299.2736512902029 iter/sec (stddev: 0.0082067744513638) |
374.0754667581391 iter/sec (stddev: 0.0005332760263343914) |
1.25 |
tests/benchmark/test_operations.py::TestConfigLookupBenchmarks::test_acquire_cold_config |
227.09609023350396 iter/sec (stddev: 0.008100116269139448) |
283.85108120986445 iter/sec (stddev: 0.00035752013219356377) |
1.25 |
This comment was automatically generated by workflow using github-action-benchmark.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
sodre
force-pushed
the
perf/stress-test
branch
from
3b9613a to
9e8c017
Compare
sodre
force-pushed
the
perf/stress-test
branch
from
5c8aca4 to
0dcf949
Compare
sodre
force-pushed
the
perf/stress-test
branch
2 times, most recently
from
4053f02 to
c515738
Compare
These outputs allow the stress test stack to import IAM configuration from the target stack, ensuring consistent permission boundaries and providing an S3 bucket for large Lambda packages. Refs: discussion#7 Co-Authored-By: Claude Opus 4.5 <[email protected]>
Configuration dataclasses for stress testing with: - Whale entity (50% traffic) - Spike entity (3% baseline + 1500 RPM bursts) - Power law distribution (47% across remaining entities) Refs: discussion#7 Co-Authored-By: Claude Opus 4.5 <[email protected]>
Implements traffic distribution: - Whale entity receives ~50% of traffic - Spike entity has baseline + periodic bursts - Power law (Zipf) distribution for remaining entities Refs: discussion#7 Co-Authored-By: Claude Opus 4.5 <[email protected]>
…cture Creates: - ECS cluster with Fargate Spot capacity provider - Locust master task definition and service (desired=0) - Lambda functions with placeholder code - VPC endpoints for SSM (ECS Exec) - Security groups for master and workers - IAM roles with optional permission boundary Refs: discussion#7 Co-Authored-By: Claude Opus 4.5 <[email protected]>
Builds Locust master image with zae-limiter: - Auto-detects dev mode (builds wheel) vs installed (uses PyPI version) - Creates build context with Dockerfile and locustfile - Pushes to ECR repository Refs: discussion#7 Co-Authored-By: Claude Opus 4.5 <[email protected]>
Builds Lambda deployment zip with: - locust and gevent dependencies - zae-limiter (from wheel or PyPI) - stress_lambda handler code - locustfile and distribution modules Refs: discussion#7 Co-Authored-By: Claude Opus 4.5 <[email protected]>
Worker handler: - Headless mode: self-contained test, returns stats - Worker mode: connects to Fargate master Setup handler: - Creates whale and spike entities - Configures system and resource defaults - Creates entities with custom limits Refs: discussion#7 Co-Authored-By: Claude Opus 4.5 <[email protected]>
Implements RateLimiterUser with: - acquire_tokens task (100 weight) - primary workload - check_available task (5 weight) - read-only checks - Shared SyncRateLimiter across users - Traffic distribution via TrafficDistributor - Proper Locust event firing for stats Refs: discussion#7 Co-Authored-By: Claude Opus 4.5 <[email protected]>
Commands: - stress deploy: Deploy Fargate + Lambda infrastructure - stress setup: Create test entities and configure limits - stress connect: Print SSM port-forward command - stress teardown: Delete stress test stack Refs: discussion#7 Co-Authored-By: Claude Opus 4.5 <[email protected]>
Adds [stress] extra with: - locust>=2.20 - gevent>=23.0 Install with: pip install 'zae-limiter[stress]' Refs: discussion#7 Co-Authored-By: Claude Opus 4.5 <[email protected]>
The stress module contains code that requires AWS/Docker/Locust runtime to test (Lambda handlers, CLI commands, locustfile). Excluding from coverage measurement via [tool.coverage.run] omit. Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Use nest_asyncio to allow nested event loops with gevent - Fix stats collection: initialize env.stats before starting runner - Use self.environment.events instead of global events for request firing - Change from SyncRateLimiter to async RateLimiter with _run_async wrapper - Use gevent.sleep() instead of greenlet.join() for proper greenlet execution - Add RoleNameFormat parameter to CFN templates for IAM role naming - Change Lambda timeout from 900s to 60s for faster iteration - Use AdminPolicyArn for Lambda worker role (TransactWriteItems needs underlying permissions) - Add nest-asyncio dependency to Lambda package Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Remove per-user startup logging - Replace per-entry stats with single summary line Co-Authored-By: Claude Opus 4.5 <[email protected]>
…er detection Add --benchmark-disable to addopts so benchmark warnings don't appear during regular test runs. CI benchmark steps now pass --benchmark-enable explicitly. Fix pytest_ignore_collect to also detect xdist worker processes via hasattr(config, 'workerinput'), since workers don't have numprocesses set and were still importing gevent files during collection. Co-Authored-By: Claude Opus 4.6 <[email protected]>
Cover the two missing lines flagged by diff-cover: - locust.py:59 — _configure_boto3_pool early return on second call - discovery.py:118 — list_limiters stack_type filter branch Co-Authored-By: Claude Opus 4.6 <[email protected]>
Update test expectations across load test suite to match recent changes: - TestScalingConfig: users_per_worker=10, remove rps_per_worker - TestCalculateDesiredWorkers: new scaling math without rps_per_worker - TestMain: update worker counts, remove DESIRED_WORKERS/RPS_PER_WORKER env - TestGenerateDockerfile: LOCUSTFILE set at runtime, not baked in - TestDeployCommand: mock AcquireOnlyPolicyArn/FullAccessPolicyArn outputs - TestConnectCommand: add required -f locustfile to all 16 invocations - TestBuildTaskOverrides: remove desired_workers/rps_per_worker params - TestSetupCommand: remove test for nonexistent setup subcommand - Document gevent marker and two-step unit test workflow Co-Authored-By: Claude Opus 4.6 <[email protected]>
…oint reuse Add --policy-name-format to the required IAM flags alongside --permission-boundary and --role-name-format. PowerUserAccess requires all three for both iam:CreateRole and iam:CreatePolicy. Document common mistakes (missing policy format, using --no-iam) and add copy-paste deploy/benchmark commands to aws-testing.md. Update CLAUDE.md load testing section with step 0 (deploy limiter stack first). Fix load deploy to skip DynamoDB VPC endpoint creation when one already exists, preventing CloudFormation AlreadyExists errors on redeployment. Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add --capacity-provider option to `load deploy` (FARGATE_SPOT default, FARGATE for on-demand). FARGATE_SPOT is only registered on the ECS cluster when selected, so partitions without Spot support (GovCloud, China) work correctly with --capacity-provider FARGATE. Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Discard unused return values from client.images.build() (builder.py) - Log warning on teardown scale-down failure instead of bare pass (cli.py) - Refactor _boto3_pool_configured global to function attribute pattern in both worker.py and locust.py to satisfy CodeQL analysis - Use importlib.import_module() to avoid mixed import styles in tests Co-Authored-By: Claude Opus 4.6 <[email protected]>
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add MaxRpsCascadeUser and SimpleCascadeUser to locustfiles for measuring cascade overhead. Add --user-classes CLI option to benchmark command to select specific User classes without needing --class-picker UI. Bump system default limits to 1B RPM to avoid false rate limiting during throughput benchmarks. Co-Authored-By: Claude Opus 4.6 <[email protected]>
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Replace hardcoded ThreadPoolExecutor with configurable parallel_mode
parameter ("auto", "gevent", "threadpool", "serial") on SyncRepository.
ThreadPoolExecutor caused 58% cascade throughput regression under
concurrent load on Lambda (1 vCPU, GIL contention).
Auto mode silently picks the best strategy: gevent (if monkey-patched),
serial (single-CPU), or threadpool (multi-CPU). Explicit modes warn on
suboptimal conditions. Resolution happens once at __init__ time.
ThreadPoolExecutor is created lazily on first cascade request.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
…AWS cascade tests Add speculative_writes=False baseline to TestAWSCascadeSpeculativeComparison for side-by-side comparison with speculative path. Parametrize test_cascade_concurrent_throughput_aws by parallel_mode (serial/gevent/threadpool) to measure how internal cascade parallelism interacts with external thread contention. Enable gevent in CI benchmark-aws job (--extra stress + GEVENT=1). Co-Authored-By: Claude Opus 4.6 <[email protected]>
Rename `load benchmark` → `load run` and add `load calibrate` for binary-search calibration of optimal per-worker user count using Little's Law efficiency metric (baseline_p50 / observed_p50). - Extract `_invoke_lambda_headless()` and `_get_lambda_client_and_config()` helpers - Add `_calibrate_lambda()` with binary search and `_display_calibration_results()` - Auto-extend baseline duration when requests < 100, extrapolating from observed rate (e.g. 28 reqs in 15s → retry at 60s) instead of doubling - Cap baseline retry at Lambda timeout - 30s headroom - Report request count per step in calibration output - Update locustfile docstrings with calibration results: MaxRpsUser: 3 users/worker optimal (p50=7ms, 371 RPS, 90% efficiency) SimpleUser: 40+ users/worker (no GIL pressure due to wait time) - Fix OOM in cascade locustfiles: use shared `self.client._limiter` instead of creating throwaway SyncRateLimiter per user Co-Authored-By: Claude Opus 4.6 <[email protected]>
sodre
force-pushed
the
perf/stress-test
branch
from
9bab8fe to
c7600aa
Compare
Replace plain midpoint bisection with linear interpolation of efficiency values. The midpoint is weighted toward where the threshold is likely to fall, converging faster when efficiency drops steeply (e.g. MaxRpsCascadeUser: 100% at 1 user, 8% at 40). Co-Authored-By: Claude Opus 4.6 <[email protected]>
…rallel_mode Add logger.debug() message when gevent ImportError is caught during auto mode resolution, replacing the bare pass that was flagged in PR review. The debug log survives AST round-tripping (comments don't). Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Use baseline duration (possibly auto-extended) for all search steps, ensuring fair p50 comparison across rows - Replace Reqs column with Reqs/s in calibration table (RPS already shown, raw request counts were misleading across different durations) - Add --extra stress to CI unit test job so questionary is available Co-Authored-By: Claude Opus 4.6 <[email protected]>
Now that all steps use the same duration, raw request counts are comparable across rows. Show both columns for completeness. Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Use RPS column header instead of Reqs/s in calibration table - Change --baseline-duration default from 15s to 60s for stable percentiles without needing auto-retry Co-Authored-By: Claude Opus 4.6 <[email protected]>
…ean up deps - Add p95 and p99 latency columns to calibration output (step logs, table, summary) - Fix weighted bisection searching wrong direction (toward high instead of low) - Move questionary from [stress] extra to core dependencies - Remove unused asyncio-gevent from [stress] extra - Add docker to [stress] extra - Add types-gevent to dev dependencies and pre-commit mypy env - Add questionary to pre-commit mypy env - Remove gevent/questionary from mypy ignore_missing_imports overrides - Revert --extra stress workaround in CI (questionary now core dep) Co-Authored-By: Claude Opus 4.6 <[email protected]>
Co-Authored-By: Claude Opus 4.6 <[email protected]>
The gevent tests need gevent which moved from implicit availability to the [bench] extra (renamed from [stress]). Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Add questionary to required deps in CLAUDE.md - Add [local] and [bench] extras to CLAUDE.md - Add types-gevent to [dev] description - Fix --baseline-duration default from 15 to 60 in load-testing.md Co-Authored-By: Claude Opus 4.6 <[email protected]>
Code ReviewI found 4 issues during review: 1. 🐛 CloudFormation deployment will fail with empty RoleNameFormatFile: The load CLI uses: role_name_format = outputs.get("RoleNameFormat", "{}")When the main limiter stack is deployed without This empty string is passed to the load stack's CloudFormation template, which uses: Prefix: \!Select [0, \!Split ['{}', \!Ref RoleNameFormat]]
Suffix: \!Select [1, \!Split ['{}', \!Ref RoleNameFormat]]
Fix: Change line 391 to: role_name_format = outputs.get("RoleNameFormat") or "{}"This treats both absent keys and empty strings as falsy values, using See: zae-limiter/src/zae_limiter/load/cli.py Lines 390 to 392 in 42438bd 2. 📝 CLAUDE.md Project Structure missing
|
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
load deploy,load connect,load run,load calibrate,load teardown,load listRateLimiterUserandRateLimiterSessionfor Locust integration with instrumented eventsload calibratecommand: binary-search for optimal per-worker concurrency using Little's Law efficiency metricCalibration Results
Lambda calibration (90% efficiency threshold, us-east-1, 1 vCPU Lambda, 60s per step):
max_rps.py — MaxRpsUser (zero wait, max throughput)
Optimal: 3 users/lambda — p50 stays at floor (8ms), ~364 RPS per worker.
max_rps.py — MaxRpsCascadeUser (zero wait, cascade overhead)
Optimal: 2 users/lambda — p50 stays near floor (10ms), ~176 RPS per worker. Cascade adds ~2ms floor latency vs non-cascade.
simple.py — SimpleUser (wait 0.1–1.0s, realistic traffic)
Optimal: 40+ users/lambda — p50 stays at floor (10ms), wait time dominates. No GIL pressure at any concurrency.
simple.py — SimpleCascadeUser (wait 0.1–1.0s, cascade)
Optimal: 40+ users/lambda — p50 stays at floor (9ms), wait time dominates. No GIL pressure at any concurrency.
Test plan
🤖 Generated with Claude Code