Repository navigation
Build the Docker image in CI, and pin claude.yml to the stable channel #13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -208,3 +208,52 @@ jobs: | |
|
|
||
| - name: The ${{ matrix.engine }} CLI answers --help | ||
| run: .venv/bin/python ${{ matrix.engine }}_scraper.py --help >/dev/null | ||
|
|
||
| docker: | ||
| # Nothing else in this repo builds the image, which is how all three of | ||
| # the older repos in this family shipped one that died with | ||
| # ModuleNotFoundError on every invocation, `--help` included: the | ||
| # Dockerfile COPYs an explicit list (right -- the image should carry no | ||
| # test suite and no stray .env) and the list fell behind the imports. | ||
| # Four steps, each for a distinct failure. Under a minute. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "Under a minute" reads as a measured number but nothing here measures it, and the job has no docker layer cache or actions/cache for pip/playwright (no |
||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Build the image | ||
| # Plain `docker build`, no registry and no push: the point is that the | ||
| # Dockerfile still builds, not that anything is published. | ||
| run: docker build -t amazon-scraper:ci . | ||
|
|
||
| - name: The entrypoint runs inside the image | ||
| # Its CMD is --help, so a bare `run` exercises the default path. This | ||
| # is the invocation a missing module breaks. | ||
| run: docker run --rm amazon-scraper:ci | ||
|
|
||
| - name: Chromium is actually installed in the image | ||
| # `playwright install --with-deps` is the layer this job exists to | ||
| # prove, so check the browser is there rather than trusting that the | ||
| # layer exited 0. | ||
| run: | | ||
| docker run --rm --entrypoint python3 amazon-scraper:ci -c " | ||
| from playwright.sync_api import sync_playwright | ||
| with sync_playwright() as p: | ||
| b = p.chromium.launch() | ||
| print('chromium', b.version) | ||
| b.close()" | ||
|
|
||
| - name: The image carries no credentials and no test material | ||
| # It COPYs an explicit list precisely so that it does not. Verify | ||
| # rather than assume: a .env baked into an image is a credential | ||
| # published to everyone who can pull it. | ||
| run: | | ||
| set -e | ||
| found=0 | ||
| for f in .env smoke_test.py tests sample_output.json; do | ||
| if docker run --rm --entrypoint sh amazon-scraper:ci -c "test -e /app/$f"; then | ||
| echo "::error::$f is inside the image and should not be" | ||
| found=1 | ||
| fi | ||
| done | ||
| test "$found" = "0" | ||
| echo "no .env, no test suite, no fixtures in the image" | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,6 +35,7 @@ | |
| """ | ||
|
|
||
| import ast | ||
| import importlib | ||
| import inspect | ||
| import json | ||
| import os | ||
|
|
@@ -1403,6 +1404,116 @@ def test_wording(): | |
| return ok | ||
|
|
||
|
|
||
| def test_shared_calls_bind_against_the_real_signature(): | ||
| """§17's check #1: bind every call into a shared module against the | ||
| callee's real signature. | ||
|
|
||
| This exists because of a defect that no other check in this suite can | ||
| see. A sibling repo shipped `classify(html, url=…)` in two of three | ||
| engines against a callee whose second parameter is `status`, and BOTH | ||
| crashed on their FIRST fetch — invisible to import, `--help`, | ||
| `compileall`, the undefined-name walk and several hundred green | ||
| assertions, because none of those calls a function the way a live run | ||
| does. | ||
|
|
||
| Two failure modes, and the second is the one a weaker version of this | ||
| check swallows: a call whose arguments do not fit the signature, and a | ||
| call to a name the shared module **does not define at all**. Another | ||
| repo in this family resolved the callee with `getattr(..., None)` and | ||
| skipped whatever came back not-callable, so three calls into an API that | ||
| did not exist sat under a green run of its own binding check. Absent is | ||
| the loudest failure available, not "nothing to bind". | ||
|
|
||
| Deliberately conservative: a call using `*args` or `**kwargs` is skipped | ||
| rather than guessed at, so this under-reports and never invents a | ||
| problem. | ||
| """ | ||
| group("every call into a shared module binds against its real signature") | ||
| ok = True | ||
| shared = {} | ||
| for name in ("product_parser", "output_writer", "page_flow", "proxy_pool", | ||
| "captcha_solver", "env_config", "scraper_api_client", | ||
| "fingerprint_client"): | ||
| if os.path.exists(os.path.join(REPO_ROOT, name + ".py")): | ||
| try: | ||
| shared[name] = importlib.import_module(name) | ||
| except Exception as exc: # noqa: BLE001 | ||
| ok &= check("%s imports (%r)" % (name, exc), False) | ||
|
|
||
| callers = [e + ".py" for e in ENGINES] | ||
| callers += [n for n in ("page_flow.py", "product_parser.py", | ||
| "scraper_api_client.py") | ||
| if os.path.exists(os.path.join(REPO_ROOT, n))] | ||
|
Comment on lines
+1443
to
+1446
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
That undercuts the claim this check makes in CHANGELOG.md (line 15): "binds every call into a shared module against the callee's real signature." It's not every call — calls made from one shared module into another are skipped entirely, and this is exactly the file/call-path ( Suggest adding |
||
| checked = 0 | ||
| for filename in callers: | ||
| path = os.path.join(REPO_ROOT, filename) | ||
| if not os.path.exists(path): | ||
| continue | ||
| tree = ast.parse(open(path, encoding="utf-8").read()) | ||
|
|
||
| # Names bound anywhere in this file shadow a same-named module: an | ||
| # engine takes `proxy_pool` as a PARAMETER, and `proxy_pool.next()` | ||
| # on that parameter is a method call, not a module attribute. | ||
| bound = set() | ||
| for node in ast.walk(tree): | ||
| if isinstance(node, ast.Name) and isinstance(node.ctx, (ast.Store, ast.Del)): | ||
| bound.add(node.id) | ||
| elif isinstance(node, ast.arg): | ||
| bound.add(node.arg) | ||
| elif isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): | ||
| bound.add(node.name) | ||
|
|
||
| direct = {} | ||
| for node in ast.walk(tree): | ||
| if isinstance(node, ast.ImportFrom) and node.module in shared and not node.level: | ||
| for alias in node.names: | ||
| direct[alias.asname or alias.name] = (node.module, alias.name) | ||
|
|
||
| for node in ast.walk(tree): | ||
| if not isinstance(node, ast.Call): | ||
| continue | ||
| target = None | ||
| if isinstance(node.func, ast.Name) and node.func.id in direct: | ||
| target = direct[node.func.id] | ||
|
Comment on lines
+1476
to
+1477
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Minor asymmetry: this Not triggered by any current caller, but worth the same |
||
| elif (isinstance(node.func, ast.Attribute) | ||
| and isinstance(node.func.value, ast.Name) | ||
| and node.func.value.id in shared | ||
| and node.func.value.id not in bound): | ||
| target = (node.func.value.id, node.func.attr) | ||
| if target is None: | ||
| continue | ||
| module_name, attr = target | ||
| module = shared[module_name] | ||
| if not hasattr(module, attr): | ||
| ok &= check("%s:%d calls %s.%s, which does not exist — a " | ||
| "live run reaches this as AttributeError" | ||
| % (filename, node.lineno, module_name, attr), | ||
| False) | ||
| continue | ||
| callee = getattr(module, attr) | ||
| if not (inspect.isfunction(callee) or inspect.isclass(callee)): | ||
| continue | ||
| if (any(isinstance(a, ast.Starred) for a in node.args) | ||
| or any(k.arg is None for k in node.keywords)): | ||
| continue # unpacking: cannot bind statically | ||
| try: | ||
| signature = inspect.signature(callee) | ||
| except (TypeError, ValueError): | ||
| continue | ||
| try: | ||
| signature.bind(*([inspect.Parameter.empty] * len(node.args)), | ||
| **{k.arg: inspect.Parameter.empty | ||
| for k in node.keywords}) | ||
| except TypeError as exc: | ||
| ok &= check("%s:%d %s.%s%s — %s" | ||
| % (filename, node.lineno, module_name, attr, | ||
| signature, exc), False) | ||
| else: | ||
| checked += 1 | ||
| ok &= check("...and there were calls to bind (%d)" % checked, checked > 10) | ||
| return ok | ||
|
|
||
|
|
||
| def test_sample_output(): | ||
| group("shipped sample output") | ||
| ok = True | ||
|
|
@@ -1556,6 +1667,7 @@ def main() -> int: | |
| ok &= test_ci_checks_is_actually_wired_up() | ||
| ok &= test_fingerprint_client_reads_env() | ||
| ok &= test_wording() | ||
| ok &= test_shared_calls_bind_against_the_real_signature() | ||
| ok &= test_sample_output() | ||
|
|
||
| print() | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Altitude / reuse: this duplicates
claude-code-review.ymlverbatim, reproducing the exact drift risk this PR's own CHANGELOG describes.This "Install Claude Code (stable channel)" block (and the matching
path_to_claude_code_executable: ${{ env.CLAUDE_BIN }}line below) is a line-for-line copy of the block already in.github/workflows/claude-code-review.yml. The CHANGELOG entry for this PR even says so explicitly: "The fix had already landed inclaude-code-review.ymlin this repo and never reached its twin."Copy-pasting the fix into the second file doesn't remove the underlying problem — it just resets the clock on it. There's no
.github/actions/composite action in this repo, so the next time this install step needs to change (upstream deprecatesstable, the install path changes, etc.), whoever edits one workflow can just as easily forget the other again, exactly as happened here.Suggest factoring this into a composite action (e.g.
.github/actions/install-claude-stable/action.yml) that bothclaude.ymlandclaude-code-review.ymlcall, so there is exactly one place to fix next time instead of two files that can silently drift apart.