feat: add platform-aware compute-worker + compute-service with 3-tier dev setup#52
Open
pyramation wants to merge 11 commits into
Open
feat: add platform-aware compute-worker + compute-service with 3-tier dev setup#52pyramation wants to merge 11 commits into
pyramation wants to merge 11 commits into
Conversation
- job/compute-worker: platform-aware worker that discovers functions from constructive_infra_public.platform_function_definitions (TTL-cached), tracks invocations in platform_function_invocations, dispatches via HTTP - job/compute-service: orchestrator (callback server + ComputeWorker + Scheduler), mirrors job/service patterns - scripts/setup-platform-db.sh: Tier 1 setup (pgpm deploy + seed) - scripts/dev-compute.ts: dev launcher for compute-service + functions - scripts/seed-functions.sql: registers send-email + send-verification-link - docker-compose.yml: adds platform-setup service (Tier 2) - k8s/overlays/local-simple/compute-service.yaml: K8s manifest (Tier 3) - Makefile: adds setup-platform, dev-compute targets + tier documentation - .agents/skills/dev-tiers: skill for the 3-tier dev model - .agents/skills/compute-worker: skill for the compute-worker system - pgpm/constructive-infra: adds @pgpm/database-jobs dependency
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
…error) pgpm deploy requires module dependencies to be pre-installed via 'pgpm install' before deployment. Without this, packages like @pgpm/database-jobs fail with 'extension not available'. Adds pgpm install to: - scripts/setup-platform-db.sh (Tier 1) - docker-compose.yml platform-setup service (Tier 2) - .github/workflows/pgpm-test.yaml (CI)
|
All alerts resolved. Learn more about Socket for GitHub. This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored. |
- make status: shows Docker containers, PG connection, databases with infra schema, Node/pnpm/pgpm versions, and build state - make verify-platform: checks DB exists, infra + jobs schemas deployed, tables present, functions seeded — exits non-zero with fix instructions if anything is wrong
Dan committed all extension deps (database-jobs, metaschema-modules, services, etc.) directly into extensions/. pgpm install is no longer needed before deploy. Also restores pgpm volume to :ro in Docker Compose and mounts extensions/ for the platform-setup service.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Built-in function definitions (send-email, send-verification-link) are now deployed as a pgpm fixture inside constructive-infra. The fixture uses the standard deploy/revert/verify pattern following the inflection module's fixtures convention. - deploy: INSERT ON CONFLICT DO NOTHING - revert: DELETE WHERE is_built_in AND scope='platform' - verify: SELECT 1 from seed row Removes manual psql seed step from setup script, docker-compose, and verify-platform fix instructions. pgpm deploy now handles everything.
Moves the built-in function seed data (send-email, send-verification-link) from constructive-infra into its own pgpm package: constructive-infra-seed. This keeps constructive-infra as pure DDL (schemas, tables, triggers) and the seed data as a separate deployable unit. The seed package depends on constructive-infra via the .control file requires. Both packages pass pgpm test-packages --full-cycle.
compute-service discovers functions from the database, so it doesn't need direct workspace deps on send-email-fn or send-verification-link-fn (which are generated packages in generated/ and not in the workspace).
…nfigs - New fixture: seed_built_in_secrets seeds MAILGUN_* and SMTP_* into platform_secret_definitions (with well-known default database_id) - Function definitions now include required_secrets and required_configs arrays (function_requirement[] type) linking functions to their deps - All SQL files now use proper pgpm format: BEGIN/COMMIT wrappers, '-- Deploy:' header style with '-- made with <3 @ constructive.io' - New script: load-platform-env.sh reads .env, cross-references against DB function requirements, reports satisfied vs missing keys - .env.example updated with SMTP/Mailpit + dry-run toggle sections - Makefile: added 'make check-env' target
Procedural lifecycle targets: make up # prereqs → docker → bootstrap → deploy → seed → verify make up DB_NAME=mydb # same with custom DB make down # stop docker compose + pgpm docker stop DROP=1 make down DB_NAME=mydb # also drop the DB make up:email-job # start mailpit + compute-service (SMTP mode) make down:email-job # stop mailpit + compute-service email-job-up verifies platform is up first, starts mailpit, loads .env (with sane SMTP defaults), then launches compute-service. status now shows mailpit container state.
…rrides - Skip python-example and other non-node-graphql functions from startup - Print a clear port/service summary table before launching processes - Respect SEND_EMAIL_DRY_RUN, SEND_VERIFICATION_LINK_DRY_RUN, EMAIL_SEND_USE_SMTP, and SMTP_FROM from environment (was hardcoded)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a platform-aware job dispatch system that discovers functions from the database (instead of static manifests) and tracks invocations. Includes full dev environment integration across all 3 tiers.
New packages
job/compute-worker— Platform-aware job dispatcher:FunctionDiscovery: lazy SQL queries againstconstructive_infra_public.platform_function_definitions, TTL-cached (default 60s)InvocationTracker: records lifecycle inplatform_function_invocations(create → complete/fail)ComputeWorker: pollsapp_jobs.jobs, resolves function via discovery, HTTP POSTs toservice_urljob/compute-service— Orchestrator (mirrors existingjob/servicepattern):ComputeWorker+SchedulerbootCompute()checks for required schemas before startingpgpm module (
constructive-infra)pgpm installneeded)pgpm deployhandles everything, no manual SQL steppgpm test-packages --full-cycle.github/workflows/pgpm-test.yaml) runs on PRs touchingpgpm/Dev environment (3-tier model)
pgpm docker start+ local Nodemake setup-platform,make dev-computedocker compose up+ local Nodeplatform-setupservice added to composecompute-service.yamlmanifest addedState verification:
make status(environment overview) andmake verify-platform DB_NAME=<db>(checks schemas, tables, seeded functions).Skills
.agents/skills/dev-tiers/SKILL.md— documents the 3-tier dev model.agents/skills/compute-worker/SKILL.md— compute-worker architecture and usageLink to Devin session: https://app.devin.ai/sessions/e4d315e507cf4d95ac42056ad22b8925
Requested by: @pyramation