UN-3266 [MISC] Remove redundant worker_celery.py and fix ide-callback worker#1892
Merged
chandrasekharan-zipstack merged 1 commit intomainfrom Apr 3, 2026
Merged
Conversation
Remove the unnecessary second Celery app (worker_celery.py) introduced in PR #1849 and consolidate on the existing celery_service.app. Both apps used the same RabbitMQ broker, PostgreSQL result backend, and JSON serialization — the duplication solved a non-existent problem. Also removes dead legacy Celery tasks (run_index_document, run_fetch_response, run_single_pass_extraction) that have zero callers since the executor migration, and fixes the ide-callback worker which was missing from run-worker-docker.sh's WORKERS, WORKER_QUEUES, and WORKER_HEALTH_PORTS arrays — causing it to crash locally. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
WalkthroughThe PR removes a worker-specific Celery application factory and its associated task definitions, consolidating to use a centralized Celery app instance. References across Prompt Studio modules are updated accordingly, and worker configuration mappings are added for IDE callback service naming. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Test ResultsSummary
Runner Tests - Full Report
SDK1 Tests - Full Report
|
|
Contributor
|
| Filename | Overview |
|---|---|
| backend/backend/worker_celery.py | Deleted: 114-line custom Celery subclass with double-checked locking singleton. Superseded by celery_service.app which uses identical broker/backend/serialization config. |
| backend/prompt_studio/prompt_studio_core_v2/prompt_studio_helper.py | _get_dispatcher() now uses module-level celery_app import (consistent with 6 other files in the codebase) instead of a lazy import of the deleted worker_celery.py. |
| backend/prompt_studio/prompt_studio_core_v2/tasks.py | Deleted: 274-line file with legacy tasks (run_index_document, run_fetch_response, run_single_pass_extraction) that had zero callers after views were rewritten to use ExecutionDispatcher. |
| backend/prompt_studio/prompt_studio_core_v2/test_tasks.py | Deleted: 493-line test file for the deleted legacy tasks; deletion is appropriate. Tests for the live ide_callback path exist in workers/tests/test_ide_callback.py. |
| backend/prompt_studio/prompt_studio_core_v2/views.py | task_status() now uses module-level celery_app (consistent pattern); lazy imports removed. Docstring on lines 751–753 still references the old two-app design and should be updated. |
| docker/docker-compose.yaml | ide-callback service command corrected from ide_callback to ide-callback, matching the key in WORKERS array and aligning with other service naming conventions. |
| workers/run-worker-docker.sh | ide-callback/ide_callback entries added to WORKERS, WORKER_QUEUES, and WORKER_HEALTH_PORTS; this is the core fix for the broken local worker. detect_worker_type_from_args has no case for ide_callback queue (PATH 1 path, minor). |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["Docker Compose\ncommand: [ide-callback]"] --> B["run-worker-docker.sh\nPATH 2: Traditional worker type"]
B --> C["resolve_worker_type('ide-callback')\nWORKERS['ide-callback'] = 'ide_callback'"]
C --> D["run_worker('ide_callback')"]
D --> E["WORKER_QUEUES['ide_callback'] = 'ide_callback'\nWORKER_HEALTH_PORTS['ide_callback'] = '8089'"]
E --> F["export IDE_CALLBACK_HEALTH_PORT=8089\nexport IDE_CALLBACK_METRICS_PORT=8089"]
F --> G["celery -A worker worker\n--queues=ide_callback"]
H["Django View: task_status()"] --> I["AsyncResult(task_id, app=celery_app)"]
J["PromptStudioHelper._get_dispatcher()"] --> K["ExecutionDispatcher(celery_app=celery_app)"]
I --> L["celery_service.app\nBroker: RabbitMQ CELERY_BROKER_URL\nBackend: PostgreSQL"]
K --> L
style A fill:#d4edda
style G fill:#d4edda
style L fill:#d4edda
Comments Outside Diff (2)
-
workers/run-worker-docker.sh, line 200-212 (link)ide_callbackmissing from queue-to-worker-type detectiondetect_worker_type_from_args(PATH 1) has no case for theide_callbackqueue. If the worker is ever started with a full Celery command (/app/.venv/bin/celery -A worker worker --queues=ide_callback ...) the function falls through to the*) echo "general"fallback, and the health/metrics port env vars are set forgeneral(8081) instead ofide_callback(8089).PATH 2 (the docker-compose path changed in this PR) is unaffected, but it's worth covering PATH 1 for parity with all other workers.
Prompt To Fix With AI
This is a comment left during a code review. Path: workers/run-worker-docker.sh Line: 200-212 Comment: **`ide_callback` missing from queue-to-worker-type detection** `detect_worker_type_from_args` (PATH 1) has no case for the `ide_callback` queue. If the worker is ever started with a full Celery command (`/app/.venv/bin/celery -A worker worker --queues=ide_callback ...`) the function falls through to the `*) echo "general"` fallback, and the health/metrics port env vars are set for `general` (8081) instead of `ide_callback` (8089). PATH 2 (the docker-compose path changed in this PR) is unaffected, but it's worth covering PATH 1 for parity with all other workers. How can I resolve this? If you propose a fix, please make it concise.
-
backend/prompt_studio/prompt_studio_core_v2/views.py, line 749-753 (link)Stale docstring references removed two-app design
Lines 751–753 still say "Both apps share the same PostgreSQL result backend, so we use the worker app to look up results." This refers to the now-deleted two-app design (
worker_celery.py+celery_service.py). Since this PR consolidates everything on a singlecelery_service.app, the mention of two apps and the "worker app" concept is no longer accurate.Prompt To Fix With AI
This is a comment left during a code review. Path: backend/prompt_studio/prompt_studio_core_v2/views.py Line: 749-753 Comment: **Stale docstring references removed two-app design** Lines 751–753 still say "Both apps share the same PostgreSQL result backend, so we use the worker app to look up results." This refers to the now-deleted two-app design (`worker_celery.py` + `celery_service.py`). Since this PR consolidates everything on a single `celery_service.app`, the mention of two apps and the "worker app" concept is no longer accurate. How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: workers/run-worker-docker.sh
Line: 200-212
Comment:
**`ide_callback` missing from queue-to-worker-type detection**
`detect_worker_type_from_args` (PATH 1) has no case for the `ide_callback` queue. If the worker is ever started with a full Celery command (`/app/.venv/bin/celery -A worker worker --queues=ide_callback ...`) the function falls through to the `*) echo "general"` fallback, and the health/metrics port env vars are set for `general` (8081) instead of `ide_callback` (8089).
PATH 2 (the docker-compose path changed in this PR) is unaffected, but it's worth covering PATH 1 for parity with all other workers.
```suggestion
*"file_processing"*) echo "file_processing" ;;
*"celery_api_deployments"*) echo "api_deployment" ;;
*"file_processing_callback"*) echo "callback" ;;
*"notifications"*) echo "notification" ;;
*"celery_log_task_queue"*) echo "log_consumer" ;;
*"scheduler"*) echo "scheduler" ;;
*"${EXECUTOR_WORKER_TYPE}"*) echo "${EXECUTOR_WORKER_TYPE}" ;;
*"ide_callback"*) echo "ide_callback" ;;
*"celery"*) echo "general" ;;
*) echo "general" ;; # fallback
```
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: backend/prompt_studio/prompt_studio_core_v2/views.py
Line: 749-753
Comment:
**Stale docstring references removed two-app design**
Lines 751–753 still say "Both apps share the same PostgreSQL result backend, so we use the worker app to look up results." This refers to the now-deleted two-app design (`worker_celery.py` + `celery_service.py`). Since this PR consolidates everything on a single `celery_service.app`, the mention of two apps and the "worker app" concept is no longer accurate.
```suggestion
"""Poll the status of an async Prompt Studio task.
Task IDs point to executor worker tasks dispatched via the
main Celery app (celery_service.app).
```
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "[MISC] Remove redundant worker_celery.py..." | Re-trigger Greptile
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
workers/run-worker-docker.sh (1)
42-73:⚠️ Potential issue | 🟡 MinorAdd
ide_callbackinference in full-command mode.Line 42/60/73 add support for the worker in the traditional path, but
detect_worker_type_from_args()still does not recognize--queues=ide_callback. In PATH 1, this can fall back togeneraland export incorrectWORKER_TYPE/health variables.Suggested patch
diff --git a/workers/run-worker-docker.sh b/workers/run-worker-docker.sh @@ case "$queues" in *"file_processing"*) echo "file_processing" ;; *"celery_api_deployments"*) echo "api_deployment" ;; *"file_processing_callback"*) echo "callback" ;; + *"ide_callback"*) echo "ide_callback" ;; *"notifications"*) echo "notification" ;; *"celery_log_task_queue"*) echo "log_consumer" ;; *"scheduler"*) echo "scheduler" ;; *"${EXECUTOR_WORKER_TYPE}"*) echo "${EXECUTOR_WORKER_TYPE}" ;; *"celery"*) echo "general" ;;🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@workers/run-worker-docker.sh` around lines 42 - 73, detect_worker_type_from_args() doesn't recognize the new "ide_callback" queue flag, so when users pass "--queues=ide_callback" the script can fall back to "general" and export wrong WORKER_TYPE/health settings; update detect_worker_type_from_args() to detect the "--queues=" argument containing "ide_callback" (similar to how it handles "file_processing" or "callback") and map it to "ide_callback", and ensure PLUGGABLE_WORKERS, WORKER_QUEUES and WORKER_HEALTH_PORTS logic consistently accept "ide_callback" so exported WORKER_TYPE and health port match the configured queue.
🧹 Nitpick comments (1)
workers/run-worker-docker.sh (1)
42-73: Extractide_callbackinto a constant for consistency.
ide_callbackis now repeated across multiple maps. A single readonly constant reduces drift risk when renaming or refactoring worker types.Suggested refactor
diff --git a/workers/run-worker-docker.sh b/workers/run-worker-docker.sh @@ readonly EXECUTOR_WORKER_TYPE="executor" +readonly IDE_CALLBACK_WORKER_TYPE="ide_callback" @@ - ["ide-callback"]="ide_callback" + ["ide-callback"]="${IDE_CALLBACK_WORKER_TYPE}" @@ - ["ide_callback"]="ide_callback" + ["${IDE_CALLBACK_WORKER_TYPE}"]="${IDE_CALLBACK_WORKER_TYPE}" @@ - ["ide_callback"]="8089" + ["${IDE_CALLBACK_WORKER_TYPE}"]="8089"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@workers/run-worker-docker.sh` around lines 42 - 73, Introduce a single readonly constant for the repeated worker type (e.g., readonly IDE_CALLBACK="ide_callback") and replace all literal occurrences of "ide_callback" in the script with the constant reference (${IDE_CALLBACK})—update the PLUGGABLE_WORKERS declaration, the WORKER_QUEUES key mapping and the WORKER_HEALTH_PORTS key mapping, and any other places using the literal string so the worker type is defined in one place and used consistently.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@workers/run-worker-docker.sh`:
- Around line 42-73: detect_worker_type_from_args() doesn't recognize the new
"ide_callback" queue flag, so when users pass "--queues=ide_callback" the script
can fall back to "general" and export wrong WORKER_TYPE/health settings; update
detect_worker_type_from_args() to detect the "--queues=" argument containing
"ide_callback" (similar to how it handles "file_processing" or "callback") and
map it to "ide_callback", and ensure PLUGGABLE_WORKERS, WORKER_QUEUES and
WORKER_HEALTH_PORTS logic consistently accept "ide_callback" so exported
WORKER_TYPE and health port match the configured queue.
---
Nitpick comments:
In `@workers/run-worker-docker.sh`:
- Around line 42-73: Introduce a single readonly constant for the repeated
worker type (e.g., readonly IDE_CALLBACK="ide_callback") and replace all literal
occurrences of "ide_callback" in the script with the constant reference
(${IDE_CALLBACK})—update the PLUGGABLE_WORKERS declaration, the WORKER_QUEUES
key mapping and the WORKER_HEALTH_PORTS key mapping, and any other places using
the literal string so the worker type is defined in one place and used
consistently.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 634f6c5f-99bc-4a41-ad7a-e5ea8f676359
📒 Files selected for processing (7)
backend/backend/worker_celery.pybackend/prompt_studio/prompt_studio_core_v2/prompt_studio_helper.pybackend/prompt_studio/prompt_studio_core_v2/tasks.pybackend/prompt_studio/prompt_studio_core_v2/test_tasks.pybackend/prompt_studio/prompt_studio_core_v2/views.pydocker/docker-compose.yamlworkers/run-worker-docker.sh
💤 Files with no reviewable changes (3)
- backend/prompt_studio/prompt_studio_core_v2/test_tasks.py
- backend/backend/worker_celery.py
- backend/prompt_studio/prompt_studio_core_v2/tasks.py
chandrasekharan-zipstack
requested review from
harini-venkataraman and
muhammad-ali-e
muhammad-ali-e
approved these changes
Apr 3, 2026
chandrasekharan-zipstack
changed the title
harini-venkataraman
approved these changes
Apr 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
worker_celery.py) and consolidate all task dispatch on the existingcelery_service.apprun_index_document,run_fetch_response,run_single_pass_extraction) and their testsworker-ide-callbackdocker-compose and entrypoint script configurationWhy
PR #1849 introduced
worker_celery.py— a custom Celery subclass with connection overrides, thread-safe singleton, and double-checked locking — under the mistaken premise that the Django backend used a separate Redis-backed Celery app. In reality, both apps use the identical RabbitMQ broker (settings.CELERY_BROKER_URL), PostgreSQL result backend, and JSON serialization. Every other dispatch site in the codebase (workflow_helper.py,notification,manual_review,pipeline_api) already usescelery_service.appsuccessfully.The legacy tasks (
prompt_studio_index_document,prompt_studio_fetch_response,prompt_studio_single_pass) have zero callers — the views were rewritten to useExecutionDispatcherdirectly.The
worker-ide-callbackservice was added by #1849 but never worked locally because:ide_callbackwas missing fromrun-worker-docker.sh'sWORKERS,WORKER_QUEUES, andWORKER_HEALTH_PORTSarraysIDE_CALLBACK_METRICS_PORT="", whichWorkerConfig.from_env()stripped toMETRICS_PORT="", crashing withint("")How
from backend.worker_celery import get_worker_celery_appwithfrom backend.celery_service import app as celery_appin 2 filesbackend/backend/worker_celery.py(114 lines)backend/prompt_studio/prompt_studio_core_v2/tasks.py(274 lines) andtest_tasks.py(493 lines)ide_callbacktoWORKERS,WORKER_QUEUES, andWORKER_HEALTH_PORTSinrun-worker-docker.sh["ide_callback"]to["ide-callback"]for consistency with other workersCan this PR break any existing features? If yes, please list possible items. If no, please explain why.
No. The
celery_service.appuses the same broker, result backend, and serialization as the deletedworker_celery.py. The deleted tasks had zero callers. The ide-callback fix enables functionality that was previously broken locally.Database Migrations
Env Config
Related Issues or PRs
Notes on Testing
worker-ide-callbackstarts and binds to correctide_callbackqueueChecklist
🤖 Generated with Claude Code