feat(cli): add airbyte-cloud and airbyte-local CLIs#1010
feat(cli): add airbyte-cloud and airbyte-local CLIs#1010Aaron ("AJ") Steers (aaronsteers) wants to merge 41 commits into
airbyte-cloud and airbyte-local CLIs#1010Conversation
Implements a new CLI invokable as 'uvx airbyte ...' or 'airbyte' when installed. Commands: - airbyte workspaces list/get - airbyte sources list/get/create/delete - airbyte destinations list/get/create/delete - airbyte connections list/get/create/delete/sync - airbyte jobs list/get Features: - Structured JSON output for agent consumption - --describe flag for schema discovery - Credential resolution: env vars -> ~/.airbyte/credentials file - Thin wrappers over existing api_util core module
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This PyAirbyte VersionYou can test this version of PyAirbyte using the following: # Run PyAirbyte CLI from this branch:
uvx --from 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1775171846-airbyte-cli' pyairbyte --help
# Install PyAirbyte from this branch for development:
pip install 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1775171846-airbyte-cli'PR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful ResourcesCommunity SupportQuestions? Join the #pyairbyte channel in our Slack workspace. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a JSON-first Click Cloud CLI with structured error payloads, lazy credential resolution helpers, SDK-to-dict serializers, command groups for workspaces/sources/destinations/connections/jobs, a main() runner that formats errors as JSON, a docs generator to render Click commands to Markdown, and a console script entry point. ChangesCloud CLI
CLI docs generator & generated references
Sequence Diagram(s)sequenceDiagram
actor User
participant CLI as Cloud CLI
participant Auth as Auth Resolver
participant Env as Env/Credentials File
participant API as Airbyte API
User->>CLI: run command with options
CLI->>Auth: resolve_* (explicit or ctx values)
Auth->>Env: check explicit → short env vars → long env vars → credentials file
Env-->>Auth: return value or empty
Auth-->>CLI: resolved credentials or raise PyAirbyteInputError
CLI->>API: call endpoint (workspaces/sources/...)
API-->>CLI: SDK response
CLI->>CLI: serialize to dict/JSON
CLI-->>User: emit JSON (stdout) or error JSON (stderr)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
PyTest Results (Fast Tests Only, No Creds)85 tests - 344 84 ✅ - 345 23s ⏱️ - 5m 38s For more details on these failures, see this check. Results for commit a8236b5. ± Comparison against base commit 70a7d82. This pull request removes 344 tests.♻️ This comment has been updated with latest results. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…ructured errors - Store raw auth values in ctx.obj; resolve lazily in subcommands so --describe works without credentials configured - Catch OSError and yaml.YAMLError separately in _read_credentials_file using guard statements instead of broad try/except - Add --force flag to delete commands (sources, destinations, connections) with safe_mode=True by default - Wrap main() entry point to produce structured JSON errors on stderr - Add PyAirbyteInputError import for structured error handling
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…, describe accuracy
All commands now defer required-option validation until after the --describe check. This lets agents discover schemas without providing auth or resource IDs.
- Add _register_schema() and _emit_json_help() helpers - Create _JsonHelpGroup and _JsonHelpCommand Click classes - Add --format option to root cli group (text|json) - Remove all --describe flags from all commands - All commands now support --format json --help for JSON schemas - Root-level --format json --help works via sys.argv fallback
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
- Change _error_json return type to NoReturn for type narrowing - Always call resolve_workspace_id() in workspaces_list for env/creds fallback - Add catch-all except Exception handler in main() for JSON error contract
Broad exception catching violates team coding standards. Only catch specific exceptions that can be handled meaningfully.
Pyrefly does not narrow Optional types after NoReturn calls without an explicit return statement in the guard clause.
PyTest Results (Full)512 tests +13 493 ✅ +12 23m 46s ⏱️ - 1m 14s For more details on these failures, see this check. Results for commit a8236b5. ± Comparison against base commit 70a7d82. ♻️ This comment has been updated with latest results. |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Aaron ("AJ") Steers (aaronsteers)
changed the title
airbyte cloud CLI
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
Aaron ("AJ") Steers (@aaronsteers) Done — the requested revert is pushed as Changed:
Validation passed:
|
devin-ai-integration
Bot
changed the title
airbyte cloud CLI
Aaron ("AJ") Steers (aaronsteers)
changed the title
airbyte-cloud and airbyte-local CLIs
| client_secret=client_secret, | ||
| bearer_token=bearer_token, | ||
| ) | ||
| ).get_organization(organization_id=organization_id, organization_name=organization_name) |
There was a problem hiding this comment.
Use this syntax with two or more named args.
| ).get_organization(organization_id=organization_id, organization_name=organization_name) | |
| ).get_organization( | |
| organization_id=organization_id, | |
| organization_name=organization_name, | |
| ) |
| def list_workspaces_in_organization( | ||
| self, | ||
| organization_id: str | None = None, | ||
| *, | ||
| name_contains: str | None = None, | ||
| limit: int | None = None, | ||
| ) -> list[dict[str, object]]: | ||
| """List workspaces in an organization using the Config API.""" | ||
| resolved_organization_id = organization_id or self.organization_id | ||
| if not resolved_organization_id: | ||
| raise exc.PyAirbyteInputError( | ||
| message="Organization ID is required.", | ||
| guidance="Provide an organization ID.", | ||
| ) | ||
|
|
||
| return api_util.list_workspaces_in_organization( | ||
| organization_id=resolved_organization_id, | ||
| api_root=self.public_api_root, | ||
| config_api_root=self.config_api_root, | ||
| client_id=self.client_id, | ||
| client_secret=self.client_secret, | ||
| bearer_token=self.bearer_token, | ||
| name_contains=name_contains, | ||
| limit=limit, | ||
| ) |
There was a problem hiding this comment.
This method can be combined with list_workspaces, with an optional org id input.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Reverting to draft (on hold), as we are pivoting to first an internal CLI app launch. |
|
Closing. Migrated to ops repo cli (internal). |
Summary
Adds Cyclopts-backed PyAirbyte CLI entrypoints split by operating mode and routes Cloud auth/workspace construction through shared public Cloud APIs:
airbyte-cloud ...for Airbyte Cloud and self-managed control-plane API operations.airbyte-local ...for local PyAirbyte connector workflows migrated from the old local CLI commands.airbyte,pyab, andpyairbyteconsole-script entrypoints.airbyte.cloud.CloudClientas the higher-level authenticated Cloud/self-managed API facade, with optionalorganization_idsupport.CloudClientwithCloudWorkspaceinairbyte.cloud.workspacesand removesairbyte/cloud/client.py, avoiding circular dependencies and inline imports.CloudClient.get_workspace_from_auth(); CLI workspace-scoped commands use the publicCloudWorkspace(...)constructor directly, andCloudWorkspaceresolves credentials throughCloudClientinternally.airbyte.cloud.credentials, including explicit inputs, env vars, and~/.airbyte/credentials._cli_auth.pyhelper layer; Cloud CLI and MCP now call shared Cloud code instead of building credentials/workspaces themselves.CloudClient.login(interactive: bool | None = None)andCloudClient.logout(); non-interactive client-credentials login stores a bearer token locally, while interactive browser login is intentionally left as aNotImplementedErrorwith a TK-TODO before merge.public_api_rootandconfig_api_rootnaming for new shared APIs. Self-managed login requires--public-api-root,--config-api-root,--client-id, and--client-secret.CloudClient,CloudWorkspace,CloudConnection,CloudSource,CloudDestination, andSyncResult.--jsonpayload.renamename-only and reservesupdatefor config/settings changes.connections enable,connections disable,connections schedule set, andconnections schedule manualinstead of exposing raw status changes.connections sync --wait/--no-wait --wait-timeoutandairbyte-cloud jobs wait.cyclopts.docs.generate_markdown_docs, matching the Airbyte-Ops-MCP CLI documentation pattern.docs/generate_cli_docs.pyfor clarity and updates imports, tests, docstrings, and contributor docs.Current public command shape:
Review & Testing Checklist for Human
Verify the command hierarchy and help text match the intended public CLI surface:
Verify Cloud auth behavior against real credentials, including:
uv run airbyte-cloud login --client-id ... --client-secret ... uv run airbyte-cloud workspaces list uv run airbyte-cloud logoutVerify a self-managed non-interactive login requires both base API roots:
Verify existing public Python construction still works with direct
CloudWorkspace(...)usage:Verify MCP Cloud tools still authenticate from MCP config and can list/describe workspaces and organizations.
Confirm the interactive browser-login TK-TODO is resolved before merging, or intentionally keep this PR draft until the browser-login implementation lands.
Notes
Local validation passed after the latest same-module
CloudClient/CloudWorkspacecleanup:pyreflyreports 0 errors with existing suppressed warnings.ruffreports only the existing repo-level deprecation warning for top-level linter settings.Link to Devin session: https://app.devin.ai/sessions/150e0657e5344e518579eab6db4f0221
Requested by: Aaron ("AJ") Steers (@aaronsteers)