Skip to content

Cross platform packaging - #587

Open
JoeZiminski wants to merge 51 commits into
mainfrom
cross-platform-packaging
Open

Cross platform packaging#587
JoeZiminski wants to merge 51 commits into
mainfrom
cross-platform-packaging

Conversation

@JoeZiminski

@JoeZiminski JoeZiminski commented Sep 2, 2025

Copy link
Copy Markdown
Member

This PR introduces cross-platform packaging of datashuttle. This requires both packaging datashuttle itself as well as vendoring a terminal emulator to deal with some rendering issues.

The packaging process is threefold:

  1. package datashuttle
  2. package a lightweight script that launches the vendored terminal emulated and runs the packaged datashutlte within it.
  3. wrap in an installer (inno setup on Windows, bundle on macOS, unsure on Linux, we might not wrap in an installer there).

Currently, this PR:

  1. Works on Windows
  2. Packaging works on macOS intel and silicon. However, the silicon version does not work on intel, and vice versa. A little more work should be done to investigate why this is (a single distribution is possible in some cases). If it's not possible, we will have two separate distributions. Still need to wrap in a installer bundle.
  3. Packaging is working on ubuntu, need to look into packaging on manylinux and whether to wrap in an installer.

Currently the architecture is:

  1. datashuttle.spec is shared between operating systems, and is the pyinstaller config file to control packaging of datashuttle itself. It will package datashuttle itself, by packaging the datashuttle_launcher.py script. This will then create a executable to be called in the vendored terminal emulator.
  2. There are two top-level packaging scripts, one for each OS (package_windows.py, package_macos.py). These coordinate the download of the terminal emulator to vendor, packaging of datashuttle, then triggers the running of terminal_launcher_xxx.spec which in turn packages terminal_launcher.py. This handles how the vendored terminal should call the datashuttle executable.

So essentially, the idea is to package datashuttle, and then package a script that opens the vendored terminal and runs the datashuttle executable within it.

One the packaging is complete, we will need to extend the CI to run these installers, generate the artifact to distribute, and add it to the website.

Note that this PR is still in the prototype phase and requires some refactoring and tidying up.

PLAN:

  • Release Fix issue running transfers when Linux bash does not reside in /bin/bash #742
  • Split this PR and make a PR for windows release only, including artefact upload and docs link. We will have to make rc releases, which will still upload the docs, so the install page might look strange for a day or so while this is tested.
  • Once the windows release works, then add the macOS release. This will need to include code signing and support for intel and arm.

TODO:
windows, check all raw "rclone" calls and make sure they use the new "get_command". Need to add this new convention to dev docs

@JoeZiminski
JoeZiminski force-pushed the cross-platform-packaging branch 5 times, most recently from 00cf5ae to acdaada Compare September 2, 2025 20:52
Comment thread .github/workflows/package_windows.yml Fixed
@JoeZiminski
JoeZiminski force-pushed the cross-platform-packaging branch from 554cd8a to 5f06120 Compare February 24, 2026 00:34
Comment thread .github/workflows/package_windows.yml Fixed
Comment thread .github/workflows/package_windows.yml Fixed
Comment thread .github/workflows/package_macos.yml Fixed
Comment thread .github/workflows/package_windows.yml Fixed
@JoeZiminski
JoeZiminski force-pushed the cross-platform-packaging branch from 077a919 to f079b5d Compare May 25, 2026 11:43
@JoeZiminski
JoeZiminski force-pushed the cross-platform-packaging branch from ea4063d to 77b26d9 Compare May 25, 2026 11:46
fetch-depth: 0

- name: Set up Conda
uses: conda-incubator/setup-miniconda@v4
startsWith(github.ref, 'refs/tags/v') ||
(github.event_name == 'workflow_dispatch' &&
inputs.test_release_upload)
uses: apple-actions/import-codesign-certs@v3
startsWith(github.ref, 'refs/tags/v') ||
(github.event_name == 'workflow_dispatch' &&
inputs.test_release_upload)
uses: softprops/action-gh-release@v2
steps:
- uses: actions/checkout@v6
- name: Set up Conda
uses: conda-incubator/setup-miniconda@v4
startsWith(github.ref, 'refs/tags/v') ||
(github.event_name == 'workflow_dispatch' &&
inputs.test_release_upload)
uses: softprops/action-gh-release@v2
)
else:
output = subprocess.run(command, shell=True)
output = subprocess.run(format_command, shell=True)

- name: Install datashuttle and packaging deps
run: |
python -m pip install --upgrade pip
- name: Install datashuttle and packaging deps
run: |
python -m pip install --upgrade pip
python -m pip install .[dev]
run: |
python -m pip install --upgrade pip
python -m pip install .[dev]
python -m pip install pyinstaller requests
@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.71429% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.12%. Comparing base (24b4ca0) to head (4dddb72).

Files with missing lines Patch % Lines
datashuttle/utils/rclone.py 68.75% 5 Missing ⚠️
datashuttle/tui_launcher.py 0.00% 4 Missing ⚠️
datashuttle/tui/app.py 75.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #587      +/-   ##
==========================================
- Coverage   90.28%   90.12%   -0.16%     
==========================================
  Files          54       54              
  Lines        4424     4445      +21     
==========================================
+ Hits         3994     4006      +12     
- Misses        430      439       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants