Skip to content

build: Makefile wrappers for macOS/Linux/Windows/Android/iOS, bootstrap, opt-in Xcode download, BOM check and docs - #1482

Open
parasew wants to merge 23 commits into
cypherstack:stagingfrom
parasew:feat/makefile-macos-build
Open

parasew wants to merge 23 commits into
cypherstack:stagingfrom
parasew:feat/makefile-macos-build

Conversation

@parasew

@parasew parasew commented Nov 1, 2026 •

Copy link
Copy Markdown

Summary

Developer-experience tooling for building Stack Wallet from a checkout with make, plus the build fixes and docs needed to make those targets match CI.

Current head: 4481c00d, based on staging at 7659bc866 (the branch is not behind). The PR is ready for review; all four hosted make-build jobs and both upstream checks passed, and the commit content tree is unchanged after rewriting the author/committer metadata to parasew <9727+parasew@users.noreply.github.com>.

Review fixes in the latest update

  • MWC FFI tests: removed integration_test/mwc_ffi_test.dart, mwc_ffi_test_service.dart, mwc_test_result.dart and make test-mwc. These tests exercise MWC itself and belong in flutter_libmwc, not Stack Wallet. They are ported to cypherstack/flutter_libmwc#16; the new plugin-level suite passes locally on macOS arm64 and Linux x86_64 (9 tests each).
  • Linux bundle: make build-linux now runs scripts/linux/prepare_release_bundle.py; check-reqs verifies patchelf, readelf and python3.
  • APP_NAME: macOS builds pass $(APP_NAME) instead of a literal stack_wallet, and Windows asset selection uses the selected app variant.
  • Windows copy mode: link_assets.sh copies directly when WINDOWS_COPY_ASSETS=1. It no longer relies on mklink succeeding first, and removes old junctions with cmd.exe rmdir before falling back to a normal delete.
  • Linux pc file: the tracked scripts/linux/pc/libsecret-1.pc is removed and git-ignored. build_secure_storage_deps.sh generates it; the Makefile no longer rewrites it.
  • Docker step: removed the duplicate coinlib:build_linux Docker step; build_app.sh already builds secp256k1.
  • Pub cache: only build-linux uses a repo-local .pub-cache, for the local cargo PROTOC config. Other targets use Flutter's normal per-user cache.
  • BOM check: verify_bom.dart fails on missing plugin pins, parses only stdout for Flutter's JSON, uses runInShell on Windows and honours the FLUTTER override.
  • Bootstrap/Xcode: Flutter defaults to 3.47.2 and accepts 3.47.4; the managed SDK is only added to PATH when needed; Homebrew paths are detected instead of hard-coded; Rust is pinned with rustup override rather than changing the global default; Xcode selection prefers the newest full app, and license/first-launch only run when pending; partial .xip downloads resume.
  • Benchmarks: bm/render.sh only opens output with --open; bm/bench.sh survives missing du/sccache; unused imports removed; bm/collect.sh is wired into bm/run_all.sh.
  • Runner queue: the fork has one hosted macOS concurrency slot, so make-build.yaml now runs macOS first and iOS after it. Running both together left one job queued until GitHub’s 15-minute timeout cancelled it.
  • Dead code: removed the orphaned NixOS installer and the old Windows asset-copy helper.
  • Whitespace: all 13 git diff --check errors are fixed.

What it adds

  • make build-macos: CI flow, PREBUILTS=1 by default (-d), PREBUILTS=0 compiles native assets from source.
  • make build-linux: CI flow, SYSTEM_DEPS=1 for system libsecret/jsoncpp, then bundle preparation.
  • make build-windows (Git Bash), make build-android (throw-away dev key if no signing key), make build-ios (no code signing).
  • make bootstrap-macos, make bootstrap-xcode, opt-in make bootstrap-xcode-download, check-reqs, check-sdk-macos, diagnose-env-macos.
  • make clean-app / clean-native, make sccache-setup / cache-stats.
  • stack-bom.toml + tool/verify_bom.dart + make verify-bom.
  • .github/workflows/make-build.yaml (manual only): Windows, macOS, Linux (with verify-bom) and iOS make targets.
  • bm/ benchmark scripts.
  • New “Building with make” documentation in docs/building.md.

Verification

  • Upstream PR checks on b82bc47da: linux-release-bundle pass, test pass (24m44s).
  • Fork Make build on b82bc47da: Windows, macOS, Linux and iOS all pass. The macOS job previously lost the fork’s single macOS concurrency slot to the iOS job; the workflow now serialises them.
  • Local macOS fallback: make build-macos inside nix develop on macOS 27.2 with Xcode 27.2 and Flutter 3.47.4: pass, Stack Wallet.app 318.8 MB.
  • Clean-room macOS 26.6.2 arm64 VM (Tart; Homebrew present, no Xcode, Flutter or Rust): make bootstrap-xcode-download ARGS=-y downloaded, SHA-1/signature-verified and installed Xcode 27.1; make bootstrap-macos installed Flutter 3.47.2 and Rust 1.99.1; make build-macos produced Stack Wallet.app 319.5 MB. all pass.
  • Xcode download resume harness: a pre-created partial .part was resumed with curl -C -, a relative cache path resolved correctly, and SHA-1/signature/install/-y selection all passed.
  • Android on x86_64 Linux (trek) in ghcr.io/cypherstack/stackwallet-ci:android: make build-android produced APKs 183.5 / 218.8 / 235.4 MB (arm / arm64 / x86_64). pass.
  • make verify-bom inside nix develop: 0 failures, 1 warning (local Xcode 27.2 vs recommended 27.1).
  • git diff --check, shell syntax, shellcheck, dart format, dart analyze, nix-instantiate, Python compile: pass.
  • link_assets.sh copy mode, repeat runs and symlink restoration tested locally.
  • Xcode selection and already-installed download_xcode.sh -y paths tested with stubs.

Earlier platform verification (before the latest review-fix commit): make build-macos on macOS 27.2 cold/from-source/in Nix, make build-linux on NixOS and Ubuntu 24.04, Android in the Docker image, and Windows/macOS/Linux/iOS on GitHub-hosted runners all succeeded. The latest commit re-validated all four make targets on hosted runners, both upstream checks, and Android packaging in the upstream Android image on an x86_64 Linux host.

Not tested / known limitations

  • Physical Windows developer machine: only the hosted windows-2022 runner was exercised.
  • Android with a real release key / app bundle: the APKs were debug-signed with the throw-away key.
  • iOS simulator and signing: the hosted job is a device build with --no-codesign.
  • Julian’s reproducible-build PR Repro dnix packages #1505 is a separate draft. Compatibility test on a throw-away merge: only docs/building.md conflicts; all make -n targets still parse; make verify-bom needs its plugin pins refreshed after Repro dnix packages #1505 changes pubspec.lock. The docs need a manual merge of this PR’s “Building with make” section into his rewritten guide. This PR does not depend on Repro dnix packages #1505, but should be rebased and re-checked after Repro dnix packages #1505 lands.

Follow-ups

  • The MWC FFI suite now lives in cypherstack/flutter_libmwc#16; it is no longer part of this PR.
  • Raise Flutter to ^3.47.4 in CI/docs in a separate change. The Nix shell already uses 3.47.4; CI and the Homebrew bootstrap currently use 3.47.2.
  • The flutter_libmwc protoc path fix and a per-library dylib install name remain separate follow-ups.

Supersedes #1336.

Adds a Makefile that wraps scripts/build_app.sh:
- make build-macos runs the same steps as CI (build_app.sh, pub get,
  prebuild.sh, flutter build macos). PREBUILTS=0 compiles native assets
  from source (the generated pubspec.yaml is removed first so an earlier
  -d run does not stick). Produces the same app as upstream
  (Stack Wallet.app, com.cypherstack.stack_wallet).
- make bootstrap-macos (Homebrew; pins Flutter 3.47.2), nix develop via
  flake.nix/.envrc (sets FLUTTER_ROOT so `dart run` works), make
  bootstrap-xcode, and an opt-in make bootstrap-xcode-download
  (archive.org; SHA-1 and Apple signature are both checked).
- make sccache-setup (opt-in): sccache for native-asset hook builds via
  ~/.cargo/config.toml.
- make build-linux; writes a git-ignored .cargo/config.toml with PROTOC so
  crates that ship a generic-Linux protoc build on NixOS.
- bm/ benchmark scripts and MWC integration tests.

Docs: new "Building with make" section in docs/building.md.

Verified (one machine each, one run):
- macOS 27.2 / Xcode 27.1: make build-macos from a clean checkout
  (946 s cold, 319.0 MB), PREBUILTS=0 (596 s), and inside nix develop
  (Flutter 3.47.4, 261 s).
- Linux: NixOS x86_64, nix develop, Rust 1.99.1, make build-linux
  produced build/linux/x64/release/bundle/stack_wallet (291 MB).
- Xcode 27.1 download from archive.org: SHA-1 and pkgutil signature
  check pass (install step not run).
Not tested: Ubuntu, Windows, Android, iOS, bootstrap-macos on a fresh Mac.
…set hooks

Measured on macOS with the unmodified upstream tree: a repeated build takes 33-40 s with an
unchanged environment, but 536 s after one extra directory in PATH and 156-164 s after adding a
NIX_* variable or restoring PATH, because hooks_runner reruns every hook when the environment it
hashes changes.
download_xcode.sh used sudo for rm/mv/xattr unconditionally. It now checks whether the parent of the target
directory is writable and only then uses sudo, so Xcode can be installed into ~/Applications without a password.
Verified on macOS 27.2: -d ~/Applications/Xcode_27.1_test.app installed Xcode 27.1 (27A9275), the extracted app
has no quarantine attribute, and free space dropped by about 4 GB; hash and signature checks passed before it.
The install-time prompt text now says about 20 GB free are needed.
…ptional

install_macos_build_tools.sh aborted at the first brew failure. On a machine where weasyprint was installed another way,
brew could not link it, set -e stopped the script, and the Flutter step was never reached. It also upgraded formulae that
were already installed (cmake, go), changing the build machine under the user.
- export HOMEBREW_NO_INSTALL_UPGRADE=1 for the required packages
- install pandoc/weasyprint/toilet/figlet (only used by the bm/ report rendering) in a separate, non-fatal step

Verified on macOS 27.2: first run failed as described; after the change the installer ended with rc=0, cloned Flutter 3.47.4
into a temporary FLUTTER_DIR, left the existing Flutter 3.47.2 untouched, and only appended the brew shellenv and Flutter PATH
lines to ~/.zshrc (restored afterwards).
…lock

make build-macos now runs ./build_app.sh from the scripts directory (as the CI jobs and the other targets do), so
scripts/build_app.sh stays exactly as in staging; the earlier path-resolution and variable-rename edits are reverted.
make build-linux no longer recreates lib/external_api_keys.dart: prebuild.sh already creates it, and the block never
ran in any of the Linux builds I did.

Verified on macOS 27.2: make build-macos OK, Stack Wallet.app with the unchanged bundle ID. The Linux target was only
checked with make -n after this change.
archive.org sometimes resets the connection (curl exit 56) a minute into the transfer, which aborted the script even
though the download can be resumed. curl now retries up to 8 times (--retry-all-errors) and resumes with -C -.

Seen in a fresh macOS 27.2 VM; the retry itself is exercised in the next VM run.
…e, docs

make build-linux now passes -d (PREBUILTS=1, the default, like make build-macos and the CI job) and -s with SYSTEM_DEPS=1.
With -s the Makefile sets only USE_SYSTEM_SECURE_STORAGE_DEPS=1 for the app build; the local libsecret pkg-config override
is kept for the default path (it is needed where libsecret is built from source, for example on NixOS).
download_xcode.sh accepts XCODE_XIP_URL to download from another mirror; the SHA-1 and the Apple signature are still checked.
docs: a second Xcode via DEVELOPER_DIR, and what reaches the hooks for sccache (exported RUSTC_WRAPPER does not, the cargo
config file does).

Verified: NixOS (nix develop) make build-linux with -d builds the bundle; Ubuntu 24.04 container make build-linux
SYSTEM_DEPS=1 (-d -s, the CI flags) builds the bundle; make -n and bash -n pass for all targets and scripts.
… bm/bench.sh

clean-app removes build/ and the Pods but keeps .dart_tool/hooks_runner, so the next build does not rebuild the
native assets; clean-native removes only the native-asset outputs. bm/bench.sh zeroes the sccache statistics
before the build and stores 'sccache hits/executed requests' in the comment column of the CSV.
stack-bom.toml records the expected toolchain (Flutter, Dart, Rust, Go, Xcode)
and the git revisions of the pinned native plugins. make verify-bom
(dart tool/verify_bom.dart, no packages needed) checks the local machine and
pubspec.lock against it: a mismatch in Rust, Dart, minimum Flutter/Go or a
plugin pin fails, a different recommended Flutter or Xcode only warns.
dart tool/verify_bom.dart --update regenerates the [plugins] section from
pubspec.lock.

Verified on macOS 27.2: passes with 1 warning (Flutter 3.47.2 vs. recommended
3.47.4); a wrong Rust version and a wrong plugin pin both exit 1.
…targets

make build-windows wraps the steps of the build-windows CI job (prebuild.sh, build_app.sh -p windows -d -s,
pub get, coinlib:build_windows, flutter build windows). WINDOWS_COPY_ASSETS=1 replaces the asset links with
copies for runners that cannot create symlinks.
.github/workflows/make-build.yaml runs make build-windows and make build-macos on hosted runners
(manual dispatch or pushes to ci/make-* branches).
…ing key exists

The release build fails at packageRelease without android/key.properties (SigningConfig release is missing
storeFile). android-dev-key creates a debug keystore and key.properties for local builds only; both files are
git-ignored and the APKs are not for distribution.
@parasew parasew changed the title build: Makefile wrapper, macOS/Linux bootstrap, opt-in Xcode download and docs build: Makefile wrappers for macOS/Linux/Windows/Android/iOS, bootstrap, opt-in Xcode download, BOM check and docs Nov 2, 2026
…deps and bm

- build-windows: with WINDOWS_COPY_ASSETS=1 remove the asset copies of an earlier run before
  build_app.sh runs link_assets.sh, whose `rm -f` cannot delete real directories (second run failed)
- bootstrap-macos: skip the Homebrew installer in a Nix shell with one conditional recipe instead of
  `exit 0`, which only ended that line's shell; failures of the Xcode step now propagate
- build_secure_storage_deps.sh: stop when cmake, make, meson or ninja fails, before writing the
  pkg-config file; reconfigure an existing meson build directory instead of failing on it
- install_macos_build_tools.sh: put rustup on PATH after `brew install` (resolved with
  `brew --prefix rustup`), not only if its directory existed before
- install_xcode.sh: keep any valid selected Xcode (Xcode_27.1.app, ~/Applications, ...) and only
  pick one from /Applications when the selection is missing or the Command Line Tools
- mwc_ffi_test_service: the invalid-slatepack assertion no longer sits inside its own catch
- bm/compare.py: pair measurements by label and host instead of the first row of each file
- bm/bench.sh: record failed builds and return make's exit status
- Remove the MWC FFI integration tests; they belong in flutter_libmwc.
- Run prepare_release_bundle.py from make build-linux.
- Honor APP_NAME on macOS and make WINDOWS_COPY_ASSETS copy in link_assets.sh.
- Stop rewriting the tracked libsecret pkg-config file and drop duplicate toolchain steps.
- Harden verify_bom.dart for Windows and missing plugin pins.
- Fix bootstrap/Xcode selection, download resume, and benchmark script edge cases.
Resolve docs/building.md: keep patchelf and staging's removal of libopencv-dev.
@parasew

parasew commented Nov 11, 2026 •

Copy link
Copy Markdown
Author

@julian-CStack thanks — I worked through the review items and the follow-ups.

  • The MWC FFI suite is out of this PR. It lives in cypherstack/flutter_libmwc#16 now: wallet create/open, address validation, slatepack round-trip with a real MWC fixture, encrypted-slatepack wallet requirement, and malformed-input handling. The plugin suite passes locally (9 tests); its CI needs workflow approval because it comes from a fork.
  • Must-fix items are addressed: Linux bundle preparation runs from make build-linux; APP_NAME is honored; Windows copy mode now copies instead of relying on mklink; the tracked libsecret pc file is generated/ignored; the duplicate Docker secp256k1 step is gone; the pub cache is Linux-only; the BOM check, bootstrap/Xcode scripts and benchmark scripts are fixed.
  • Upstream checks on 5d9489551: test pass and linux-release-bundle pass.
  • Fork Make build: Windows and Linux pass. iOS is running. macOS was cancelled twice by the hosted-runner queue before starting; I’ll retry when capacity is available.
  • Compatibility with your Repro dnix packages #1505: a throw-away merge has only a docs/building.md conflict; all make -n targets still parse; make verify-bom needs the plugin pins refreshed after Repro dnix packages #1505 changes pubspec.lock. I did not merge your draft into this PR — it should stay independent until Repro dnix packages #1505 is ready.
  • PR description is updated.

@parasew
parasew marked this pull request as ready for review November 11, 2026 18:16
The fork has a single hosted macOS concurrency slot. Running the two jobs together leaves one queued until GitHub's 15-minute timeout cancels it. Run macOS first, then iOS, so each waits for the runner instead of competing.
@parasew

parasew commented Nov 11, 2026

Copy link
Copy Markdown
Author

Runner follow-up: the macOS cancellations were caused by the fork's single hosted macOS concurrency slot, with macos and ios started together. One job held the slot while the other waited 15 minutes and was cancelled.

Commit 2b0a0e21e serialises them: macOS first, iOS after. A fresh run is queued at https://github.com/parasew/stack_wallet/actions/runs/43081040312; Windows and Linux are already running. As a fallback I also ran make build-macos locally in nix develop (macOS 27.2, Xcode 27.2, Flutter 3.47.4): it passed and produced Stack Wallet.app (318.8 MB). The upstream test check is re-running on the new head.

@parasew

parasew commented Nov 12, 2026

Copy link
Copy Markdown
Author

All checks are green on b82bc47da:

  • Upstream test: pass (24m44s)
  • Upstream linux-release-bundle: pass
  • Fork Make build: Windows, macOS, Linux and iOS all pass

The macOS runner issue is resolved by serialising the two macOS jobs. The PR is up to date with staging and ready for review.

@parasew parasew mentioned this pull request Nov 12, 2026
@parasew

parasew commented Nov 12, 2026

Copy link
Copy Markdown
Author

Clean-room follow-up: the two remaining platform gaps are now actually validated.

macOS VM (Tart, macOS 26.6.2 arm64; Homebrew present, no Xcode/Flutter/Rust):

  • make bootstrap-xcode-download ARGS=-y: real Xcode 27.1 download, SHA-1 and Apple signature verification, extraction and install.
  • make bootstrap-macos: Homebrew packages, Flutter 3.47.2, Rust 1.99.1, PATH/zshrc.
  • make build-macos: Stack Wallet.app 319.5 MB.

Android (x86_64 Linux host, ghcr.io/cypherstack/stackwallet-ci:android):

  • make build-android: arm 183.5 MB, arm64 218.8 MB, x86_64 235.4 MB.

Xcode resume harness: a partial .part was resumed with curl -C -, a relative cache path resolved correctly, and signature/install/-y selection passed.

The remaining limitations are now only physical-Windows hardware, real Android release signing, and iOS simulator/signing. The PR is still green and ready for review.

@parasew
parasew force-pushed the feat/makefile-macos-build branch from b82bc47 to 4481c00 Compare November 12, 2026 07:51
@parasew

parasew commented Nov 12, 2026

Copy link
Copy Markdown
Author

Author metadata update: all commits on this branch are now authored and committed by parasew <9727+parasew@users.noreply.github.com>. The content tree is unchanged from the validated b82bc47da runs; GitHub is re-running the checks on the rewritten head 4481c00d.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant