feat: align deprecated bump-upstream with the org bump engine

README.md

@@ -76,38 +76,26 @@ npx @dappnode/dappnodesdk <command>
 
 ## Github Actions
 
-### `bump-upstream` action
+### `bump-upstream` action (deprecated)
 
-This Github Action automatically creates pull requests bumping the version to the latest released upstream version on DAppNode Packages. It then tests building the new bumped upstream version and automatically posts an IPFS hash if the build was successful so that the new version can be tested further with other Github Actions and by human testers.
+`dappnodesdk github-action bump-upstream` is deprecated and will be removed in a future release. DAppNode org packages should call the reusable workflow `dappnode/workflows/.github/workflows/bump-upstream.yml@master`. Other packages should use the `dappnode/workflows/actions/bump-upstream@master` action, which edits the files, followed by their own PR step.
 
-_Note: This action also requires a Pinata account with an API Access Token and a Github Personal Access Token stored in the repo's, individual's, or organization's secrets in order to work properly._
-
-#### Sample Usage
-
-`.github/workflows/auto_check.yml`
+Replace `.github/workflows/auto_check.yml` with:
 
 ```yaml
 name: Bump upstream version
-
 on:
   schedule:
     - cron: "00 */4 * * *"
-  push:
-    branches:
-      - "master"
-
+  workflow_dispatch:
 jobs:
-  build:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - run: npx @dappnode/dappnodesdk github-action bump-upstream
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          PINATA_API_KEY: ${{ secrets.PINATA_API_KEY }}
-          PINATA_SECRET_API_KEY: ${{ secrets.PINATA_SECRET_API_KEY }}
+  bump-upstream:
+    uses: dappnode/workflows/.github/workflows/bump-upstream.yml@master
+    secrets: inherit
 ```
 
+The deprecated command continues to create bump PRs and run package builds until its removal. It preserves Compose comments and digest pins, reads optional upstream `provider` and `tagPattern` metadata, and automatically detects `package_variants/`. `--use_variants` is accepted but ignored; `--variants_dir` supports legacy directory layouts. Use `--upstream_version` to request an explicit tag for a single upstream.
+
 You must specify in the package manifest what upstream Github repo the package is tracking. Also indicate what build variable in the docker-compose file should be updated.
 
 `dappnode_package.json`

doc/package-spec.md

@@ -29,7 +29,7 @@ Repository counts and migration observations belong to dated audit reports; see
 
 **SPEC-M04: The upstream `arg` is a compose build arg.** Every upstream entry names an `arg` that some compose service declares in `build.args`. Packages using the legacy fields often rely on the implicit default `UPSTREAM_VERSION`.
 
-**SPEC-M05: The upstream `version` equals every matching build arg value.** Every compose service declaring the upstream `arg` must use the same string as `upstream[].version`. Matching one service does not excuse a conflicting value in another. A value `<version>@sha256:<digest>` also matches when the digest contains exactly 64 hexadecimal characters. Findings identify the conflicting service and the compose fragment that supplies its value.
+**SPEC-M05: The upstream `version` equals every matching build arg value.** Every compose service declaring the upstream `arg` must use the same string as `upstream[].version`. Matching one service does not excuse a conflicting value in another. A value `<version>@sha256:<digest>` also matches when the digest contains exactly 64 lowercase hexadecimal characters. Findings identify the conflicting service and the compose fragment that supplies its value.
 
 **SPEC-M06: A variant `upstream[]` keeps the root's positions.** Root and variant manifests are merged by array position. A variant may list fewer or more entries than the root, but an entry at an existing position must refer to the same `repo`.
 
@@ -43,6 +43,8 @@ Repository counts and migration observations belong to dated audit reports; see
 
 **SPEC-M11: Global environment names match the runtime catalogue.** Entries in `globalEnvs[].envs` use short names from the `GLOBAL_ENVS` catalogue bundled with `@dappnode/types`, for example `PUBLIC_IP`, `HOSTNAME` or `ENVS_ACTIVE`. DAPPMANAGER adds `_DAPPNODE_GLOBAL_` at install, so `_DAPPNODE_GLOBAL_PUBLIC_IP` in the manifest would be prefixed twice. Catalogue keys are not aliases: the key `ACTIVE` maps to `ENVS_ACTIVE`, which is the valid manifest value. Catalogue updates must be released through shared types and consumed by the SDK before packages adopt newly supported names.
 
+**SPEC-M12: Upstream bump metadata is valid.** Each `upstream[]` entry may set `provider` to `docker` (the default), `github`, or `docker_tags`, case-insensitively. `docker-tags` and `dockerhub` are aliases for `docker_tags`; legacy manifests use `upstreamProvider`. An optional `tagPattern` filters upstream tags and extracts capture group 1, or the whole match when no capture is present. Patterns must compile with JavaScript `RegExp` and use the common JavaScript/Rust regex subset: no lookaround or backreferences. Unknown providers and invalid patterns are reported as warnings.
+
 ### Manifest semantics
 
 The source manifest describes package metadata and installation instructions. A template root and its variant manifests are source fragments; the merged result is the installable manifest and is validated against the full shared schema. Root fragments need not repeat each variant’s required identity fields.
@@ -118,7 +120,7 @@ These are not checked yet, either because they need consumer changes first or be
 - **Upstream identity and inheritance:** define duplicate-entry behavior, a stable identity for overrides, and any migration away from positional array merging before changing either consumer.
 - **Compose `version:`** is obsolete. Rather than warn, the SDK should strip it from releases by default once versionless composes are verified with DAPPMANAGER (`--strip_compose_version` exists today as an opt-in).
 - **Dockerfiles:** no `FROM …:latest` or untagged base images, and no remote `ADD` without `--checksum`. The staking `ADD ${STAKER_SCRIPTS_URL}/*.sh` pattern needs a shared replacement first.
-- **Declarative upstream metadata:** optional `upstream[].provider`, `tagPattern` and `imageTag` fields would let tropibot drop its built-in provider table and tag-to-image-tag inference. This needs a manifest schema change.
+- **Declarative upstream metadata:** `upstream[].provider` and `tagPattern` are now checked by SPEC-M12. Publishing the shared schema/types support and defining an optional image override would allow more package-specific bump configuration to move into manifests.
 - **ENS names of new variants:** `<base>-<variant>.<dnp|public>.dappnode.eth` for new variants, with an allowlist of existing on-chain names (`hoodi-geth`, `nethermind-xdai`, …).
 - **Explicit `architectures`**, replacing reliance on the implicit `linux/amd64`.
 - **Setup wizard targets:** every wizard target refers to a service and environment variable that exist in the compose (service/environment existence should be checked after merging).
@@ -138,24 +140,24 @@ These came up while defining the rules and should be fixed regardless of the rol
 
 tropibot and the SDK share most package handling. Once a rule is an error, tropibot can drop the matching tolerance:
 
-| Rule | tropibot code that becomes removable |
+| Rule | TropiBot tolerance that becomes removable |
 |---|---|
 | SPEC-M01, SPEC-C01 | Hard-coded name assumptions in discovery, pin cleanup, on-chain sync and dashboard artifacts become guarantees. |
-| SPEC-M03 | The legacy `upstreamRepo/Version/Arg` path in bump discovery and edits (`bump/package/discovery.rs`, `edits.rs`); dashboards keep it only for history. |
-| SPEC-M04, SPEC-C05 | Upstream arg detection heuristics and the `UPSTREAM_VERSION` default (`discovery.rs`, `compose.rs`), list-style arg parsing and string `build:` handling. |
-| SPEC-M05 | Part of the tag/arg format reconciliation in `resolve.rs` and `version_format.rs`; external tag formats (`n8n@x`, `v` prefixes) remain. |
-| SPEC-M06 | The workaround for variants that re-declare `upstream` (`package.rs`). |
-| SPEC-T01 | Phantom root packages in dashboard manifest merging (`source/manifests.rs`) and root-only pin cleanup. |
-| SPEC-T02 | The `variants/` vs `package_variants/` dual layout (`bump/package/types.rs`, `discovery.rs`). |
+| SPEC-M03 | Legacy upstream-field discovery and editing; dashboards keep the fields only for history. |
+| SPEC-M04, SPEC-C05 | Upstream arg detection heuristics, the `UPSTREAM_VERSION` default, list-style arg parsing and string `build:` handling. |
+| SPEC-M05 | Some version/build-arg reconciliation; external tag formats (`n8n@x`, `v` prefixes) remain. |
+| SPEC-M06 | The workaround for variants that re-declare `upstream`. |
+| SPEC-T01 | Phantom root packages in dashboard manifest merging and root-only pin cleanup. |
+| SPEC-T02 | The `variants/` vs `package_variants/` dual layout. |
 | SPEC-M02, SPEC-T03 | Divergent name and variant regexes across modules collapse into one. |
 
 Issues in tropibot independent of the spec:
 
 - Pin cleanup on branch close or delete uses only the root manifest `name`, so it errors for templates whose root has no name and misses variant packages. The SDK now cleans root and variant names (`cleanPinsFromDeletedBranches`); tropibot should do the same or leave cleanup to the SDK.
 - Pinata `pinList` is not paginated (default page size 10), so cleanup sees only the first page.
 - tropibot unpins when a PR closes even if its branch still exists; the SDK only cleans deleted branches.
-- `docs/PACKAGE_CREATION_GUIDE.md` recommends a `variants/<v>/` layout with per-variant Dockerfiles, which contradicts SPEC-T02 and every real template.
-- Two upstream bumpers exist: tropibot and the SDK's `github-action bump-upstream`. One should become canonical.
+- Older package-creation guidance recommends a `variants/<v>/` layout with per-variant Dockerfiles, which contradicts SPEC-T02.
+- The SDK’s `github-action bump-upstream` is deprecated and will be removed in a future release. DAppNode org packages should call `dappnode/workflows/.github/workflows/bump-upstream.yml@master`. Other packages should use `dappnode/workflows/actions/bump-upstream@master` for file edits, followed by their own PR step. See the migration example in [README.md](../README.md#bump-upstream-action-deprecated).
 
 Contract between the SDK and tropibot that the spec work must keep:
 

package.json

@@ -60,8 +60,9 @@
     "source-map-support": "^0.5.19",
     "tar-fs": "^3.1.3",
     "timestring": "^5.0.1",
-    "yargs": "^17.7.1",
-    "wrap-ansi": "^9.0.2"
+    "wrap-ansi": "^9.0.2",
+    "yaml": "^2.8.1",
+    "yargs": "^17.7.1"
   },
   "devDependencies": {
     "@eslint/js": "^10.0.1",

src/bump/discovery.ts

@@ -0,0 +1,166 @@
+import fs from "fs";
+import path from "path";
+import { Manifest } from "@dappnode/types";
+import {
+  defaultManifestFileName,
+  defaultVariantsDirName,
+  UPSTREAM_VERSION_VARNAME
+} from "../params.js";
+import { arrIsUnique } from "../utils/array.js";
+import { getAllVariantsInPath } from "../files/variants/getAllPackageVariants.js";
+import {
+  BumpUpstreamItem,
+  compileTagPattern,
+  parseProvider,
+  UpstreamProvider
+} from "./metadata.js";
+
+export interface Upstream {
+  repo: string;
+  version: string;
+  arg: string;
+  provider: UpstreamProvider;
+  tagPattern?: string;
+}
+
+/**
+ * The part of a compose file that upstream bumps read. Unlike the shared
+ * `Compose` type, build args may use list syntax (`- ARG=value`).
+ */
+export interface BumpCompose {
+  services: Record<string, BuildService>;
+}
+
+export interface BuildService {
+  build?:
+    | string
+    | {
+        context?: string;
+        dockerfile?: string;
+        args?: Record<string, unknown> | string[];
+      };
+}
+
+export type BumpManifest = Partial<Manifest> & { upstreamProvider?: string };
+
+type UpstreamEntry = Partial<BumpUpstreamItem> & { repo: string };
+
+/**
+ * Returns the build args of a compose service as [name, value] pairs, for both
+ * mapping and list syntax. A list arg without `=` has a null value.
+ */
+export function getBuildArgs(service: BuildService): [string, unknown][] {
+  if (typeof service.build !== "object" || !service.build.args) return [];
+
+  const args = service.build.args;
+  if (!Array.isArray(args)) return Object.entries(args);
+
+  return args.map(arg => {
+    const separator = arg.indexOf("=");
+    if (separator < 0) return [arg.trim(), null];
+    return [arg.slice(0, separator).trim(), arg.slice(separator + 1)];
+  });
+}
+
+/**
+ * Lists the upstreams declared in the manifest, from `upstream[]` or from the
+ * legacy `upstreamRepo`, `upstreamVersion` and `upstreamArg` fields.
+ */
+export function discoverUpstreams(
+  manifest: BumpManifest,
+  compose: BumpCompose
+): Upstream[] {
+  const entries = getUpstreamEntries(manifest);
+  const versionArgs = getVersionArgs(compose);
+  const upstreams = entries.map(entry =>
+    parseUpstream(entry, entries.length > 1, versionArgs)
+  );
+
+  if (!arrIsUnique(upstreams.map(({ repo }) => repo)))
+    throw Error("Upstream repositories must be unique");
+  if (!arrIsUnique(upstreams.map(({ arg }) => arg)))
+    throw Error("Upstream args must be unique");
+
+  return upstreams;
+}
+
+/**
+ * Returns the directories whose manifest version a bump updates: each variant
+ * in `variantsDirName`, or the package root when the package has no variants.
+ */
+export function getPackageManifestDirs(
+  dir: string,
+  variantsDirName = defaultVariantsDirName
+): string[] {
+  const variantsPath = path.resolve(dir, variantsDirName);
+  if (!fs.existsSync(variantsPath)) return [dir];
+
+  const variantDirs = getAllVariantsInPath(variantsPath).map(variant =>
+    path.join(variantsPath, variant)
+  );
+  if (variantDirs.length === 0) return [dir];
+
+  for (const variantDir of variantDirs) {
+    if (!fs.existsSync(path.join(variantDir, defaultManifestFileName)))
+      throw Error(`Variant has no manifest: ${variantDir}`);
+  }
+
+  return variantDirs;
+}
+
+function getUpstreamEntries(manifest: BumpManifest): UpstreamEntry[] {
+  if (manifest.upstream) return manifest.upstream;
+  if (!manifest.upstreamRepo) return [];
+
+  return [
+    {
+      repo: manifest.upstreamRepo,
+      version: manifest.upstreamVersion,
+      arg: manifest.upstreamArg,
+      provider: manifest.upstreamProvider
+    }
+  ];
+}
+
+/**
+ * Compose build args that look like versions. They identify the arg of a
+ * single upstream that does not name one.
+ */
+function getVersionArgs(compose: BumpCompose): string[] {
+  const argNames = Object.values(compose.services).flatMap(service =>
+    getBuildArgs(service).map(([name]) => name)
+  );
+  return [...new Set(argNames)].filter(name => /VERSION|TAG/.test(name));
+}
+
+function parseUpstream(
+  entry: UpstreamEntry,
+  isMultiUpstream: boolean,
+  versionArgs: string[]
+): Upstream {
+  if (!entry.repo || !entry.version)
+    throw Error("Upstream repo and version are required");
+
+  compileTagPattern(entry.tagPattern);
+
+  return {
+    ...entry,
+    version: entry.version,
+    arg: entry.arg || inferUpstreamArg(isMultiUpstream, versionArgs),
+    provider: parseProvider(entry.provider)
+  };
+}
+
+function inferUpstreamArg(
+  isMultiUpstream: boolean,
+  versionArgs: string[]
+): string {
+  if (isMultiUpstream)
+    throw Error("Every entry in a multi-upstream manifest must name an arg");
+  if (versionArgs.length > 1)
+    throw Error(
+      `Cannot uniquely detect upstream arg: ${versionArgs.join(", ")}`
+    );
+
+  return versionArgs[0] || UPSTREAM_VERSION_VARNAME;
+}