[Venkumahanti Subhankar (V-Subhankar-infy)]

Fix Podman Feature cache misses with older Buildah

Fixes #1262

Problem and Root Cause

Podman builds using bundled Buildah older than 1.41.1 can reinstall unchanged Features instead of reusing their installation layers, repeating work and downloads. This CLI fallback also covers prerelease, unrecognized, or unavailable Buildah versions. The investigation reproduced misses with Podman 4.9.3 / Buildah 1.33.7 and Podman 5.4.0 / Buildah 1.39.0. Since distributions bundle different Buildah versions, check with podman info --format '{{.Host.BuildahVersion}}' rather than relying on the Podman version alone.

This affects the CLI-generated Feature installation path, not a particular Feature installer. The issue was reproduced with Color, Git, and Common Utilities; other Features using the same path may be affected too.

Two upstream Buildah bugs explain the reported behavior:

• Before Buildah 1.39.0, named additional-context bind mounts can bypass cache lookup entirely (Buildah #5693).
• Buildah 1.39.0 permits lookup, but includes source timestamps in its bind-mount checksum. The CLI regenerates Feature staging files and directories on each invocation, so identical content can still produce different cache keys. The timestamp fix was backported to 1.41.1 and released on mainline in 1.42.0 (Buildah #6311).

Solution

Podman uses its bundled Buildah, so adding a Buildah pin to the CLI manifest or upgrading a separate buildah executable cannot change the builder used by podman build. Buildah before 1.39.0 can bypass named-context cache lookup; 1.39.0 still hashes regenerated source timestamps and can miss cache. The upstream timestamp fix is in 1.41.1 (backport) and 1.42.0 (mainline). Upgrading Podman to include both fixes is valid, but leaves users on older supported runtimes without a solution.

This change checks Podman's bundled Buildah and uses the existing COPY-based Feature path below 1.41.1 or when detection is uncertain. Newer stable versions retain the named-context path. The policy is limited to Feature transport, preserves Docker behavior, and skips detection when --buildkit never already selects COPY.

Impact

Affected Podman users can reuse unchanged Feature installation layers without manually setting --buildkit never. The supplied runtime investigation confirmed that the existing COPY fallback cached unchanged Color Feature builds on Podman 4.9.3 / Buildah 1.33.7 and Podman 5.4.0 / Buildah 1.39.0; a modern fixed builder also cached the default path. Changed inputs and --no-cache still rebuild as expected. COPY may increase image size because Feature source files can remain in layers.

Files Changed

File	Change
src/spec-shutdown/dockerUtils.ts	Add bundled-Buildah detection, version policy, conservative fallback, and diagnostics.
src/spec-node/devContainers.ts	Compute the policy during resolver setup; skip detection for explicit non-BuildKit builds.
src/spec-node/utils.ts	Carry the Feature-only fallback policy in typed resolver parameters.
src/spec-node/containerFeatures.ts	Gate shared Feature transport so source paths, installer instructions, content-image creation, and returned contexts remain consistent.
src/test/dockerUtils.test.ts	Test version boundaries, suffixes, prereleases, malformed responses, failures, command plumbing, and non-Podman behavior.
src/test/container-features/generateFeaturesConfig.test.ts	Test COPY, mount, and no-BuildKit generation with two local Features, including content-image commands, paths, entrypoints, ordering, and fixture cleanup.
README.md	Add a compact notice for CLI 0.90.0 and earlier, recommending a newer CLI and rebuild.
CHANGELOG.md	Record the automatic compatibility fix under Unreleased.
PR_DESCRIPTION.md	PR drafting artifact containing this description; not a runtime or generated file.

No package manifest, lockfile, generated artifact, published Feature installer, or third-party notice update is required. The implementation uses the existing semver dependency.

Validation and Limits

• TypeScript project check passed: npm run type-check.
• Version-policy tests passed: 18 tests via npm test -- --grep 'Podman Feature cache compatibility'.
• Shared-generation tests passed: 3 tests via npm run test-container-features -- --grep 'Feature content transport'.
• ESLint passed for all changed TypeScript files.
• Runtime evidence above comes from the supplied investigation using CLI 0.87.0, not an end-to-end run of this patch. Those tests verified the existing fallback on Podman 4.9.3 / Buildah 1.33.7 and Podman 5.4.0 / Buildah 1.39.0, and the normal path on Podman 5.8.9 / Buildah 1.43.5.
• The exact Buildah 1.41.1 boundary, native Ubuntu/storage-driver matrix, and patched-CLI end-to-end reuse/invalidation checks remain unverified here. Unit tests establish policy and generation behavior, not engine cache hits. The full integration suite and production packaging have not been run for this patch.

[Copilot]

🟡 Changes recommended

The COPY fallback can break Podman no-cache builds by pulling a local-only image, while its shared mutable image tag can cause concurrent builds to run the wrong Feature content.

3 open issues

• Shared mutable feature tag can mix concurrent build contents · New
• COPY fallback still adds --pull to Dockerfile builds · New
• Podman probe errors lose actionable stderr details · New

What changed in this PR

Adds an automatic compatibility fallback for Podman Feature caching (#1262), using the existing COPY transport when bundled Buildah cannot reliably cache named-context mounts.

Changes:

• Detect bundled Buildah and select COPY below 1.41.1 or when version detection is uncertain.
• Carry the Feature-only policy through resolver setup without disabling BuildKit globally or changing Docker behavior.
• Add version-policy and transport-generation tests, plus user-facing caching guidance.

File	Description
src/​test/​dockerUtils.test.ts	Tests Buildah version policy, probe failures, and command plumbing.
src/​test/​container-features/​generateFeaturesConfig.test.ts	Tests consistent COPY, mount, and non-BuildKit Feature generation.
src/​spec-shutdown/​dockerUtils.ts	Adds Buildah detection, fallback selection, and diagnostics.
src/​spec-node/​utils.ts	Adds the typed Feature transport policy field.
src/​spec-node/​devContainers.ts	Resolves the policy and skips probing for explicit non-BuildKit builds.
src/​spec-node/​containerFeatures.ts	Applies the policy to Feature transport selection.
README.md	Adds Podman caching guidance for affected CLI users.
CHANGELOG.md	Records the compatibility fix under Unreleased.

_{Review effort: Max}