From ba7388ae96053a91f5fc535620bd5c3b8c2a51e5 Mon Sep 17 00:00:00 2001 From: Magnus Henoch Date: Tue, 3 Nov 2026 10:58:30 +0000 Subject: [PATCH 1/2] Add python-lambda-zip action Build a zip file for a Python lambda function, and check that the modules can be loaded in the relevant runtime. --- .github/dependabot.yml | 6 ++ README.md | 16 +++- actions/python-lambda-zip/README.md | 78 +++++++++++++++++++ actions/python-lambda-zip/action.yml | 81 ++++++++++++++++++++ actions/python-lambda-zip/build.sh | 109 +++++++++++++++++++++++++++ 5 files changed, 288 insertions(+), 2 deletions(-) create mode 100644 actions/python-lambda-zip/README.md create mode 100644 actions/python-lambda-zip/action.yml create mode 100755 actions/python-lambda-zip/build.sh diff --git a/.github/dependabot.yml b/.github/dependabot.yml index cf876f7..30cde65 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -31,3 +31,9 @@ updates: interval: monthly cooldown: default-days: 7 + - package-ecosystem: "github-actions" + directory: "/actions/python-lambda-zip" + schedule: + interval: monthly + cooldown: + default-days: 7 diff --git a/README.md b/README.md index 8e8141f..21d31a4 100644 --- a/README.md +++ b/README.md @@ -9,9 +9,21 @@ avoid breaking things. If you feel lucky you might use `v0` branch. # actions Number of helpful actions for GitHub Actions -* [build-push-ecr](./actions/build-push-ecr/README.md) +* [build-push-ecr](./actions/build-push-ecr/README.md): + builds and pushes an image to AWS ECR using buildx -* [download-private-asset](./actions/download-private-asset/README.md) +* [download-private-asset](./actions/download-private-asset/README.md): + installs a release asset from a private GitHub repository + +* [python-lambda-zip](./actions/python-lambda-zip/README.md): + builds a zip deployment package for a Python AWS Lambda using runtime-compatible + wheels, and checks that it loads in the AWS Lambda base image + +* [terraform](./actions/terraform/README.md): + Terraform format linting, required-tag checks and tfsec security scanning + +* [upgrade-erlang-elixir](./actions/upgrade-erlang-elixir/README.md): + resolves target Erlang/Elixir versions, updates them in CI files and opens a pull request # reusable workflows Workflows that a caller repo runs with `uses` at job level. Intentionally sit at diff --git a/actions/python-lambda-zip/README.md b/actions/python-lambda-zip/README.md new file mode 100644 index 0000000..d1dbdc3 --- /dev/null +++ b/actions/python-lambda-zip/README.md @@ -0,0 +1,78 @@ +# Python Lambda zip + +Builds a zip-file deployment package for a Python AWS Lambda and checks that it actually loads in +the AWS Lambda runtime. + +Dependencies with compiled extensions (`cryptography`, `numpy`, ...) ship platform-specific wheels. +A plain `pip install` on a GitHub runner picks wheels built for the runner's (new) glibc, which the +Lambda runtime may be too old to load, giving errors like +``version `GLIBC_2.28' not found`` only at import time in AWS. This action + +1. installs the dependencies with `pip --platform ... --only-binary=:all:`, so that only wheels + compatible with the Lambda runtime are used, and +2. imports your modules inside the matching `public.ecr.aws/lambda/python` image, so that any + remaining problem fails the build instead of the deployment. + +The action does not upload the zip anywhere; use the `zip-path` output (or `output-dir`) for that. + +## Usage + +```yaml +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: actions/setup-python@v7 + with: + python-version: '3.11' + - run: pip install pipenv + - uses: gameanalytics/github/actions/python-lambda-zip@v0 + with: + python-version: '3.11' + source-dir: app + requirements-command: pipenv requirements + smoke-test-glob: 'lambda_*.py' +``` + +Run it in your pull request workflow too, not only on releases, to find problems before release. +The runner needs `pip`, `zip`, `unzip` and Docker, which `ubuntu-latest` has. + +## Inputs + +| Input | Default | Description | +|---|---|---| +| `python-version` | (required) | Python version of the Lambda runtime, e.g. `3.11` | +| `architecture` | `x86_64` | `x86_64` or `arm64` (arm64 is smoke tested using QEMU emulation) | +| `source-dir` | | Directory with the Lambda code; its contents end up at the root of the zip | +| `include` | | Newline-separated paths copied into the zip keeping their relative path, e.g. `app` | +| `exclude` | | Newline-separated patterns of files to leave out of the zip, as for `zip -x`, e.g. `boto3/*` | +| `requirements-file` | `requirements.txt` | Fully resolved requirements file | +| `requirements-command` | | Command printing the requirements, e.g. `pipenv requirements`; used instead of the file when set | +| `smoke-test-glob` | `*.py` | Modules (relative to the zip root) to import; build fails if nothing matches | +| `output-dir` | `build` | Where to put the zip | +| `zip-name` | `package.zip` | File name of the zip | + +At least one of `source-dir` and `include` is required. `__pycache__` directories and `.pyc` files are left out of the zip. + +Output: `zip-path`, the absolute path of the zip. + +## How `python-version` maps to wheels and image + +| Python version | Lambda OS | glibc | Wheels installed | Smoke test image | +|---|---|---|---|---| +| 3.9 - 3.11 | Amazon Linux 2 | 2.26 | `manylinux2014` | `public.ecr.aws/lambda/python:` | +| 3.12 and later | Amazon Linux 2023 | 2.34 | `manylinux_2_28` | `public.ecr.aws/lambda/python:` | + +The dependencies are installed with `--no-deps`, so the requirements must already be fully +resolved, as with a lock file. Only pure Python and manylinux wheels are accepted: a package that +has only a source distribution is built from source if it turns out to be pure Python, but if it +is compiled, or only has newer manylinux wheels, the build fails instead of building something +that breaks at runtime. + +pip evaluates environment markers in the requirements (such as `python_version`) against the host +Python, so the host Python must be the same version as `python-version` (the build fails +otherwise): set it up with `actions/setup-python` first. Building `arm64` on an `x86_64` runner +works for the wheels, but platform markers are then evaluated for the runner, so a warning is issued. + +When a new Lambda runtime moves to a newer OS, update the mapping in `build.sh`. diff --git a/actions/python-lambda-zip/action.yml b/actions/python-lambda-zip/action.yml new file mode 100644 index 0000000..3265e67 --- /dev/null +++ b/actions/python-lambda-zip/action.yml @@ -0,0 +1,81 @@ +name: Python Lambda zip +description: > + Builds a zip-file deployment package for a Python AWS Lambda, using wheels compatible with the + Lambda runtime, and checks that it can be imported inside the AWS Lambda base image. + +inputs: + python-version: + description: > + Python version of the Lambda runtime, e.g. '3.11'. Determines the Lambda base image used for + the smoke test and which manylinux wheels are installed. + required: true + architecture: + description: "Lambda architecture: 'x86_64' or 'arm64'" + required: false + default: 'x86_64' + source-dir: + description: 'Directory with the Lambda code. Its contents end up at the root of the zip.' + required: false + include: + description: > + Newline-separated paths (files or directories) to copy into the zip keeping their relative + path, e.g. 'app' to get an 'app' package at the root of the zip. Can be combined with source-dir. + At least one of source-dir and include is required. + required: false + exclude: + description: > + Newline-separated patterns of files to leave out of the zip, as for 'zip -x', matched against + paths in the zip, e.g. 'boto3/*' for dependencies the Lambda runtime provides. The smoke test + runs in the Lambda image, where such packages are available. + required: false + requirements-file: + description: 'Fully resolved requirements file, e.g. generated from a lock file.' + required: false + default: 'requirements.txt' + requirements-command: + description: > + Shell command that prints the requirements to stdout, e.g. 'pipenv requirements'. Used + instead of requirements-file when set. The tool it needs must already be installed. + required: false + smoke-test-glob: + description: > + Glob, relative to the root of the zip, of the modules to import in the smoke test, e.g. + '*.py' or 'app/*.py' (module names starting with an underscore are skipped). The build + fails if nothing matches. + required: false + default: '*.py' + output-dir: + description: 'Directory to put the zip in' + required: false + default: 'build' + zip-name: + description: 'File name of the zip' + required: false + default: 'package.zip' + +outputs: + zip-path: + description: 'Absolute path of the built zip' + value: ${{ steps.build.outputs.zip-path }} + +runs: + using: composite + steps: + - name: Set up QEMU + if: inputs.architecture == 'arm64' + uses: docker/setup-qemu-action@v4.4.0 + - name: Build zip and smoke test it + id: build + shell: bash + env: + PYTHON_VERSION: ${{ inputs.python-version }} + ARCHITECTURE: ${{ inputs.architecture }} + SOURCE_DIR: ${{ inputs.source-dir }} + INCLUDE: ${{ inputs.include }} + EXCLUDE: ${{ inputs.exclude }} + REQUIREMENTS_FILE: ${{ inputs.requirements-file }} + REQUIREMENTS_COMMAND: ${{ inputs.requirements-command }} + SMOKE_TEST_GLOB: ${{ inputs.smoke-test-glob }} + OUTPUT_DIR: ${{ inputs.output-dir }} + ZIP_NAME: ${{ inputs.zip-name }} + run: ${{ github.action_path }}/build.sh diff --git a/actions/python-lambda-zip/build.sh b/actions/python-lambda-zip/build.sh new file mode 100755 index 0000000..2260333 --- /dev/null +++ b/actions/python-lambda-zip/build.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# Builds a Lambda deployment zip for the given Python version and architecture, +# then imports it inside the matching AWS Lambda base image as a smoke test. +# Configured through environment variables, see action.yml. +set -euo pipefail + +: "${PYTHON_VERSION:?}" "${ARCHITECTURE:?}" "${OUTPUT_DIR:?}" "${ZIP_NAME:?}" +: "${SMOKE_TEST_GLOB:?}" + +if [[ -z "${SOURCE_DIR:-}" && -z "${INCLUDE:-}" ]]; then + echo "at least one of source-dir and include must be set" >&2 + exit 1 +fi + +if [[ ! "$PYTHON_VERSION" =~ ^3\.[0-9]+$ ]]; then + echo "python-version must look like '3.11', got '$PYTHON_VERSION'" >&2 + exit 1 +fi +minor="${PYTHON_VERSION#3.}" + +# Lambda runtimes up to python3.11 run on Amazon Linux 2 (glibc 2.26), so only +# manylinux2014 (glibc 2.17) wheels can be loaded. python3.12 and later run on +# Amazon Linux 2023 (glibc 2.34), which can load manylinux_2_28 wheels. +if (( minor <= 11 )); then + manylinux=manylinux2014 +else + manylinux=manylinux_2_28 +fi + +case "$ARCHITECTURE" in + x86_64) docker_platform=linux/amd64 ;; + arm64) docker_platform=linux/arm64 ;; + *) echo "architecture must be x86_64 or arm64, got '$ARCHITECTURE'" >&2; exit 1 ;; +esac +arch=$([[ $ARCHITECTURE == arm64 ]] && echo aarch64 || echo x86_64) +# Also accept manylinux2014 wheels, which pip does not do implicitly for newer tags +wheel_platforms=(--platform "manylinux2014_${arch}") +[[ $manylinux == manylinux2014 ]] || wheel_platforms+=(--platform "${manylinux}_${arch}") +image="public.ecr.aws/lambda/python:${PYTHON_VERSION}" + +# pip evaluates environment markers in the requirements (python_version etc.) against the +# host Python, so a mismatch would silently drop or add dependencies +host_version="$(python3 -c 'import sys; print("%d.%d" % sys.version_info[:2])')" +if [[ "$host_version" != "$PYTHON_VERSION" ]]; then + echo "Host Python is $host_version but python-version is $PYTHON_VERSION: set up the same version first, e.g. with actions/setup-python" >&2 + exit 1 +fi +if [[ "$ARCHITECTURE" != "$(uname -m)" ]]; then + echo "::warning::Building for $ARCHITECTURE on a $(uname -m) host: platform markers in the requirements are evaluated for the host" +fi + +stage="$(mktemp -d)" +trap 'rm -rf "$stage"' EXIT +mkdir -p "$stage/package" "$OUTPUT_DIR" +zip_path="$(cd "$OUTPUT_DIR" && pwd)/$ZIP_NAME" +rm -f "$zip_path" + +if [[ -n "${REQUIREMENTS_COMMAND:-}" ]]; then + bash -c "$REQUIREMENTS_COMMAND" > "$stage/requirements.txt" +else + cp "$REQUIREMENTS_FILE" "$stage/requirements.txt" +fi + +# --no-deps because the requirements are expected to be fully resolved (a lock file). It also +# allows pip to fall back to building a source distribution when a package has no wheel +# for the target; the check below makes sure that this only happens for pure Python packages. +echo "::group::Install dependencies for $manylinux $arch, Python $PYTHON_VERSION" +python3 -m pip install -r "$stage/requirements.txt" --no-deps -t "$stage/package" \ + "${wheel_platforms[@]}" --python-version "$PYTHON_VERSION" \ + --implementation cp +echo "::endgroup::" + +bad=0 +for wheel_file in "$stage"/package/*.dist-info/WHEEL; do + if grep '^Tag: ' "$wheel_file" | grep -qv -e '-any$' -e 'manylinux'; then + echo "::error::$(dirname "$wheel_file" | xargs basename) is not a pure Python or manylinux package (built from source for the host?): $(grep '^Tag: ' "$wheel_file" | tr '\n' ' ')" + bad=1 + fi +done +(( bad == 0 )) + +if [[ -n "${SOURCE_DIR:-}" ]]; then + cp -r "$SOURCE_DIR"/. "$stage/package/" +fi +while IFS= read -r path; do + [[ -n "$path" ]] && cp -r --parents "$path" "$stage/package/" +done <<< "${INCLUDE:-}" +zip_excludes=(-x "*__pycache__*" -x "*.pyc") +while IFS= read -r pattern; do + [[ -n "$pattern" ]] && zip_excludes+=(-x "$pattern") +done <<< "${EXCLUDE:-}" +(cd "$stage/package" && zip -qr "$zip_path" . "${zip_excludes[@]}") +echo "Built $zip_path" + +mkdir -p "$stage/smoke" +unzip -q "$zip_path" -d "$stage/smoke" +echo "::group::Smoke test in $image ($docker_platform)" +docker run --rm --platform "$docker_platform" --entrypoint python \ + -v "$stage/smoke:/var/task:ro" -w /var/task -e GLOB="$SMOKE_TEST_GLOB" "$image" \ + -c "import glob, importlib, os +files = glob.glob(os.environ['GLOB'], recursive=True) +mods = sorted(f[:-3].replace('/', '.') for f in files if not os.path.basename(f).startswith('_')) +assert mods, 'no modules matched ' + os.environ['GLOB'] +for m in mods: + importlib.import_module(m) + print('imported', m)" +echo "::endgroup::" + +echo "zip-path=$zip_path" >> "${GITHUB_OUTPUT:-/dev/null}" From 050b6bb0b6e9cf8fa1065c2061b53cd4161307d2 Mon Sep 17 00:00:00 2001 From: Magnus Henoch Date: Tue, 17 Nov 2026 12:50:53 +0000 Subject: [PATCH 2/2] architecture argument is now required Should be less confusing than having a default that doesn't always apply. --- actions/python-lambda-zip/README.md | 2 +- actions/python-lambda-zip/action.yml | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/actions/python-lambda-zip/README.md b/actions/python-lambda-zip/README.md index d1dbdc3..b079c3b 100644 --- a/actions/python-lambda-zip/README.md +++ b/actions/python-lambda-zip/README.md @@ -43,7 +43,7 @@ The runner needs `pip`, `zip`, `unzip` and Docker, which `ubuntu-latest` has. | Input | Default | Description | |---|---|---| | `python-version` | (required) | Python version of the Lambda runtime, e.g. `3.11` | -| `architecture` | `x86_64` | `x86_64` or `arm64` (arm64 is smoke tested using QEMU emulation) | +| `architecture` | (required) | `x86_64` or `arm64` (arm64 is smoke tested using QEMU emulation) | | `source-dir` | | Directory with the Lambda code; its contents end up at the root of the zip | | `include` | | Newline-separated paths copied into the zip keeping their relative path, e.g. `app` | | `exclude` | | Newline-separated patterns of files to leave out of the zip, as for `zip -x`, e.g. `boto3/*` | diff --git a/actions/python-lambda-zip/action.yml b/actions/python-lambda-zip/action.yml index 3265e67..0eef49a 100644 --- a/actions/python-lambda-zip/action.yml +++ b/actions/python-lambda-zip/action.yml @@ -11,8 +11,7 @@ inputs: required: true architecture: description: "Lambda architecture: 'x86_64' or 'arm64'" - required: false - default: 'x86_64' + required: true source-dir: description: 'Directory with the Lambda code. Its contents end up at the root of the zip.' required: false