feat: Add flag overrides to the contract test services by kinyoklion · Pull Request #530 · launchdarkly/python-server-sdk

kinyoklion · 2026-09-25T22:34:50Z

Summary

This is the last step of the flag overrides port described by the OVERRIDE specification. It is based on the file source branch because the phases are stacked; retarget to feat/overrides once that branch merges.

Both contract test services (the Flask service for LDClient and the aiohttp service for AsyncLDClient) advertise the flag-overrides capability and accept the overrides object in the SDK configuration as the test harness service specification describes: filePaths (required, in precedence order) and the optional duplicateKeysHandling, changeDetection, and pollIntervalMs properties. The values configure a FileOverrideSourceBuilder on the FDv2 data system through ConfigBuilder.overrides(...) or AsyncDataSystemConfig.override_source. An overrides object without a dataSystem configuration is rejected with an error, because overrides are an option of the FDv2 data system only.

Verification with the prebuilt harness from sdk-test-harness branch rlamb/sdk-2657/flag-overrides-tests (v3.2.0-alpha.10):

Sync service: -run 'flag overrides' 38 ran, all passed; full suite 4809 total, 15 skipped, 4794 ran, all passed.
Async service: -run 'flag overrides' 38 ran, all passed; full suite 4807 total, 16 skipped, 4791 ran, all passed.

The 15 tests skipped by the overrides runs are the other top-level suites excluded by the filter. The tests skipped in the full runs are those for capabilities the services do not advertise (strongly-typed, tls:verify-peer, service-endpoints, user-type, track-hooks, wrapper, and on the async service hook-environment-id), the long-running tests, and the persistent data store tests, all of which were skipped before this change too.

SDK-3250

Note

Overview
Both contract-test services (sync Flask and async aiohttp) now advertise the flag-overrides capability and map harness overrides config into the FDv2 data system.

When dataSystem is present, an optional overrides block builds a FileOverrideSourceBuilder from filePaths and optional duplicateKeysHandling, changeDetection, and pollIntervalMs—wired via datasystem.overrides(...) on the sync client and AsyncDataSystemConfig.override_source on the async client. overrides without dataSystem raises ValueError, since overrides are FDv2-only.

No production SDK behavior changes; this is harness wiring so sdk-test-harness flag-overrides suites can run against Python.

^{Reviewed by Cursor Bugbot for commit dac6abe. Bugbot is set up for automated code reviews on this repo. Configure here.}

kinyoklion · 2026-10-03T01:23:46Z

bugbot review

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit dac6abe. Configure here.}

…thout the init key (#539)  ## Summary Overrides only add to what the SDK reports: a flag that the override layer does not hold evaluates as it does without an override source, and the all-flags state holds every flag it holds without one, plus the overridden flags. A client that has not initialized is where SDKs broke this, and only when the store holds flags without being initialized, for example a persistent store that holds data but no init key. The flag overrides suite uses the SDK's in-memory store, which is empty until a data source provides data, so it could not tell a read of the override layer apart from a read of everything the store returns. A new `flag overrides` group in the persistent data store suite (`server_side_persistence_overrides.go`) runs when the test service has the `flag-overrides` capability. It writes flags to the store without the init key and gives the client a data source that never provides data. Each test compares a client with an override source to a client without one against the same store, so it holds whatever an SDK reports for an uninitialized store: - **a flag without an override evaluates as it does without an override source**: an override file that holds a different flag leaves the evaluation of a store flag unchanged (OVERRIDE Requirements 1.5.1 and 1.5.3). - **an empty override source leaves the all-flags state unchanged** (Requirement 1.2.2). - **overrides add to the all-flags state and make it valid**: the state holds exactly the flags of the client without a source, with override values for the overridden keys, plus the flags that exist only in the override layer, and it is valid (launchdarkly/sdk-specs#280, Requirements 1.4.1.2 and 1.4.1.3). The flag overrides suite gains **evaluate all flags returns only the override flags as a valid state** in its uninitialized client group, where the store is empty and every SDK reports only the override flags. ## Results Run with `-enable-persistence-tests` against Redis. | Test service | flag without an override | empty override source | overrides add | |---|---|---|---| | Go `v7` | fails: served from the store | fails: valid state of the store's flags | fails: the store's flags join the overrides | | Go launchdarkly/go-server-sdk#491 | passes | passes | passes | | Node `main` | passes | fails: valid state of the store's flags | passes | | Node launchdarkly/js-core#2132 first commit (`57498a9e`) | passes | passes | fails: drops a store flag the state holds without overrides | | Node launchdarkly/js-core#2132 | passes | passes | passes | | Python (top of the launchdarkly/python-server-sdk#530 stack, sync and async) | passes | passes | passes | | Ruby (top of the launchdarkly/ruby-server-sdk#454 stack) | passes | passes | passes | The Java (launchdarkly/java-core#225) and .NET (`dotnet-server-sdk-v9`) test services do not support persistent stores, so the group skips there; the new uninitialized client test passes on both. The full Redis persistent data store suite and the flag overrides suite pass against Go #491 at `30d4f216` (131 tests run). `golangci-lint` reports no issues. Go and Node fail these tests until launchdarkly/go-server-sdk#491 and launchdarkly/js-core#2132 are merged, for test services whose CI runs the persistent data store suite with this harness version. [SDK-3419](https://launchdarkly.atlassian.net/browse/SDK-3419)   Opened from termag [SDK-3419]: https://launchdarkly.atlassian.net/browse/SDK-3419?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ  --- > [!NOTE] > **Overview** > Adds harness coverage for **file flag overrides** when the SDK never initializes but a **persistent store already holds feature data without the `$inited` key**—a case the existing in-memory override suite could not distinguish from “read everything from the store.” > > A new **`flag overrides`** group under the server-side persistent data store suite (`server_side_persistence_overrides.go`, gated on `flag-overrides`) seeds Redis/Consul/Dynamo with flags, uses a blocking unavailable data source, and compares clients with and without override files via `uninitializedClient`. Scenarios assert: store flags not in the override file behave like no override source; an empty override file leaves **evaluate all flags** unchanged; overrides merge into all-flags state, mark it **valid**, and apply override values while retaining other store flags. > > The standalone flag-overrides **`uninitialized client`** group gains **evaluate all flags returns only the override flags as a valid state** (empty store, overrides-only valid state). The persistent suite registers the new group from `server_side_persistence_base.go`. > > Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 09a8e02. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).

Carries the override-affected marking on the evaluation event input, so the event processor keys on the marking alone and never reads the evaluation reason. An evaluation marked as override-affected produces no individual feature event and no debug event, even when the flag requests them, and it is counted in summary events like any other evaluation. The marking is part of the summary counter key, so override-affected and ordinary evaluations of the same flag, variation, and version accumulate into separate counters, and a counter that aggregates marked evaluations carries the overrideAffected marker, present only when true, like the unknown marker. The evaluator passes each prerequisite record's own marking to its event, so a marked prerequisite record produces no individual event while an unaffected prerequisite inside a marked evaluation is recorded as usual. The client passes the top-level marking to the evaluation event, keeps the marking on the reason when a migration evaluation replaces it with a wrong-type error, and presents an override-affected flag in the all-flags state with trackEvents and trackReason false and no debugEventsUntilDate, so a consumer bootstrapped from the state sends no individual events for it. The async client mirrors these changes. The OVERRIDE specification vector runner now also checks the marking each evaluation contributes to its summary counter.

…ons after the loop closes When an evaluation raises and the flag definition came from the override layer, the EXCEPTION reason carries the override indicator, the default event record is marked so no individual event is produced, and the all-flags state presents the flag with tracking off. The async data system drops a flag change notification that arrives after the event loop closed instead of raising on the reload thread.

… async client

…ange now covers

Adds ldclient.integrations.overrides.FileOverrideSourceBuilder, the file-based override source described by the OVERRIDE specification. The source reads one or more JSON or YAML files in the file data source document format, with optional flags, flagValues, and segments members, and supplies each successful load to the SDK's override sink as a full snapshot. Files are combined in the configured order. The duplicate keys handling is fail by default, which rejects the reload and keeps the previously loaded overrides, or ignore, which keeps the first file's entry. A configured file that does not exist contributes no overrides, so a file can be created later and deleting a file removes its overrides. A file that exists but cannot be read or parsed fails that reload, the last good overrides stay in effect, the failure is logged, and the load is retried after a bounded delay and on the next detected change. Change detection is one of two modes: polling, the default, examines the files once per second by default with a one second minimum, and watching reacts to file system notifications through the watchdog package. Watching without the watchdog package and a builder with no paths are construction errors. The initial load completes during client construction. Every applied change is logged at Info level with the overrides in effect and what each configured file supplied.

…l of the file override source Change detection is set up before the initial load, so an edit made while the files are first read is picked up. Close stops the reloader even when the change detector fails to close. The builder rejects a poll interval that is not a number or not finite.

…ntries as duplicate keys A path listed more than once would make every entry of that file a duplicate of itself: under the default fail handling the load failed, and under ignore the first listing won by accident. The builder now drops the repeats, keeps the first occurrence in its place in the order, and logs a warning once at build time. This is the behavior the Node SDK has.

…t on Windows The builder dropped repeated paths by exact string match, so on Windows a path listed twice in different case survived as two paths, and every key of that file failed as a duplicate of itself under the default fail handling. Paths are now compared with os.path.normcase, which ignores case on Windows and compares exactly elsewhere. The first spelling keeps its place in the order.

… SDKs A flagValues entry becomes a flag without client-side availability, so an all-flags state for client-side SDKs leaves it out even when the flag it overrides is client-side. A full definition with clientSide is the way to override such a flag. A test pins both.

The override source keeps the overrides in effect while it finds a file empty during an in-place rewrite. A YAML file without the pyyaml package is now a configuration error when the source is built, since the file name chooses the format and such a file can never load. The builder docs say that the file name decides the format, that files must be UTF-8, that files can be rewritten in place, how to remove a file's overrides, that an empty JSON file is not a valid document, and that a YAML file is best replaced in one operation because a partly written one can still parse. End-to-end tests rewrite the files in place, slowly, in watching and polling modes, for JSON and YAML, and check that neither the source nor a client evaluating the flag ever goes without the override.

Watching follows edits to the configured files and to the targets of configured links, including a ConfigMap update, but not a link further along the path or a configured directory that is pointed elsewhere or replaced. The change detection documentation now says so and points such deployments to polling.

Both contract test services advertise the flag-overrides capability and accept the overrides object in the SDK configuration: filePaths, and the optional duplicateKeysHandling, changeDetection, and pollIntervalMs properties, which configure the file-based override source on the FDv2 data system. An overrides object without a data system configuration is rejected, because overrides are an option of the FDv2 data system only.

**Requirements** - [x] I have added test coverage for new or changed functionality - [x] I have followed the repository's [pull request submission guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests) - [ ] I have validated my changes against all supported platform versions - [x] Sync/async parity: matching changes made to `async_*` siblings (or N/A). N/A: `LDClient` and `AsyncLDClient` use the same `FileOverrideSourceBuilder`. **Related issues** - Bugbot finding on #529: #529 (comment) **Describe the solution you've provided** `FileOverrideSourceBuilder` accepted one path as a `str`, or a list of paths. A single `pathlib.Path`, or any other `os.PathLike`, was passed to `list()`. A path object is not iterable, so the constructor raised `TypeError: 'PosixPath' object is not iterable`. The Bugbot finding describes a different failure, a path split into its components that fails every reload, but that cannot happen. The constructor raises before a client exists. The constructor now treats a `str` or an `os.PathLike` as a single path, and converts every path with `os.fspath`. A path that does not name a string path raises `TypeError` from the constructor, with a message that names the value. That includes `bytes` (given alone, in a list, or named by a path object), an `int`, and `None`. `bytes` and `bytearray` count as single values, so a bare `bytes` path is named whole instead of being read as a sequence of integers. The type hint is `Sequence` rather than `List`, so that type checkers accept a list of path objects but not a set, whose order is not fixed. The docstring says that paths may be path objects. Tests cover: - a single path object; - a list of path objects, which already worked; - a path object whose string form differs from its path, as with `os.DirEntry`; - a source that loads a file given as a path object; - the rejection of each of the values above. Each of six broken variants of the constructor fails at least one of these tests, including the original code and a variant that converts with `str()` instead of `os.fspath`. **Describe alternatives you've considered** Leaving the builder to accept only strings, as its type hint said. `pathlib` is the usual way to build paths, a list of path objects already worked, and the error did not say what was wrong. **Additional context** This PR is stacked on #530, the top of the overrides stack, so that the approved PRs below it do not change. Retarget it as the PRs below it merge. The review of this change found three existing gaps that this PR leaves alone: - An empty path resolves to the working directory, and its reads keep failing. - In polling mode, a path that contains a NUL character makes the source fail to start, so the client runs without overrides. In watching mode, the reload fails and keeps retrying. - The file data sources treat a tuple of paths differently from a list. Testing: `mypy ldclient`, `isort --check` and `pycodestyle` pass locally. The test suite passes (2,111 tests). It was run without the Redis, DynamoDB and Consul store integration tests, which need servers that were not running. CI runs only for PRs based on `main` or `feat/**`, so it will not run on this PR until it is retargeted.   Opened from termag  --- > [!NOTE] > **Overview** > Fixes a constructor bug where a **single** `pathlib.Path` (or other `os.PathLike`) was passed to `list()`, causing `TypeError: 'PosixPath' object is not iterable`. > > `FileOverrideSourceBuilder` now wraps a lone `str`, `bytes`/`bytearray`, or `PathLike` as one path, normalizes every entry with `os.fspath()` (so objects like `os.DirEntry` use their filesystem path, not `str()`), and stores string paths. Invalid inputs (`bytes` paths, ints, `None`, path objects that do not resolve to a `str`) fail fast with an explicit `TypeError`. Type hints use `Sequence` plus `PathLike` so callers can pass path objects while unordered collections like `set` are discouraged. > > New tests cover single/list path objects, `__fspath__` behavior, invalid types, and loading overrides from a path object. > > Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 154b5b8. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 021778c to 2074b74 Compare September 28, 2026 20:29

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from b5ff7c2 to 0a9c386 Compare September 28, 2026 20:29

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 2074b74 to 1950d45 Compare September 30, 2026 20:31

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from 0a9c386 to bad30c5 Compare September 30, 2026 20:31

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 1950d45 to 3e47045 Compare October 1, 2026 23:43

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from bad30c5 to a355c3d Compare October 1, 2026 23:43

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 3e47045 to cfb781a Compare October 3, 2026 01:18

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from a355c3d to dac6abe Compare October 3, 2026 01:19

cursor Bot reviewed Oct 3, 2026

View reviewed changes

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from cfb781a to ea4dc19 Compare October 13, 2026 21:02

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from dac6abe to 3de3ec9 Compare October 13, 2026 21:02

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from ea4dc19 to 3de11ab Compare October 13, 2026 22:40

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch 2 times, most recently from 8d2c06c to 6190743 Compare October 15, 2026 22:25

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from bb5a108 to 5e42ebc Compare October 16, 2026 21:34

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from 6190743 to ac6bdd0 Compare October 16, 2026 21:34

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 5e42ebc to 314dea2 Compare October 28, 2026 22:17

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from ac6bdd0 to 0d759ba Compare October 28, 2026 22:17

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 314dea2 to 4bace41 Compare October 28, 2026 22:40

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch 2 times, most recently from 1dbbf2c to 1f4ce1a Compare October 28, 2026 22:53

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 99b851f to fac9ec8 Compare October 29, 2026 16:48

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from 1f4ce1a to d1b617b Compare October 29, 2026 16:48

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from fac9ec8 to 0415ec1 Compare October 29, 2026 17:16

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from d1b617b to 0d55468 Compare October 29, 2026 17:16

kinyoklion mentioned this pull request Nov 2, 2026

feat: Add flag override tests for a client whose store holds flags without the init key launchdarkly/sdk-test-harness#539

Merged

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 0415ec1 to a8e9294 Compare November 3, 2026 23:56

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from 0d55468 to 5800e36 Compare November 3, 2026 23:56

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 1cba56a to d6b022b Compare November 4, 2026 22:55

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from 6172b2a to 7cf3365 Compare November 4, 2026 22:55

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from d6b022b to a52ba14 Compare November 4, 2026 23:09

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from 7cf3365 to 71b52ee Compare November 4, 2026 23:09

keelerm84 approved these changes Nov 5, 2026

View reviewed changes

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from 71b52ee to 7e42fc2 Compare November 6, 2026 17:53

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from a52ba14 to 36ceb3f Compare November 6, 2026 17:53

kinyoklion mentioned this pull request Nov 9, 2026

fix: Accept a path object as an override file path #572

Merged

3 of 4 tasks

kinyoklion added 13 commits November 10, 2026 22:19

test: Run the override vectors and the event marking scenarios on the…

f96aa4c

… async client

test: Drop the async offline override test that the override store ch…

1588f31

…ange now covers

fix: Reject a poll interval longer than the runtime can wait

2ede91b

kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 36ceb3f to 9a582aa Compare November 10, 2026 22:24

kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from 7e42fc2 to 5d19e57 Compare November 10, 2026 22:24

Base automatically changed from rlamb/overrides-python-file-source to feat/overrides November 10, 2026 23:09

kinyoklion marked this pull request as ready for review November 10, 2026 23:09

kinyoklion requested a review from a team as a code owner November 10, 2026 23:09

Merge branch 'feat/overrides' into rlamb/overrides-python-contract-tests

ed2e3bf

kinyoklion merged commit 4b5a4b5 into feat/overrides Nov 10, 2026
14 checks passed

kinyoklion deleted the rlamb/overrides-python-contract-tests branch November 10, 2026 23:27

Navigation Menu

Repository navigation

feat: Add flag overrides to the contract test services - #530

feat: Add flag overrides to the contract test services#530
kinyoklion merged 14 commits into
feat/overridesfrom
rlamb/overrides-python-contract-tests

kinyoklion commented Sep 25, 2026 •

edited by cursor Bot

Loading

Uh oh!

kinyoklion commented Oct 3, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kinyoklion commented Sep 25, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

kinyoklion commented Oct 3, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kinyoklion commented Sep 25, 2026 •

edited by cursor Bot

Loading