[vahid-ahmadi]

Cause

Two separate failures. Only the first is the 303.

1. Redirect failures: same cause as #236, fixed here

Deploy smoke test 42863423889 and Validate live deployment 42977456119:

FAILED tests/test_remote_mcp.py::test_obr_shock_cgg - HTTPStatusError('Redirect to https://policyengine--policyengine-macro-mcp-serve.modal.run/mcp not followed; ...')
FAILED tests/test_remote_mcp.py::test_score_reform_microsim_scoreresult - HTTPStatusError('Redirect to https://policyengine--policyengine-macro-mcp-serve.modal.run/mcp not followed; ...')

Both tests call a slow tool directly. On a cold container the call runs past Modal's 150 s web-request ceiling, and Modal answers HTTP 303. The MCP client (mcp 1.30) does not follow a 303 on a POST. Following it would not help either: it becomes a GET, and Modal's result URL rejects that (jobs.py).

• test_obr_shock_cgg has failed this way on most validation runs since early October.
• test_score_reform_microsim_scoreresult passed every day until 30 Oct and has failed on most runs since 31 Oct (it passed once, on 4 Nov), which suggests it now sits just over the ceiling.

Fix. Both tools are already on jobs.JOB_TOOLS, so no server change or deploy is needed. The two tests now use start_job and then get_job_result polls of 30 s, through integration/tests/remote_jobs.py::run_via_job, with a 300 s deadline each.

• A transport error on a poll is retried.
• A tool error, or a job still running at the deadline, fails with the job id.

2. test_score_reform_default_window_works_through_a_job: not the 303, written up, not changed

This test already uses the job API, and the job starts fine. It fails because the job is still running after 20 polls:

AssertionError: score_reform did not finish within 600s: {'job_id': 'fc-01M74QJB6ZM861ZRT14DDP8NA8', 'status': 'running', 'waited_seconds': 30, ...}

Evidence that this is a runtime regression on the server, not a client problem:

• It passed on every daily validation through 30 Oct 07:10 (runs 40833809437, 41018496516), with the same test code.
• It has failed on every run since 31 Oct (41179371760 onwards).
• The deploy between those runs was 41063876937 (30 Oct, 153cf419). It was the first deploy since 12 Oct and shipped the ed47a936..153cf419 integration changes:
  • policyengine[models] >=4,<5 → ==6.11.1
  • policyengine-us 1.764.6 → 2.2.1
  • new pins policyengine-uk==2.151.0 and policyengine-core==3.33.3
  • in core.py, a required SPM geography for US households (01260bff) and the US population dataset name (5b207e0e), both from 22–28 Oct
• The direct microsim test started hitting the 150 s ceiling on the same date, which points to the PolicyEngine static costing (the package bump) as the part that slowed down.

I have not confirmed this. Modal's per-call timings would, and nothing in CI records them.

The checks that would settle it all hit the hosted server, so they are listed for Vahid rather than run:

1. get_job_result on fc-01M74QJB6ZM861ZRT14DDP8NA8 (11 Nov) shows whether that job finished or errored.
2. Modal's function-call duration for run_tool_job before and after 30 Oct.

Raising the 600 s deadline without that evidence would only hide the regression, so this PR leaves it alone. Until it is resolved, the deploy smoke test and daily validation stay red on this one test.

Tests

• integration/tests/test_remote_jobs.py: 8 unit tests. Six exercise run_via_job against a scripted server, with no network. They cover start then poll to done, never calling the tool directly, retrying a transport error, giving up after repeated errors, a job error being fatal, and the deadline. Two more run mcp_caller, the real call function, through fakes that hold live anyio task groups. The suite's existing _call asserts inside those contexts, where an ExceptionGroup would wrap the AssertionError and a failed job would be retried as a dropped connection, so mcp_caller judges the reply after the contexts close (a3e10bc4). On main the file fails on import; here all 8 pass.
• test_remote_mcp.py stays skipped in PR CI (it needs POLICYENGINE_MACRO_REMOTE_TESTS=1). The two converted tests are not yet verified live. I did not dispatch a validation run, to avoid an extra Modal run. The first scheduled Validate live deployment after merge (07:00 UTC) will exercise them.
• Budget. The deploy job has a 25-minute cap, and the suite took 16.5 min on 10 Nov, with the microsim test failing fast at 150 s. With jobs, those two tests can each take up to 300 s, so a deploy smoke run may now come close to the cap while item 2 above is unresolved. The modal deploy step runs first, so the deploy itself is unaffected.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
macromod	Ready	Preview	Nov 11, 2026 10:23am UTC