[felixbuenemann]

In thread-safe builds, setlocale() changes the locale of the whole process:

• a request uses the locale set by requests running in other threads;
• the reset at the end of a request also resets the locale of all other requests;
• concurrent calls can crash, because the C library's setlocale() isn't thread-safe.

The manual documents the first point as a limitation. Windows doesn't have it: PHP keeps the locale per thread there since 5.3.6 (0778bed) and 5.6.20/7.0.5 (632fc51, bug #71596). This PR does the same for the other platforms, using newlocale() and uselocale().

It takes the approach of #20531, which was closed in favour of a broader ZTS rework. On internals (externals.io/message/128991), a pull request was suggested rather than an RFC.

NTS builds use the same code, so there is a single implementation. There, setlocale() no longer changes the process locale either.

Changes

• New zend_setlocale() in Zend/zend_locale.c. On systems with uselocale(), in ZTS and NTS builds alike, during a request it switches the current thread to a locale of its own. The locale-dependent functions of the C library (printf(), localeconv(), strftime(), strcoll(), nl_langinfo(), mblen()…) use that locale in place of the process locale. Elsewhere it's a macro for setlocale().
• PHP's own setlocale() calls go through it: the setlocale() function, the reset in PHP_RSHUTDOWN(basic), zend_reset_lc_ctype_locale(), zend_update_current_locale() and NumberFormatter::parse().
• configure.ac checks for uselocale() and xlocale.h.

Behaviour that doesn't change:

• The process locale is still set during startup, and it is the initial locale of every thread. Outside of a request (!EG(active)), zend_setlocale() changes the process locale as before.
• A request that changed the locale still resets it to C at the end. That now only affects its own thread, as on Windows.
• setlocale() returns the same names as the C library's setlocale(), including the name of LC_ALL when the categories have different locales. That name has a different format in glibc, musl and macOS/BSD, so setlocale(LC_ALL, $saved) keeps working.
• Windows is unaffected.

Implementation notes

• A locale object is created once for each combination of category locales and shared by all threads. Threads only switch between these objects with uselocale().
• The locale objects are never freed. Threads keep using them, a process only needs a few, and macOS crashes when newlocale() and freelocale() run concurrently. The number of objects is not bounded, though: it grows with the distinct combinations a process sets.
• Looking up an existing locale takes no lock: a locale never changes once it is in the list, and the head of the list is an atomic pointer. Creating a locale happens under tsrm_env_lock(), and so does resolving the "" locale, which reads the environment.
• Requests that don't call setlocale() run no new code.
• A thread that hasn't set a locale never calls setlocale(LC_ALL, NULL): on macOS that call writes to a static buffer, so concurrent calls return garbage.

Tests

• setlocale_variation6.phpt (new) covers restoring a saved name of LC_ALL when the categories have different locales.
• setlocale_variation4.phpt and setlocale_variation5.phpt had a trailing semicolon in their --ENV-- section, so the locale was en_US.utf8;. glibc accepts that name in setlocale(), but not in newlocale(). The semicolon is removed.
• phpt tests can't run threads. I tested the thread behaviour with the program below, which uses the embed SAPI: 9 threads run 2,000 requests each. A third of the threads switch to German, a third to French, and a third never set a locale. Each request checks its initial locale and its results.

	macOS 27	Debian 13 (glibc 2.41)	Alpine 3.24 (musl)
This PR	passes (3 of 3 runs)	passes (3 of 3 runs)	passes (3 of 3 runs)
master	aborts (3 of 3 runs)	wrong locales (3 of 3 runs)	wrong locales (3 of 3 runs)

• On the same three systems, I compared the output of ZTS and NTS builds of this PR with an NTS build of master for 61 setlocale() edge cases (invalid names, '0', '', null, arrays, POSIX, wrong types…), 7 environments (LANG, LC_ALL, LC_NUMERIC, none…) and 15 names of LC_ALL with missing, extra or empty parts. The output is identical.
• The tests in ext/standard/tests/strings, ext/standard/tests/general_functions, ext/ctype, ext/pcre and tests/lang give the same results in ZTS and NTS builds on all three systems. The NumberFormatter tests pass on Debian and Alpine. The gettext tests pass in an NTS build on macOS (GNU libintl), as on master.
• A ThreadSanitizer build of the thread program on macOS reports nothing in the locale code.
• Not tested locally: FreeBSD, Solaris and Windows.

Limitations

• C code that calls the C library's setlocale() directly, in an extension or a library, still changes the process locale, and doesn't see the locale of the thread. That now applies to NTS builds too. Extensions should call zend_setlocale() instead (see UPGRADING.INTERNALS).
• On systems where LC_ALL is negative (AIX, as far as I know), zend_setlocale() stays a macro for setlocale(), because the category table in zend_locale.c is indexed by category.
• On other C libraries than glibc, musl and those of macOS and the BSDs, the name of LC_ALL for different category locales uses the musl format. It still round-trips through setlocale().

Thread test program (embed SAPI)

/* Runs PHP requests in several threads with the embed SAPI. A third of the
 * threads switch to German, a third to French and a third never set a locale.
 * Each request checks that it starts with the initial locale and that requests
 * in other threads don't change its locale.
 * Usage: threads [threads] [requests per thread] */
#include <sapi/embed/php_embed.h>
#include <locale.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define KINDS 3

static int requests = 2000;
static const char *locales[KINDS] = {"de_DE.UTF-8", "fr_FR.UTF-8", ""};
static char codes[KINDS][2048];
static char expected[KINDS][1024];

/* Returns "<initial locale>|<locale>|<decimal point>|<number>|<month>" */
static const char *script =
	"(function () {"
	"  $locale = '%s';"
	"  $initial = setlocale(LC_ALL, 0);"
	"  $set = $locale === '' ? $initial : setlocale(LC_ALL, $locale);"
	"  usleep(50);"
	"  $result = implode('|', [$initial, $set, setlocale(LC_ALL, 0),"
	"    localeconv()['decimal_point'], sprintf('%%.1f', 1.5), nl_langinfo(MON_1)]);"
	"  if (%d) { setlocale(LC_ALL, $initial); }"
	"  return $result;"
	"})()";

static bool eval(const char *code, char *result, size_t size)
{
	zval retval;

	if (zend_eval_string(code, &retval, "request") != SUCCESS || Z_TYPE(retval) != IS_STRING) {
		return false;
	}
	snprintf(result, size, "%s", Z_STRVAL(retval));
	zval_ptr_dtor(&retval);

	return true;
}

static void *run(void *arg)
{
	long id = (long) arg, wrong = 0;
	int kind = id % KINDS;
	char result[1024], example[1024] = "";

	(void) ts_resource(0);

	for (int i = 0; i < requests; i++) {
		php_request_startup();
		if (!eval(codes[kind], result, sizeof(result)) || strcmp(result, expected[kind]) != 0) {
			if (!wrong++) {
				snprintf(example, sizeof(example), "%s", result);
			}
		}
		php_request_shutdown(NULL);
	}

	printf("thread %2ld (%-11s): %ld of %d requests wrong%s%s\n", id,
		locales[kind][0] ? locales[kind] : "no locale", wrong, requests, wrong ? ", e.g. " : "", example);
	ts_free_thread();

	return (void *) wrong;
}

int main(int argc, char **argv)
{
	int count = argc > 1 ? atoi(argv[1]) : 9;
	long failed = 0;
	char initial[512], code[2048];
	pthread_t threads[64];

	if (argc > 2) {
		requests = atoi(argv[2]);
	}

	php_embed_init(1, argv);

	/* The expected results, from a single thread */
	for (int kind = 0; kind < KINDS; kind++) {
		snprintf(code, sizeof(code), script, locales[kind], 1);
		snprintf(codes[kind], sizeof(codes[kind]), script, locales[kind], 0);
		if (!eval(code, expected[kind], sizeof(expected[kind]))) {
			return 2;
		}
		printf("expected (%-11s): %s\n", locales[kind][0] ? locales[kind] : "no locale", expected[kind]);
	}
	snprintf(initial, sizeof(initial), "%s", setlocale(LC_ALL, NULL));

	for (long i = 0; i < count; i++) {
		pthread_create(&threads[i], NULL, run, (void *) i);
	}
	for (int i = 0; i < count; i++) {
		void *result;
		pthread_join(threads[i], &result);
		failed += (long) result;
	}

	if (strcmp(initial, setlocale(LC_ALL, NULL)) != 0) {
		printf("process locale changed from %s to %s\n", initial, setlocale(LC_ALL, NULL));
		failed++;
	}
	printf(failed ? "FAILED\n" : "OK\n");
	php_embed_shutdown();

	return failed != 0;
}

Built with --enable-zts --enable-embed, and linked against libphp.

[felixbuenemann]

@henderkes If I read you comment on php/frankenphp#2831 correctly, you are proposing to always use the thread-safe code, even in NTS builds?

[felixbuenemann]

Unrelated LINUX_X32_DEBUG_ZTS test failure should be fixed by #25423.

[henderkes]

Just some initial thoughts

[henderkes]

I'd prefer a lock free implementation where possible.

[felixbuenemann]

Looking up an existing locale no longer takes a lock. It uses an atomic pointer as the head of an append-only list.

A lock is still used for:

• Creating a locale (prevents crashes with cocurrent newloclae() calls on macOS), with additional testing could maybe be reduced to Darwin and BSD if also affected
• Resolving the "" locale, because it reads the environment (prevents getenv() race)

[felixbuenemann]

@henderkes The FREEBSD_NTS failure seems to indicate that FreeBSD has some libc functions that do not respect thread-local locale. I'll try setting up a local FreeBSD VM to verify.

[felixbuenemann]

@henderkes I am unsure what to do about the FreeBSD failure:

The problem is that on FreeBSD and DragonFly uselocale() does not update __mb_sb_limit, only setlocale() does, so if the process starts with eg. C.UTF-8 and then switches to de_DE.ISO-8859-1, the return value of isalpha() is stale.

A FreeBSD specific hack could partially solve it:

uselocale(loc);
__runes_for_locale(loc, &limit);   /* the limit of this locale */
__mb_sb_limit = limit;

But for multiple threads switching locales that is only correct, if all threads use either UTF-8 or non-UTF-8 locales :-(

Another approach is to use isalpha_l() etc. on FreeBSD which are thread-local aware, but it doesnt help for linked system libraries like PCRE2 as often done in distribution packages.

Or we could stay on setlocale() for FreeBSD, but having different behavior on one platform seems a bad compromise.

Let me know what you think or if you have a better idea.