these changes are based on the following communication via email: "I hereby grant that all of the code I have contributed to musl on or before April 23, 2012 may be licensed under the terms of the following MIT license: Copyright (c) 2011-2012 Nicholas J. Kain Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
this change is necessary or pthread_create will always fail on security-hardened kernels. i considered first trying to make the stack executable and simply retrying without execute permissions when the first try fails, but (1) this would incur a serious performance penalty on hardened systems, and (2) having the stack be executable is just a bad idea from a security standpoint. if there is real-world "GNU C" code that uses nested functions with threads, and it can't be fixed, we'll have to consider other ways of solving the problem, but for now this seems like the best fix.
pthread structure has been adjusted to match the glibc/GCC abi for where the canary is stored on i386 and x86_64. it will need variants for other archs to provide the added security of the canary's entropy, but even without that it still works as well as the old "minimal" ssp support. eventually such changes will be made anyway, since they are also needed for GCC/C11 thread-local storage support (not yet implemented). care is taken not to attempt initializing the thread pointer unless the program actually uses SSP (by reference to __stack_chk_fail).
i did some testing trying to switch malloc to use the new internal lock with priority inheritance, and my malloc contention test got 20-100 times slower. if priority inheritance futexes are this slow, it's simply too high a price to pay for avoiding priority inversion. maybe we can consider them somewhere down the road once the kernel folks get their act together on this (and perferably don't link it to glibc's inefficient lock API)... as such, i've switch __lock to use malloc's implementation of lightweight locks, and updated all the users of the code to use an array with a waiter count for their locks. this should give optimal performance in the vast majority of cases, and it's simple. malloc is still using its own internal copy of the lock code because it seems to yield measurably better performance with -O3 when it's inlined (20% or more difference in the contention stress test).
this bug probably would have gone unnoticed since it's only used in the fallback code for systems where priority-inheritance locking fails. unfortunately this approach results in one spurious wake syscall on the final unlock, when there are no waiters remaining. the alternative (possibly better) would be to use broadcast wakes instead of reflagging the waiter unconditionally, and let each waiter reflag itself; this saves one syscall at the expense of invoking the "thundering herd" effect (worse performance degredation) when there are many waiters. ideally we would be able to update all of our locks to use an array of two ints rather than a single int, and use a separate counter system like proper mutexes use; then we could avoid all spurious wake calls without resorting to broadcasts. however, it's not clear to me that priority inheritance futexes support this usage. the kernel sets the waiters flag for them (just like we're doing now) and i can't tell if it's safe to bypass the kernel when unlocking just because we know (from private data, the waiter count) that there are no waiters. this is something that could be explored in the future.
we use priority inheritance futexes if possible so that the library cannot hit internal priority inversion deadlocks in the presence of realtime priority scheduling (full support to be added later).
this was discussed on the mailing list and no consensus on the preferred solution was reached, so in anticipation of a release, i'm just committing a minimally-invasive solution that avoids the problem by ensuring that multi-threaded-capable programs will always have initialized the thread pointer before any signal handler can run. in the long term we may switch to initializing the thread pointer at program start time whenever the program has the potential to access any per-thread data.
even if pthread_create/exit code is not linked, run flag needs to be checked and cleanup function potentially run on pop. thus, move the code to the module that's always linked when pthread_cleanup_push/pop is used.
the old abi was intended to duplicate glibc's abi at the expense of being ugly and slow, but it turns out glib was not even using that abi except on non-gcc-compatible compilers (which it doesn't even support) and was instead using an exceptions-in-c/unwind-based approach whose abi we could not duplicate anyway without nasty dwarf2/unwind integration. the new abi is copied from a very old glibc abi, which seems to still be supported/present in current glibc. it avoids all unwinding, whether by sjlj or exceptions, and merely maintains a linked list of cleanup functions to be called from the context of pthread_exit. i've made some care to ensure that longjmp out of a cleanup function should work, even though it is not required to. this change breaks abi compatibility with programs which were using pthread cancellation, which is unfortunate, but that's why i'm making the change now rather than later. considering that most pthread features have not been usable until recently anyway, i don't see it as a major issue at this point.