libc.h was intended to be a header for access to global libc state and related interfaces, but ended up included all over the place because it was the way to get the weak_alias macro. most of the inclusions removed here are places where weak_alias was needed. a few were recently introduced for hidden. some go all the way back to when libc.h defined CANCELPT_BEGIN and _END, and all (wrongly implemented) cancellation points had to include it. remaining spurious users are mostly callers of the LOCK/UNLOCK macros and files that use the LFS64 macro to define the awful *64 aliases. in a few places, new inclusion of libc.h is added because several internal headers no longer implicitly include libc.h. declarations for __lockfile and __unlockfile are moved from libc.h to stdio_impl.h so that the latter does not need libc.h. putting them in libc.h made no sense at all, since the macros in stdio_impl.h are needed to use them correctly anyway.
the mode member of struct ipc_perm is specified by POSIX to have type mode_t, which is uniformly defined as unsigned int. however, Linux defines it with type __kernel_mode_t, and defines __kernel_mode_t as unsigned short on some archs. since there is a subsequent padding field, treating it as a 32-bit unsigned int works on little endian archs, but the order is backwards on big endian archs with the erroneous definition. since multiple archs are affected, remedy the situation with fixup code in the affected functions (shmctl, semctl, and msgctl) rather than repeating the same shims in syscall_arch.h for every affected arch.
it's UB to fetch variadic args when none are passed, and this caused real crashes on ppc due to its calling convention, which defines that for variadic functions aggregate types be passed as pointers. the assignment caused that pointer to get dereferenced, resulting in a crash.
per POSIX, the variadic argument has type union semun, which may contain a pointer or int; the type read depends on the command being issued. this allows the userspace part of the implementation to be type-correct without requiring special-casing for different commands. the kernel always expects to receive the argument interpreted as unsigned long (or equivalently, a pointer), and does its own handling of extracting the int portion from the representation, as needed. this change fixes two possible issues: most immediately, reading the argument as a (signed) long and passing it to the syscall would perform incorrect sign-extension of pointers on the upcoming x32 target. the other possible issue is that some archs may use different (user-space) argument-passing convention for unions, preventing va_arg from correctly obtaining the argument when the type long (or even unsigned long or void *) is passed to it.
this should not matter since the reality is that either all the sysv sem syscalls are individual syscalls, or all of them are multiplexed on the SYS_ipc syscall (depending on arch). but best to be consistent anyway.
this is a Linux-specific extension to the sysv semaphore api.
rather than returning an error, we have to increase the size argument so high that the kernel will have no choice but to fail. this is because POSIX only permits the EINVAL error for size errors when a new shared memory segment would be created; if it already exists, the size argument must be ignored. unfortunately Linux is non-conforming in this regard, but I want to keep the code correct in userspace anyway so that if/when Linux is fixed, the behavior applications see will be conforming.
rejecting invalid values for n is fine even in the case where a new sem will not be created, since the kernel does its range checks on n even in this case as well. by default, the kernel will bound the limit well below USHRT_MAX anyway, but it's presumably possible that an administrator could override this limit and break things.