fix toctou race in popen children's closing of other popen pipesHEADmaster

since commit e1a51185ceb4386481491e11f6dd39569b9e54f7, popen obeys an
obscure historical requirement to implement a sort of pseudo-cloexec
behavior for other pipe streams obtained by popen. but since popen
uses posix_spawn (and thereby posix_spawn file actions) internally,
the time of check for other pipe streams is separated from the time of
closure.

this was intended to be addressed by popen holding the open file list
lock across posix_spawn, but pclose (via fclose) closes the file
descriptor before taking the open file list lock, only using the lock
for updating the linked list pointers. this is to avoid serializing
fclose operations across the entire process, since in general close
may be a blocking operation.

multiple solutions to this problem were considered, but the simplest
and least invasive is conditionally taking the open file list lock
early for popen streams -- that is, for FILE streams where the
pipe_pid member is nonzero. since the file descriptor refers to a
pipe, closing it is a simple, nonblocking operation, and the only cost
is the time spent entering and leaving kernelspace while the lock is
held.

since individual FILE locks cannot be held while taking the open file
list lock (this would violate lock order protocol and produce
deadlocks), the FILE lock must be released after fflush but before the
ofl lock is taken. there is no point in retaking the lock afterwards,
since the FILE pointer is no longer valid and any further use by the
application would be undefined. so, simply let fflush do the locking.

note that the early return path (F_PERM) is not reachable for popen
streams, only for stdin/stdout/stderr, which are always normal FILEs.
thus, it does not provide a code path to return with the ofl lock
still held.

0 files changed, 0 insertions, 0 deletions