ftello adds pending buffered output to the position reported by the underlying seek operation. Near LLONG_MAX, the addition can overflow signed off_t and return an apparently successful negative position. Check that the buffered-byte count fits before adding it. Fail with EOVERFLOW when the logical position cannot be represented.
For SEEK_CUR, fseeko subtracts the unread input-buffer length from the caller's offset before invoking the underlying seek operation. A valid LLONG_MIN offset therefore overflows before the seek can reject the unrepresentable logical result. Detect the underflow and fail with EOVERFLOW without flushing or discarding the stream's buffers.
Fresh writable streams use null wpos and wend pointers until output is initialized. The non-ASCII path adds MB_LEN_MAX to wpos before comparing it with wend, which is invalid for a null pointer. The addition can also form a pointer beyond one past the buffer when little space remains. First require an active output buffer. Then compare the defined difference between its pointers. Preserve the existing strict capacity test and use the normal write fallback otherwise.
since commit e1a51185ceb4386481491e11f6dd39569b9e54f7, popen obeys an obscure historical requirement to implement a sort of pseudo-cloexec behavior for other pipe streams obtained by popen. but since popen uses posix_spawn (and thereby posix_spawn file actions) internally, the time of check for other pipe streams is separated from the time of closure. this was intended to be addressed by popen holding the open file list lock across posix_spawn, but pclose (via fclose) closes the file descriptor before taking the open file list lock, only using the lock for updating the linked list pointers. this is to avoid serializing fclose operations across the entire process, since in general close may be a blocking operation. multiple solutions to this problem were considered, but the simplest and least invasive is conditionally taking the open file list lock early for popen streams -- that is, for FILE streams where the pipe_pid member is nonzero. since the file descriptor refers to a pipe, closing it is a simple, nonblocking operation, and the only cost is the time spent entering and leaving kernelspace while the lock is held. since individual FILE locks cannot be held while taking the open file list lock (this would violate lock order protocol and produce deadlocks), the FILE lock must be released after fflush but before the ofl lock is taken. there is no point in retaking the lock afterwards, since the FILE pointer is no longer valid and any further use by the application would be undefined. so, simply let fflush do the locking. note that the early return path (F_PERM) is not reachable for popen streams, only for stdin/stdout/stderr, which are always normal FILEs. thus, it does not provide a code path to return with the ofl lock still held.
commit d2e061a2bd3f7674cfef2e2217e0695419041b5e wrongly added this case based on a misreading of the standard text regarding null termination in write/update modes. apparently, the condition "if it fits" was interpreted as only applying to update modes, despite the words "or for writing only" appearing.
commit f96e47a26102d537c29435f0abf9ec94676a030e introduced a new overflow past the end of the base-1e9 buffer for floating point to decimal conversion while fixing a different overflow below the start of the buffer. this bug has not been present in any release, and has not been analyzed in depth for security considerations. the root cause of the bug, incorrect size accounting for the mantissa, long predates the above commit, but was only exposed once the excessive offset causing overflow in the other direction was removed. the number of slots for expanding the mantissa was computed as if each slot could peel off at least 29 bits. this would be true if the mantissa were placed and expanded to the left of the radix point, but we don't do that because it would require repeated fmod and division. instead, we start the mantissa with 29 bits to the left of the radix point, so that they can be peeled off by conversion to integer and subtraction, followed by a multiplication by 1e9 to prepare for the next iteration. so while the first slot peels 29 bits, advancing to the next slot adds back somewhere between 20 and 21 bits: the length of the mantissa of 1e9. this means we need to account for a slot for every 8 bits of mantissa past the initial 29. add a comment to that effect and adjust the max_mant_slots formula.
commit 572a2e2eb91f00f2f25d301cfb50f435e7ae16b3 adjusted the buffer for decimal conversion to be a VLA that only uses the full size needed for long double when the argument type was long double. however, it failed to update a later expression for the positioning within the buffer, which still used a fixed offset of LDBL_MANT_DIG. this caused doubles with a large positive exponent to overflow below the start of the array, producing wrong output and potentially runaway wrong execution. this bug has not been present in any release, and has not been analyzed in depth for security considerations. it turns out the original buffer offset expression involving LDBL_MANT_DIG was incorrect as well, and only worked because the space reserved for expanding the exponent is roughly 3 times the size it needs to be when the exponent is positive, leaving plenty of extra space to compensate for the error. the actual offset should be in base-1000000000 slot units, not bits, and numerically equal to the number of slots that were previously allocated for mantissa expansion. in order to ensure consistency and make the code more comprehensible, commented subexpressions are replaced by intermediate named variables, and the newly introduced max_mant_slots is used for both the allocation and the buffer offset adjustment. the included +1 term accounts for a trailing zero slot that's always emitted.
When buffering on a FILE is disabled we still send both iovecs, even though the first one is always empty. Clean things up by skipping the empty iovec instead.
internally, printf always works with the maximal-size supported integer and floating point formats. however, the space needed to format a floating point number is proportional to the mantissa and exponent ranges. on archs where long double is larger than double, knowing that the actual value fit in double allows us to use a much smaller buffer, roughly 1/16 the size. as a bonus, making the working buffer a VLA whose dimension depends on the format specifier prevents the compiler from lifting the stack adjustment to the top of printf_core. this makes it so printf calls without floating point arguments do not waste even the smaller amount of stack space needed for double, making it much more practical to use printf in tightly stack-constrained environments.