printf: fix regression in large double formatting on ld128 archs

commit 572a2e2eb91f00f2f25d301cfb50f435e7ae16b3 adjusted the buffer
for decimal conversion to be a VLA that only uses the full size needed
for long double when the argument type was long double. however, it
failed to update a later expression for the positioning within the
buffer, which still used a fixed offset of LDBL_MANT_DIG. this caused
doubles with a large positive exponent to overflow below the start of
the array, producing wrong output and potentially runaway wrong
execution.

this bug has not been present in any release, and has not been
analyzed in depth for security considerations.

it turns out the original buffer offset expression involving
LDBL_MANT_DIG was incorrect as well, and only worked because the space
reserved for expanding the exponent is roughly 3 times the size it
needs to be when the exponent is positive, leaving plenty of extra
space to compensate for the error. the actual offset should be in
base-1000000000 slot units, not bits, and numerically equal to the
number of slots that were previously allocated for mantissa expansion.

in order to ensure consistency and make the code more comprehensible,
commented subexpressions are replaced by intermediate named variables,
and the newly introduced max_mant_slots is used for both the
allocation and the buffer offset adjustment. the included +1 term
accounts for a trailing zero slot that's always emitted.

1 files changed, 6 insertions, 6 deletions

diff --git a/src/stdio/vfprintf.c b/src/stdio/vfprintf.c
index 76733997..a68edabb 100644
--- a/src/stdio/vfprintf.c
+++ b/src/stdio/vfprintf.c
@@ -180,11 +180,11 @@ typedef char compiler_defines_long_double_incorrectly[9-(int)sizeof(long double)
 
 static int fmt_fp(FILE *f, long double y, int w, int p, int fl, int t, int ps)
 {
-	int bufsize = (ps==BIGLPRE)
-		? (LDBL_MANT_DIG+28)/29 + 1 +          // mantissa expansion
-		  (LDBL_MAX_EXP+LDBL_MANT_DIG+28+8)/9  // exponent expansion
-		: (DBL_MANT_DIG+28)/29 + 1 +
-		  (DBL_MAX_EXP+DBL_MANT_DIG+28+8)/9;
+	int max_mant_dig = (ps==BIGLPRE) ? LDBL_MANT_DIG : DBL_MANT_DIG;
+	int max_exp = (ps==BIGLPRE) ? LDBL_MAX_EXP : DBL_MAX_EXP;
+	int max_mant_slots = (max_mant_dig+28)/29 + 1;
+	int max_exp_slots = (max_exp+max_mant_dig+28+8)/9;
+	int bufsize = max_mant_slots + max_exp_slots;
 	uint32_t big[bufsize];
 	uint32_t *a, *d, *r, *z;
 	int e2=0, e, i, j, l;
@@ -266,7 +266,7 @@ static int fmt_fp(FILE *f, long double y, int w, int p, int fl, int t, int ps)
 	if (y) y *= 0x1p28, e2-=28;
 
 	if (e2<0) a=r=z=big;
-	else a=r=z=big+sizeof(big)/sizeof(*big) - LDBL_MANT_DIG - 1;
+	else a=r=z=big+sizeof(big)/sizeof(*big) - max_mant_slots - 1;
 
 	do {
 		*z = y;