<feed xmlns='http://www.w3.org/2005/Atom'>
<title>musl/src/multibyte, branch v1.0.2</title>
<subtitle>musl - an implementation of the standard library for Linux-based systems</subtitle>
<link rel='alternate' type='text/html' href='http://git.musl-libc.org/cgit/musl/'/>
<entry>
<title>include cleanups: remove unused headers and add feature test macros</title>
<updated>2013-12-12T05:09:18+00:00</updated>
<author>
<name>Szabolcs Nagy</name>
<email>nsz@port70.net</email>
</author>
<published>2013-12-12T05:09:18+00:00</published>
<link rel='alternate' type='text/html' href='http://git.musl-libc.org/cgit/musl/commit/?id=571744447c23f91feb6439948f3a619aca850dfb'/>
<id>571744447c23f91feb6439948f3a619aca850dfb</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>fix buffer overflow in mbsrtowcs</title>
<updated>2013-09-27T17:09:46+00:00</updated>
<author>
<name>Rich Felker</name>
<email>dalias@aerifal.cx</email>
</author>
<published>2013-09-27T17:09:46+00:00</published>
<link rel='alternate' type='text/html' href='http://git.musl-libc.org/cgit/musl/commit/?id=211264e46a2f1bc382a84435e904d1548de672b0'/>
<id>211264e46a2f1bc382a84435e904d1548de672b0</id>
<content type='text'>
issue reported by Michael Forney:

"If wn becomes 0 after processing a chunk of 4, mbsrtowcs currently
continues on, wrapping wn around to -1, causing the rest of the string
to be processed.

This resulted in buffer overruns if there was only space in ws for wn
wide characters."

the original patch submitted added an additional check for !wn after
the loop; to avoid extra branching, I instead just changed the wn&gt;=4
check to wn&gt;=5 to ensure that at least one slot remains after the
word-at-a-time loop runs. this should not slow down the tail
processing on real-world usage, since an extra slot that can't be
processed in the word-at-a-time loop is needed for the null
termination anyway.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
issue reported by Michael Forney:

"If wn becomes 0 after processing a chunk of 4, mbsrtowcs currently
continues on, wrapping wn around to -1, causing the rest of the string
to be processed.

This resulted in buffer overruns if there was only space in ws for wn
wide characters."

the original patch submitted added an additional check for !wn after
the loop; to avoid extra branching, I instead just changed the wn&gt;=4
check to wn&gt;=5 to ensure that at least one slot remains after the
word-at-a-time loop runs. this should not slow down the tail
processing on real-world usage, since an extra slot that can't be
processed in the word-at-a-time loop is needed for the null
termination anyway.
</pre>
</div>
</content>
</entry>
<entry>
<title>fix failure of mbsrtowcs to record stop position when dest is full</title>
<updated>2013-06-29T20:16:47+00:00</updated>
<author>
<name>Rich Felker</name>
<email>dalias@aerifal.cx</email>
</author>
<published>2013-06-29T20:16:47+00:00</published>
<link rel='alternate' type='text/html' href='http://git.musl-libc.org/cgit/musl/commit/?id=4ca442157e381690202c3bcc102627d137fd0466'/>
<id>4ca442157e381690202c3bcc102627d137fd0466</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>mbrtowc: do not leave mbstate_t in permanent-fail state after EILSEQ</title>
<updated>2013-04-09T03:09:11+00:00</updated>
<author>
<name>Rich Felker</name>
<email>dalias@aerifal.cx</email>
</author>
<published>2013-04-09T03:09:11+00:00</published>
<link rel='alternate' type='text/html' href='http://git.musl-libc.org/cgit/musl/commit/?id=23ab8c255543a7e0876c7e1858ef0d4bbd562729'/>
<id>23ab8c255543a7e0876c7e1858ef0d4bbd562729</id>
<content type='text'>
the standard is clear that the old behavior is conforming: "In this
case, [EILSEQ] shall be stored in errno and the conversion state is
undefined."

however, the specification of mbrtowc has one peculiarity when the
source argument is a null pointer: in this case, it's required to
behave as mbrtowc(NULL, "", 1, ps). no motivation is provided for this
requirement, but the natural one that comes to mind is that the intent
is to reset the mbstate_t object. for stateful encodings, such
behavior is actually specified: "If the corresponding wide character
is the null wide character, the resulting state described shall be the
initial conversion state." but in the case of UTF-8 where the
mbstate_t object contains a partially-decoded character rather than a
shift state, a subsequent '\0' byte indicates that the previous
partial character is incomplete and thus an illegal sequence.

naturally, applications using their own mbstate_t object should clear
it themselves after an error, but the standard presently provides no
way to clear the builtin mbstate_t object used when the ps argument is
a null pointer. I suspect this issue may be addressed in the future by
specifying that a null source argument resets the state, as this seems
to have been the intent all along.

for what it's worth, this change also slightly reduces code size.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
the standard is clear that the old behavior is conforming: "In this
case, [EILSEQ] shall be stored in errno and the conversion state is
undefined."

however, the specification of mbrtowc has one peculiarity when the
source argument is a null pointer: in this case, it's required to
behave as mbrtowc(NULL, "", 1, ps). no motivation is provided for this
requirement, but the natural one that comes to mind is that the intent
is to reset the mbstate_t object. for stateful encodings, such
behavior is actually specified: "If the corresponding wide character
is the null wide character, the resulting state described shall be the
initial conversion state." but in the case of UTF-8 where the
mbstate_t object contains a partially-decoded character rather than a
shift state, a subsequent '\0' byte indicates that the previous
partial character is incomplete and thus an illegal sequence.

naturally, applications using their own mbstate_t object should clear
it themselves after an error, but the standard presently provides no
way to clear the builtin mbstate_t object used when the ps argument is
a null pointer. I suspect this issue may be addressed in the future by
specifying that a null source argument resets the state, as this seems
to have been the intent all along.

for what it's worth, this change also slightly reduces code size.
</pre>
</div>
</content>
</entry>
<entry>
<title>implement mbtowc directly, not as a wrapper for mbrtowc</title>
<updated>2013-04-09T03:01:32+00:00</updated>
<author>
<name>Rich Felker</name>
<email>dalias@aerifal.cx</email>
</author>
<published>2013-04-09T03:01:32+00:00</published>
<link rel='alternate' type='text/html' href='http://git.musl-libc.org/cgit/musl/commit/?id=ea34b1b90ca5ba3b87725662f6a1ff03b7a88a1f'/>
<id>ea34b1b90ca5ba3b87725662f6a1ff03b7a88a1f</id>
<content type='text'>
the interface contract for mbtowc admits a much faster implementation
than mbrtowc can achieve; wrapping mbrtowc with an extra call frame
only made the situation worse.

since the regex implementation uses mbtowc already, this change should
improve regex performance too. it may be possible to improve
performance in other places internally by switching from mbrtowc to
mbtowc.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
the interface contract for mbtowc admits a much faster implementation
than mbrtowc can achieve; wrapping mbrtowc with an extra call frame
only made the situation worse.

since the regex implementation uses mbtowc already, this change should
improve regex performance too. it may be possible to improve
performance in other places internally by switching from mbrtowc to
mbtowc.
</pre>
</div>
</content>
</entry>
<entry>
<title>optimize mbrtowc</title>
<updated>2013-04-09T02:49:59+00:00</updated>
<author>
<name>Rich Felker</name>
<email>dalias@aerifal.cx</email>
</author>
<published>2013-04-09T02:49:59+00:00</published>
<link rel='alternate' type='text/html' href='http://git.musl-libc.org/cgit/musl/commit/?id=a49e038bab7b3927b6a9c7d0c52f9e1a9cb82629'/>
<id>a49e038bab7b3927b6a9c7d0c52f9e1a9cb82629</id>
<content type='text'>
this simple change, in my measurements, makes about a 7% performance
improvement. at first glance this change would seem like a
compiler-specific hack, since the modified code is not even used.
however, I suspect the reason is that I'm eliminating a second path
into the main body of the code, allowing the compiler more flexibility
to optimize the normal (hot) path into the main body. so even if it
weren't for the measurable (and quite notable) difference in
performance, I think the change makes sense.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
this simple change, in my measurements, makes about a 7% performance
improvement. at first glance this change would seem like a
compiler-specific hack, since the modified code is not even used.
however, I suspect the reason is that I'm eliminating a second path
into the main body of the code, allowing the compiler more flexibility
to optimize the normal (hot) path into the main body. so even if it
weren't for the measurable (and quite notable) difference in
performance, I think the change makes sense.
</pre>
</div>
</content>
</entry>
<entry>
<title>fix out-of-bounds access in UTF-8 decoding</title>
<updated>2013-04-09T02:29:46+00:00</updated>
<author>
<name>Rich Felker</name>
<email>dalias@aerifal.cx</email>
</author>
<published>2013-04-09T02:29:46+00:00</published>
<link rel='alternate' type='text/html' href='http://git.musl-libc.org/cgit/musl/commit/?id=8f06ab0eb9c22998a47803938588bb0b13092a0b'/>
<id>8f06ab0eb9c22998a47803938588bb0b13092a0b</id>
<content type='text'>
SA and SB are used as the lowest and highest valid starter bytes, but
the value of SB was one-past the last valid starter. this caused
access past the end of the state table when the illegal byte '\xf5'
was encountered in a starter position. the error did not show up in
full-character decoding tests, since the bogus state read from just
past the table was unlikely to admit any continuation bytes as valid,
but would have shown up had we tested feeding '\xf5' to the
byte-at-a-time decoding in mbrtowc: it would cause the funtion to
wrongly return -2 rather than -1.

I may eventually go back and remove all references to SA and SB,
replacing them with the values; this would make the code more
transparent, I think. the original motivation for using macros was to
allow misguided users of the code to redefine them for the purpose of
enlarging the set of accepted sequences past the end of Unicode...
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
SA and SB are used as the lowest and highest valid starter bytes, but
the value of SB was one-past the last valid starter. this caused
access past the end of the state table when the illegal byte '\xf5'
was encountered in a starter position. the error did not show up in
full-character decoding tests, since the bogus state read from just
past the table was unlikely to admit any continuation bytes as valid,
but would have shown up had we tested feeding '\xf5' to the
byte-at-a-time decoding in mbrtowc: it would cause the funtion to
wrongly return -2 rather than -1.

I may eventually go back and remove all references to SA and SB,
replacing them with the values; this would make the code more
transparent, I think. the original motivation for using macros was to
allow misguided users of the code to redefine them for the purpose of
enlarging the set of accepted sequences past the end of Unicode...
</pre>
</div>
</content>
</entry>
<entry>
<title>cleanup wcstombs</title>
<updated>2013-04-04T18:55:42+00:00</updated>
<author>
<name>Rich Felker</name>
<email>dalias@aerifal.cx</email>
</author>
<published>2013-04-04T18:55:42+00:00</published>
<link rel='alternate' type='text/html' href='http://git.musl-libc.org/cgit/musl/commit/?id=771c6cead0684487a0699750ff740e3d3894200a'/>
<id>771c6cead0684487a0699750ff740e3d3894200a</id>
<content type='text'>
remove redundant headers and comments; this file is completely trivial
now. also, avoid temp var.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
remove redundant headers and comments; this file is completely trivial
now. also, avoid temp var.
</pre>
</div>
</content>
</entry>
<entry>
<title>cleanup mbstowcs wrapper</title>
<updated>2013-04-04T18:53:53+00:00</updated>
<author>
<name>Rich Felker</name>
<email>dalias@aerifal.cx</email>
</author>
<published>2013-04-04T18:53:53+00:00</published>
<link rel='alternate' type='text/html' href='http://git.musl-libc.org/cgit/musl/commit/?id=b5a527f9ff47b0f4b32c606e385a0c27d861a475'/>
<id>b5a527f9ff47b0f4b32c606e385a0c27d861a475</id>
<content type='text'>
remove unneeded headers. this file is utterly trivial now and there's
no sense in having a comment to state that it's in the public domain.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
remove unneeded headers. this file is utterly trivial now and there's
no sense in having a comment to state that it's in the public domain.
</pre>
</div>
</content>
</entry>
<entry>
<title>minor optimization to mbstowcs</title>
<updated>2013-04-04T18:51:05+00:00</updated>
<author>
<name>Rich Felker</name>
<email>dalias@aerifal.cx</email>
</author>
<published>2013-04-04T18:51:05+00:00</published>
<link rel='alternate' type='text/html' href='http://git.musl-libc.org/cgit/musl/commit/?id=f62b12d051d68aae8bf3c0891f8f809c1fd75e4e'/>
<id>f62b12d051d68aae8bf3c0891f8f809c1fd75e4e</id>
<content type='text'>
there is no need to zero-fill an mbstate_t object in the caller;
mbsrtowcs will automatically treat a null pointer as the initial
state.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
there is no need to zero-fill an mbstate_t object in the caller;
mbsrtowcs will automatically treat a null pointer as the initial
state.
</pre>
</div>
</content>
</entry>
</feed>
