[wp-trac] [WordPress Trac] #29717: wp_check_invalid_utf8 - pcre tricks and failsafes, +mb_convert_encoding, iconv fix, performance
WordPress Trac
noreply at wordpress.org
Fri Oct 17 07:17:33 UTC 2014
#29717: wp_check_invalid_utf8 - pcre tricks and failsafes, +mb_convert_encoding,
iconv fix, performance
--------------------------------+------------------------------------------
Reporter: askapache | Owner:
Type: enhancement | Status: new
Priority: normal | Milestone: Awaiting Review
Component: Formatting | Version: trunk
Severity: normal | Resolution:
Keywords: has-patch dev- | Focuses: administration, performance
feedback |
--------------------------------+------------------------------------------
Comment (by askapache):
Spent a lot of time looking into making this more compatible without
sacrificing the speed gains of the new version. Patch 29717.5.patch
incorporates the results from the benchmarking, significantly it removes
the preg_match custom regex and replaces it with a much quicker and safer
htmlspecialchars to check for validity.
The preg_match's are still a tiny bit faster than htmlspecialchars, but in
my testing they had the exact same results for testing invalid UTF. And
htmlspecialchars is a super core named function from /ext/standard/html.c
and has supported this type of checking since before 5.0.
I also determined in the testing of the new strip code (this version
prefers mb_convert_encoding over iconv, and uses the
mb_substitute_character function instead of ini_set) that the current
wp_check_invalid_utf8 function is definitely broken when using the strip
parameter. It needs the //IGNORE and should also use '@'. As it is now
when you run it with the strip parameter turned on and with an invalid utf
string, it will return boolean false and trigger some PHP Notices.
{{{
PHP Notice: iconv(): Detected an illegal character in input string in
/wp-includes/formatting.php on line 738
PHP Notice: iconv(): Detected an incomplete multibyte character in input
string in /wp-includes/formatting.php on line 738
}}}
----
Just for fun, here are a bunch of notes from researching this stuff.
Please re-test and examine this patch. Below are just notes.
{{{
____ ____ ____ _____ ___ _ _ _____ _____
| _ \ / ___| _ \| ____| ( _ ) | | | |_ _| ___|
| |_) | | | |_) | _| / _ \/\ | | | | | | | |_
| __/| |___| _ <| |___ | (_> < | |_| | | | | _|
|_| \____|_| \_\_____| \___/\/ \___/ |_| |_|
@link http://www.pcre.org/pcre.txt @author Philip Hazel - University of
Cambridge
UTF-8 AND UNICODE PROPERTY SUPPORT
From release 3.3, PCRE has had some support for character strings encoded
in the UTF-8 format. For release 4.0
this was greatly extended to cover most common requirements, and in
release 5.0 additional support for Unicode
general category properties was added.
In order process UTF-8 strings, you must build PCRE to include UTF-8
support in the code, and, in addition,
you must call pcre_compile() with the PCRE_UTF8 option flag. When you do
this, both the pattern and any subject
strings that are matched against it are treated as UTF-8 strings instead
of just strings of bytes.
If you compile PCRE with UTF-8 support, but do not use it at run time, the
library will be a bit bigger, but the
additional run time overhead is limited to testing the PCRE_UTF8 flag
occasionally, so should not be very big.
If you are using PCRE in a non-UTF application that permits users to
supply arbitrary patterns for compilation, you
should be aware of a feature that allows users to turn on UTF support from
within a pattern, provided that PCRE was
built with UTF support. For example, an 8-bit pattern that begins with
"(*UTF8)" or "(*UTF)" turns on UTF-8 mode,
which interprets patterns and subjects as strings of UTF-8 characters
instead of individual 8-bit characters. This
causes both the pattern and any data against which it is matched to be
checked for UTF-8 validity. If the data string
is very long, such a check might use sufficiently many resources as to
cause your application to lose performance.
Alternatively, from release 8.33, you can set the PCRE_NEVER_UTF option at
compile time. This
causes an compile time error if a pattern contains a UTF-setting sequence.
In order process UTF-8 strings, you must build PCRE to include UTF-8
support in the code, and, in addition, you
must call pcre_compile() with the PCRE_UTF8 option flag, or the pattern
must start with the sequence (*UTF8). When
either of these is the case, both the pattern and any subject strings that
are matched against it are treated as
UTF-8 strings instead of strings of 1-byte characters.
VALIDITY OF UTF-8 STRINGS
When you set the PCRE_UTF8 flag, the byte strings passed as patterns and
subjects are (by default) checked for
validity on entry to the relevant functions. The entire string is checked
before any other processing takes
place. From release 7.3 of PCRE, the check is according the rules of RFC
3629, which are themselves derived from
the Unicode specification. Earlier releases of PCRE followed the rules of
RFC 2279, which allows the full range
of 31-bit values (0 to 0x7FFFFFFF). The current check allows only values
in the range U+0 to U+10FFFF, excluding
the surrogate area. (From release 8.33 the so-called "non-character" code
points are no longer excluded because
Unicode corrigendum #9 makes it clear that they should not be.)
Characters in the "Surrogate Area" of Unicode are reserved for use by
UTF-16, where they are used in pairs to
encode codepoints with values greater than 0xFFFF. The code points that
are encoded by UTF-16 pairs are available
independently in the UTF-8 and UTF-32 encodings. (In other words, the
whole surrogate thing is a fudge for UTF-16
which unfortunately messes up UTF-8 and UTF-32.)
If an invalid UTF-8 string is passed to PCRE, an error return is given.
___ ___ ___ ___ ___ _ _
| _ \/ __| _ \ __| / __| |_ __ _ _ _ __ _ ___| |___ __ _
| _/ (__| / _| | (__| ' \/ _` | ' \/ _` / -_) / _ \/ _` |
|_| \___|_|_\___| \___|_||_\__,_|_||_\__, \___|_\___/\__, |
|___/ |___/
// Release 8.33 28-May-2013
Version 8.33 28-May-2013
---------------------
00. (*LIMIT_MATCH=d), (*LIMIT_RECURSION=d) added so the pattern can
specify lower limits for the matching process.
35. Implement PCRE_NEVER_UTF to lock out the use of UTF, in particular,
blocking (*UTF) etc.
Version 8.32 30-November-2012
---------------------
14. Applied user-supplied patch to pcrecpp.cc to allow PCRE_NO_UTF8_CHECK
to be set
24. Add support for 32-bit character strings, and UTF-32
25. (*UTF) can now be used to start a pattern in any of the three
libraries.
30. In 8-bit UTF-8 mode, pcretest failed to give an error for data
codepoints greater than 0x7fffffff (which cannot be
represented in UTF-8, even under the "old" RFC 2279). Instead, it
ended up passing a negative length to pcre_exec()
Version 7.9 11-Apr-09
---------------------
28. Added support for (*UTF8) at the start of a pattern.
Version 7.3 28-Aug-07
---------------------
15. Updated the test for a valid UTF-8 string to conform to the later RFC
3629.
This restricts code points to be within the range 0 to 0x10FFFF,
excluding
the "low surrogate" sequence 0xD800 to 0xDFFF. Previously, PCRE
allowed the
full range 0 to 0x7FFFFFFF, as defined by RFC 2279. Internally, it
still
does: it's just the validity check that is more restrictive.
Version 4.4 21-Aug-03
---------------------
15. Updated the test for a valid UTF-8 string to conform to the later RFC
3629.
PCRE checks UTF-8 strings for validity by default. There is an option to
suppress
this, just in case anybody wants that teeny extra bit of performance.
Version 4.4 13-Aug-03
---------------------
10. By default, when in UTF-8 mode, PCRE now checks for valid UTF-8
strings at
both compile and run time, and gives an error if an invalid UTF-8
sequence
is found. There is a option for disabling this check in cases where
the
string is known to be correct and/or the maximum performance is
wanted.
Version 3.3 01-Aug-00
---------------------
7. Added the beginnings of support for UTF-8 character strings.
PCRE PHP)INI CONFIGURATION OPTIONS
@link http://php.net/manual/en/pcre.configuration.php "PCRE Configuration
Options"
2 PCRE INI options are available since PHP 5.2.0
pcre.backtrack_limit 1000000
PCRE's backtracking limit. Defaults to 100000 for PHP < 5.3.7.
pcre.recursion_limit 100000
PCRE's recursion limit. Please note that if you set this value too
high you may consume all the available
process stack and eventually crash PHP (due to reaching the stack size
limit imposed by the OS).
PCRE CRASHES FROM REGEXES
// Release 8.33 28-May-2013
// (*LIMIT_MATCH=d) and (*LIMIT_RECURSION=d) have been added so that the
creator of a pattern can specify lower (but not higher) limits for the
matching process.
PCRE_EXTRA_MATCH_LIMIT can be accessed through the set_match_limit()
and match_limit() member functions. Setting match_limit to a non-zero
value will limit the execution of
pcre to keep it from doing bad things like blowing the stack or taking an
eternity to return a result. A value
of 5000 is good enough to stop stack blowup in a 2MB thread stack. Setting
match_limit to zero disables match
limiting. Alternatively, you can call match_limit_recursion() which uses
PCRE_EXTRA_MATCH_LIMIT_RECURSION to limit
how much PCRE recurses. match_limit() limits the number of matches PCRE
does; match_limit_recursion() limits the
depth of internal recursion, and therefore the amount of stack that is
used.
The match_limit field provides a means of preventing PCRE from using up a
vast amount of resources when running
patterns that are not going to match, but which have a very large number
of possibilities in their search trees. The
classic example is the use of nested unlimited repeats.
Internally, PCRE uses a function called match() which it calls repeatedly
(sometimes recursively). The limit set
by match_limit is imposed on the number of times this function is called
during a match, which has the effect of
limiting the amount of backtracking that can take place. For patterns that
are not anchored, the count restarts
from zero for each position in the subject string.
The default value for the limit can be set when PCRE is built; the default
default is 10 million, which handles all
but the most extreme cases. You can override the default by suppling
pcre_exec() with a pcre_extra block in which
match_limit is set, and PCRE_EXTRA_MATCH_LIMIT is set in the flags field.
If the limit is exceeded, pcre_exec()
returns PCRE_ERROR_MATCHLIMIT.
The match_limit_recursion field is similar to match_limit, but instead of
limiting the total number of times
that match() is called, it limits the depth of recursion. The recursion
depth is a smaller number than the total
number of calls, because not all calls to match() are recursive. This
limit is of use only if it is set smaller
than match_limit.
Limiting the recursion depth limits the amount of stack that can be used,
or, when PCRE has been compiled to use
memory on the heap instead of the stack, the amount of heap memory that
can be used.
The default value for match_limit_recursion can be set when PCRE is built;
the default default is the same value
as the default for match_limit. You can override the default by suppling
pcre_exec() with a pcre_extra block in
which match_limit_recursion is set, and PCRE_EXTRA_MATCH_LIMIT_RECURSION
is set in the flags field. If the limit
is exceeded, pcre_exec() returns PCRE_ERROR_RECURSIONLIMIT.
_ _ ____
_ __ _ _ ___ __ _ _ __ __ _| |_ __| |_ / /\ \
| '_ \ '_/ -_) _` | | ' \/ _` | _/ _| ' \| | | |
| .__/_| \___\__, |_|_|_|_\__,_|\__\__|_||_| | | |
|_| |___/___| \_\/_/
preg_match() returns 1 if the pattern matches given subject, 0 if it does
not, or FALSE if an error occurred.
u (PCRE_UTF8) This modifier turns on additional functionality of PCRE that
is incompatible with Perl. Pattern and
subject strings are treated as UTF-8. This modifier is available from PHP
4.1.0 or greater on Unix and from PHP
4.2.3 on win32. UTF-8 validity of the pattern and the subject is checked
since PHP 4.3.5. An invalid subject will
cause the preg_* function to match nothing; an invalid pattern will
trigger an error of level E_WARNING. Five and
six octet UTF-8 sequences are regarded as invalid since PHP 5.3.4 (resp.
PCRE 7.3 2007-08-28); formerly those have
been regarded as valid UTF-8.
With the PCRE_UTF8 modifier 'u', preg_match() fails silently on strings
containing invalid UTF-8 byte sequences. It
does not reject character codes above U+10FFFF (represented by 4 or more
octets), though.
Originally, this function checked according to RFC 2279, allowing for
values in the range 0 to 0x7fffffff, up to 6
bytes long, but ensuring that they were in the canonical format. Once
somebody had pointed out RFC 3629 to me (it
obsoletes 2279), additional restrictions were applied. The values are now
limited to be between 0 and 0x0010ffff,
no more than 4 bytes long, and the subrange 0xd000 to 0xdfff is excluded.
However, the format of 5-byte and 6-byte
characters is still checked.
BACKTRACKING CONTROL
The following are recognized only at the start of a pattern:
(*LIMIT_MATCH=d) set the match limit to d (decimal number) ( added 8.33
28-May-2013 )
(*LIMIT_RECURSION=d) set the recursion limit to d (decimal number) ( added
8.33 28-May-2013 )
(*UTF8) set UTF-8 mode: 8-bit library (PCRE_UTF8) ( added 7.9 11-Apr-09 )
(*UTF16) set UTF-16 mode: 16-bit library (PCRE_UTF16) ( added 7.9
11-Apr-09 )
(*UTF32) set UTF-32 mode: 32-bit library (PCRE_UTF32) ( added 7.9
11-Apr-09 )
(*UTF) set appropriate UTF mode for the library in use ( added 7.9
11-Apr-09 )
In order process UTF-8 strings, you must build PCRE's 8-bit library with
UTF support, and, in addition, you
must call pcre_compile() with the PCRE_UTF8 option flag, or the pattern
must start with the sequence (*UTF8) or
(*UTF). When either of these is the case, both the pattern and any subject
strings that are matched against it
are treated as UTF-8 strings instead of strings of individual 1-byte
characters.
PCRE UTF ERRORS
From release 8.13 more information about the details of the error are
passed back in the returned value:
PCRE_UTF8_ERR0 No error
PCRE_UTF8_ERR1 Missing 1 byte at the end of the string
PCRE_UTF8_ERR2 Missing 2 bytes at the end of the string
PCRE_UTF8_ERR3 Missing 3 bytes at the end of the string
PCRE_UTF8_ERR4 Missing 4 bytes at the end of the string
PCRE_UTF8_ERR5 Missing 5 bytes at the end of the string
PCRE_UTF8_ERR6 2nd-byte's two top bits are not 0x80
PCRE_UTF8_ERR7 3rd-byte's two top bits are not 0x80
PCRE_UTF8_ERR8 4th-byte's two top bits are not 0x80
PCRE_UTF8_ERR9 5th-byte's two top bits are not 0x80
PCRE_UTF8_ERR10 6th-byte's two top bits are not 0x80
PCRE_UTF8_ERR11 5-byte character is not permitted by RFC 3629
PCRE_UTF8_ERR12 6-byte character is not permitted by RFC 3629
PCRE_UTF8_ERR13 4-byte character with value > 0x10ffff is not permitted
PCRE_UTF8_ERR14 3-byte character with value 0xd000-0xdfff is not permitted
PCRE_UTF8_ERR15 Overlong 2-byte sequence
PCRE_UTF8_ERR16 Overlong 3-byte sequence
PCRE_UTF8_ERR17 Overlong 4-byte sequence
PCRE_UTF8_ERR18 Overlong 5-byte sequence (won't ever occur)
PCRE_UTF8_ERR19 Overlong 6-byte sequence (won't ever occur)
PCRE_UTF8_ERR20 Isolated 0x80 byte (not within UTF-8 character)
PCRE_UTF8_ERR21 Byte with the illegal value 0xfe or 0xff
PCRE_UTF8_ERR22 Unused (was non-character)
PHP PCRE CONSTANTS
PREG_NO_ERROR Returned by preg_last_error() if there were no errors.
5.2.0
PREG_INTERNAL_ERROR Returned by preg_last_error() if there was an
internal PCRE error. 5.2.0
PREG_BACKTRACK_LIMIT_ERROR Returned by preg_last_error() if backtrack
limit was exhausted. 5.2.0
PREG_RECURSION_LIMIT_ERROR Returned by preg_last_error() if recursion
limit was exhausted. 5.2.0
PREG_BAD_UTF8_ERROR Returned by preg_last_error() if the last error was
caused by malformed UTF-8 data (only when
running a regex in UTF-8 mode). 5.2.0
PREG_BAD_UTF8_OFFSET_ERROR Returned by preg_last_error() if the offset
didn't correspond to the begin of a valid
UTF-8 code point (only when running a regex in
UTF-8 mode). 5.3.0
PCRE_VERSION PCRE version and release date (e.g. "7.0 18-Dec-2006").
5.2.4
PCRE CONSTANTS ON MY INSTALL get_defined_constants()
PREG_PATTERN_ORDER' => 1,
PREG_SET_ORDER' => 2,
PREG_OFFSET_CAPTURE' => 256,
PREG_SPLIT_NO_EMPTY' => 1,
PREG_SPLIT_DELIM_CAPTURE' => 2,
PREG_SPLIT_OFFSET_CAPTURE' => 4,
PREG_GREP_INVERT' => 1,
PREG_NO_ERROR' => 0,
PREG_INTERNAL_ERROR' => 1,
PREG_BACKTRACK_LIMIT_ERROR' => 2,
PREG_RECURSION_LIMIT_ERROR' => 3,
PREG_BAD_UTF8_ERROR' => 4,
PREG_BAD_UTF8_OFFSET_ERROR' => 5,
PCRE_VERSION' => '8.34 2013-12-15',
_ ____
(_)__ ___ _ ___ __/ /\ \
| / _/ _ \ ' \ V / | | |
|_\__\___/_||_\_/| | | |
\_\/_/
https://www.gnu.org/software/libiconv/
If you append the string //IGNORE, characters that cannot be represented
in the target charset are silently discarded.
Otherwise, str is cut from the first illegal character and an E_NOTICE is
generated. ( since GNU libiconv 2002-01-13 )
In other words, iconv() appears to be intended for use when converting the
contents of files - whereas mb_convert_encoding() is intended
for use when juggling strings internally, e.g. strings that aren't being
read/written to/from files, but exchanged with some other media.
ICONV CHARACTER SET ENCODINGS CONTAINING "UTF"
$ iconv -l
- ISO-10646UTF-8
- ISO-10646UTF8
- UTF-7
- UTF-8
- UTF-16
- UTF-16BE
- UTF-16LE
- UTF-32
- UTF-32BE
- UTF-32LE
- UTF7
- UTF8
- UTF16
- UTF16BE
- UTF16LE
- UTF32
- UTF32BE
- UTF32LE
If the string //IGNORE is appended to to-encoding, characters that cannot
be converted are discarded and an error is printed after conversion.
ICONV IMPLEMENTATIONS - ICONV_IMPL CONSTANT
@link http://www.gnu.org/software/libc/manual/html_node/Other-iconv-
Implementations.html "Some Details about other iconv Implementations"
@link http://www.gnu.org/software/libc/manual/html_node/Locales.html
"Locales and Internationalization"
"libiconv" - GNU libiconv is the native FreeBSD iconv implementation since
2002.
"BSD iconv" - Konstantin Chugeuv's iconv
"glibc" - GNU Glibc's
"unknown" - Not one of the above
}}}
--
Ticket URL: <https://core.trac.wordpress.org/ticket/29717#comment:11>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform
More information about the wp-trac
mailing list