<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[59076] trunk: HTML API: Add `normalize()` to give us the HTML we always wanted.</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { white-space: pre-line; overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta" style="font-size: 105%">
<dt style="float: left; width: 6em; font-weight: bold">Revision</dt> <dd><a style="font-weight: bold" href="https://core.trac.wordpress.org/changeset/59076">59076</a><script type="application/ld+json">{"@context":"http://schema.org","@type":"EmailMessage","description":"Review this Commit","action":{"@type":"ViewAction","url":"https://core.trac.wordpress.org/changeset/59076","name":"Review Commit"}}</script></dd>
<dt style="float: left; width: 6em; font-weight: bold">Author</dt> <dd>dmsnell</dd>
<dt style="float: left; width: 6em; font-weight: bold">Date</dt> <dd>2024-09-20 22:30:04 +0000 (Fri, 20 Sep 2024)</dd>
</dl>
<pre style='padding-left: 1em; margin: 2em 0; border-left: 2px solid #ccc; line-height: 1.25; font-size: 105%; font-family: sans-serif'>HTML API: Add `normalize()` to give us the HTML we always wanted.
HTML often appears in ways that are unexpected. It may be missing implicit tags, may have unquoted, single-quoted, or double-quoted attributes, may contain duplicate attributes, may contain unescaped text content, or any number of other possible invalid constructions. The HTML API understands all fo these inputs, but downline parsers may not, and HTML snippets which are safe on their own may introduce problems when joined with other HTML snippets.
This patch introduces the `serialize()` method on the HTML Processor, which prints a fully-normative HTML output, eliminating invalid markup along the way. It produces a string which contains every missing tag, double-quoted attributes, and no duplicates. A `normalize()` static method on the HTML Processor provides a convenient wrapper for constructing a fragment parser and immediately serializing.
Subclasses relying on the `serialize_token()` method may perform structural HTML modifications with as much security as the upcoming `\Dom\HTMLDocument()` parser will, though these are not
able to provide the full safety that will eventually appear with `set_inner_html()`.
Further work may explore serializing to XML (which involves a number of other important transformations) and adding constraints to serialization (such as only allowing inline/flow/formatting elements and text).
Developed in https://github.com/wordpress/wordpress-develop/pull/7331
Discussed in https://core.trac.wordpress.org/ticket/62036
Props dmsnell, jonsurrell, westonruter.
Fixes <a href="https://core.trac.wordpress.org/ticket/62036">#62036</a>.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunksrcwpincludeshtmlapiclasswphtmlprocessorphp">trunk/src/wp-includes/html-api/class-wp-html-processor.php</a></li>
<li><a href="#trunksrcwpincludeshtmlapiclasswphtmltagprocessorphp">trunk/src/wp-includes/html-api/class-wp-html-tag-processor.php</a></li>
</ul>
<h3>Added Paths</h3>
<ul>
<li><a href="#trunktestsphpunittestshtmlapiwpHtmlProcessorserializephp">trunk/tests/phpunit/tests/html-api/wpHtmlProcessor-serialize.php</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunksrcwpincludeshtmlapiclasswphtmlprocessorphp"></a>
<div class="modfile"><h4 style="background-color: #eee; color: inherit; margin: 1em 0; padding: 1.3em; font-size: 115%">Modified: trunk/src/wp-includes/html-api/class-wp-html-processor.php</h4>
<pre class="diff"><span>
<span class="info" style="display: block; padding: 0 10px; color: #888">--- trunk/src/wp-includes/html-api/class-wp-html-processor.php 2024-09-20 20:21:59 UTC (rev 59075)
+++ trunk/src/wp-includes/html-api/class-wp-html-processor.php 2024-09-20 22:30:04 UTC (rev 59076)
</span><span class="lines" style="display: block; padding: 0 10px; color: #888">@@ -1030,6 +1030,216 @@
</span><span class="cx" style="display: block; padding: 0 10px"> }
</span><span class="cx" style="display: block; padding: 0 10px">
</span><span class="cx" style="display: block; padding: 0 10px"> /**
</span><ins style="background-color: #dfd; text-decoration:none; display:block; padding: 0 10px">+ * Normalizes an HTML fragment by serializing it.
+ *
+ * This method assumes that the given HTML snippet is found in BODY context.
+ * For normalizing full documents or fragments found in other contexts, create
+ * a new processor using {@see WP_HTML_Processor::create_fragment} or
+ * {@see WP_HTML_Processor::create_full_parser} and call {@see WP_HTML_Processor::serialize}
+ * on the created instances.
+ *
+ * Many aspects of an input HTML fragment may be changed during normalization.
+ *
+ * - Attribute values will be double-quoted.
+ * - Duplicate attributes will be removed.
+ * - Omitted tags will be added.
+ * - Tag and attribute name casing will be lower-cased,
+ * except for specific SVG and MathML tags or attributes.
+ * - Text will be re-encoded, null bytes handled,
+ * and invalid UTF-8 replaced with U+FFFD.
+ * - Any incomplete syntax trailing at the end will be omitted,
+ * for example, an unclosed comment opener will be removed.
+ *
+ * Example:
+ *
+ * echo WP_HTML_Processor::normalize( '<a href=#anchor v=5 href="/" enabled>One</a another v=5><!--' );
+ * // <a href="#anchor" v="5" enabled>One</a>
+ *
+ * echo WP_HTML_Processor::normalize( '<div></p>fun<table><td>cell</div>' );
+ * // <div><p></p>fun<table><tbody><tr><td>cell</td></tr></tbody></table></div>
+ *
+ * echo WP_HTML_Processor::normalize( '<![CDATA[invalid comment]]> syntax < <> "oddities"' );
+ * // <!--[CDATA[invalid comment]]--> syntax < <> "oddities"
+ *
+ * @since 6.7.0
+ *
+ * @param string $html Input HTML to normalize.
+ *
+ * @return string|null Normalized output, or `null` if unable to normalize.
+ */
+ public static function normalize( string $html ): ?string {
+ return static::create_fragment( $html )->serialize();
+ }
+
+ /**
+ * Returns normalized HTML for a fragment by serializing it.
+ *
+ * This differs from {@see WP_HTML_Processor::normalize} in that it starts with
+ * a specific HTML Processor, which _must_ not have already started scanning;
+ * it must be in the initial ready state and will be in the completed state once
+ * serialization is complete.
+ *
+ * Many aspects of an input HTML fragment may be changed during normalization.
+ *
+ * - Attribute values will be double-quoted.
+ * - Duplicate attributes will be removed.
+ * - Omitted tags will be added.
+ * - Tag and attribute name casing will be lower-cased,
+ * except for specific SVG and MathML tags or attributes.
+ * - Text will be re-encoded, null bytes handled,
+ * and invalid UTF-8 replaced with U+FFFD.
+ * - Any incomplete syntax trailing at the end will be omitted,
+ * for example, an unclosed comment opener will be removed.
+ *
+ * Example:
+ *
+ * $processor = WP_HTML_Processor::create_fragment( '<a href=#anchor v=5 href="/" enabled>One</a another v=5><!--' );
+ * echo $processor->serialize();
+ * // <a href="#anchor" v="5" enabled>One</a>
+ *
+ * $processor = WP_HTML_Processor::create_fragment( '<div></p>fun<table><td>cell</div>' );
+ * echo $processor->serialize();
+ * // <div><p></p>fun<table><tbody><tr><td>cell</td></tr></tbody></table></div>
+ *
+ * $processor = WP_HTML_Processor::create_fragment( '<![CDATA[invalid comment]]> syntax < <> "oddities"' );
+ * echo $processor->serialize();
+ * // <!--[CDATA[invalid comment]]--> syntax < <> "oddities"
+ *
+ * @since 6.7.0
+ *
+ * @return string|null Normalized HTML markup represented by processor,
+ * or `null` if unable to generate serialization.
+ */
+ public function serialize(): ?string {
+ if ( WP_HTML_Tag_Processor::STATE_READY !== $this->parser_state ) {
+ wp_trigger_error(
+ __METHOD__,
+ "An HTML Processor which has already started processing cannot serialize it's contents. Serialize immediately after creating the instance.",
+ E_USER_WARNING
+ );
+ return null;
+ }
+
+ $html = '';
+ while ( $this->next_token() ) {
+ $html .= $this->serialize_token();
+ }
+
+ if ( null !== $this->get_last_error() ) {
+ wp_trigger_error(
+ __METHOD__,
+ "Cannot serialize HTML Processor with parsing error: {$this->get_last_error()}.",
+ E_USER_WARNING
+ );
+ return null;
+ }
+
+ return $html;
+ }
+
+ /**
+ * Serializes the currently-matched token.
+ *
+ * This method produces a fully-normative HTML string for the currently-matched token,
+ * if able. If not matched at any token or if the token doesn't correspond to any HTML
+ * it will return an empty string (for example, presumptuous end tags are ignored).
+ *
+ * @see static::serialize()
+ *
+ * @since 6.7.0
+ *
+ * @return string Serialization of token, or empty string if no serialization exists.
+ */
+ protected function serialize_token(): string {
+ $html = '';
+ $token_type = $this->get_token_type();
+
+ switch ( $token_type ) {
+ case '#text':
+ $html .= htmlspecialchars( $this->get_modifiable_text(), ENT_QUOTES | ENT_SUBSTITUTE | ENT_HTML5, 'UTF-8' );
+ break;
+
+ // Unlike the `<>` which is interpreted as plaintext, this is ignored entirely.
+ case '#presumptuous-tag':
+ break;
+
+ case '#funky-comment':
+ case '#comment':
+ $html .= "<!--{$this->get_full_comment_text()}-->";
+ break;
+
+ case '#cdata-section':
+ $html .= "<![CDATA[{$this->get_modifiable_text()}]]>";
+ break;
+
+ case 'html':
+ $html .= '<!DOCTYPE html>';
+ break;
+ }
+
+ if ( '#tag' !== $token_type ) {
+ return $html;
+ }
+
+ $tag_name = str_replace( "\x00", "\u{FFFD}", $this->get_tag() );
+ $in_html = 'html' === $this->get_namespace();
+ $qualified_name = $in_html ? strtolower( $tag_name ) : $this->get_qualified_tag_name();
+
+ if ( $this->is_tag_closer() ) {
+ $html .= "</{$qualified_name}>";
+ return $html;
+ }
+
+ $attribute_names = $this->get_attribute_names_with_prefix( '' );
+ if ( ! isset( $attribute_names ) ) {
+ $html .= "<{$qualified_name}>";
+ return $html;
+ }
+
+ $html .= "<{$qualified_name}";
+ foreach ( $attribute_names as $attribute_name ) {
+ $html .= " {$this->get_qualified_attribute_name( $attribute_name )}";
+ $value = $this->get_attribute( $attribute_name );
+
+ if ( is_string( $value ) ) {
+ $html .= '="' . htmlspecialchars( $value, ENT_QUOTES | ENT_SUBSTITUTE | ENT_HTML5 ) . '"';
+ }
+
+ $html = str_replace( "\x00", "\u{FFFD}", $html );
+ }
+
+ if ( ! $in_html && $this->has_self_closing_flag() ) {
+ $html .= ' /';
+ }
+
+ $html .= '>';
+
+ // Flush out self-contained elements.
+ if ( $in_html && in_array( $tag_name, array( 'IFRAME', 'NOEMBED', 'NOFRAMES', 'SCRIPT', 'STYLE', 'TEXTAREA', 'TITLE', 'XMP' ), true ) ) {
+ $text = $this->get_modifiable_text();
+
+ switch ( $tag_name ) {
+ case 'IFRAME':
+ case 'NOEMBED':
+ case 'NOFRAMES':
+ $text = '';
+ break;
+
+ case 'SCRIPT':
+ case 'STYLE':
+ break;
+
+ default:
+ $text = htmlspecialchars( $text, ENT_QUOTES | ENT_SUBSTITUTE | ENT_HTML5, 'UTF-8' );
+ }
+
+ $html .= "{$text}</{$qualified_name}>";
+ }
+
+ return $html;
+ }
+
+ /**
</ins><span class="cx" style="display: block; padding: 0 10px"> * Parses next element in the 'initial' insertion mode.
</span><span class="cx" style="display: block; padding: 0 10px"> *
</span><span class="cx" style="display: block; padding: 0 10px"> * This internal function performs the 'initial' insertion mode
</span></span></pre></div>
<a id="trunksrcwpincludeshtmlapiclasswphtmltagprocessorphp"></a>
<div class="modfile"><h4 style="background-color: #eee; color: inherit; margin: 1em 0; padding: 1.3em; font-size: 115%">Modified: trunk/src/wp-includes/html-api/class-wp-html-tag-processor.php</h4>
<pre class="diff"><span>
<span class="info" style="display: block; padding: 0 10px; color: #888">--- trunk/src/wp-includes/html-api/class-wp-html-tag-processor.php 2024-09-20 20:21:59 UTC (rev 59075)
+++ trunk/src/wp-includes/html-api/class-wp-html-tag-processor.php 2024-09-20 22:30:04 UTC (rev 59076)
</span><span class="lines" style="display: block; padding: 0 10px; color: #888">@@ -1984,6 +1984,9 @@
</span><span class="cx" style="display: block; padding: 0 10px"> * [#x10000-#xEFFFF]
</span><span class="cx" style="display: block; padding: 0 10px"> * > NameChar ::= NameStartChar | "-" | "." | [0-9] | #xB7 | [#x0300-#x036F] | [#x203F-#x2040]
</span><span class="cx" style="display: block; padding: 0 10px"> *
</span><ins style="background-color: #dfd; text-decoration:none; display:block; padding: 0 10px">+ * @todo Processing instruction nodes in SGML may contain any kind of markup. XML defines a
+ * special case with `<?xml ... ?>` syntax, but the `?` is part of the bogus comment.
+ *
</ins><span class="cx" style="display: block; padding: 0 10px"> * @see https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-PITarget
</span><span class="cx" style="display: block; padding: 0 10px"> */
</span><span class="cx" style="display: block; padding: 0 10px"> if ( $this->token_length >= 5 && '?' === $html[ $closer_at - 1 ] ) {
</span></span></pre></div>
<a id="trunktestsphpunittestshtmlapiwpHtmlProcessorserializephp"></a>
<div class="addfile"><h4 style="background-color: #eee; color: inherit; margin: 1em 0; padding: 1.3em; font-size: 115%">Added: trunk/tests/phpunit/tests/html-api/wpHtmlProcessor-serialize.php</h4>
<pre class="diff"><span>
<span class="info" style="display: block; padding: 0 10px; color: #888">--- trunk/tests/phpunit/tests/html-api/wpHtmlProcessor-serialize.php (rev 0)
+++ trunk/tests/phpunit/tests/html-api/wpHtmlProcessor-serialize.php 2024-09-20 22:30:04 UTC (rev 59076)
</span><span class="lines" style="display: block; padding: 0 10px; color: #888">@@ -0,0 +1,287 @@
</span><ins style="background-color: #dfd; text-decoration:none; display:block; padding: 0 10px">+<?php
+/**
+ * Unit tests covering WP_HTML_Processor serialization functionality.
+ *
+ * @package WordPress
+ * @subpackage HTML-API
+ *
+ * @since 6.7.0
+ */
+
+/**
+ * @group html-api
+ *
+ * @coversDefaultClass WP_HTML_Processor
+ */
+class Tests_HtmlApi_WpHtmlProcessor_Serialize extends WP_UnitTestCase {
+ /**
+ * Ensures that basic text is properly encoded when serialized.
+ *
+ * @ticket 62036
+ */
+ public function test_properly_encodes_text() {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( "apples > or\x00anges" ),
+ 'apples > oranges',
+ 'Should have returned an HTML string with applicable characters properly encoded.'
+ );
+ }
+
+ /**
+ * Ensures that unclosed elements are explicitly closed to ensure proper HTML isolation.
+ *
+ * When thinking about embedding HTML fragments into others, it's important that unclosed
+ * elements aren't left dangling, otherwise a snippet of HTML may "swallow" parts of the
+ * document that follow it.
+ *
+ * @ticket 62036
+ */
+ public function test_closes_unclosed_elements_at_end() {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( '<div>' ),
+ '<div></div>',
+ 'Should have provided the explicit closer to the un-closed DIV element.'
+ );
+ }
+
+ /**
+ * Ensures that boolean attributes remain boolean and do not gain values.
+ *
+ * @ticket 62036
+ */
+ public function test_boolean_attributes_remain_boolean() {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( '<input disabled>' ),
+ '<input disabled>',
+ 'Should have preserved the boolean attribute upon serialization.'
+ );
+ }
+
+ /**
+ * Ensures that attributes with values result in double-quoted attribute values.
+ *
+ * @ticket 62036
+ */
+ public function test_attributes_are_double_quoted() {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( '<p id=3></p>' ),
+ '<p id="3"></p>',
+ 'Should double-quote all attribute values.'
+ );
+ }
+
+ /**
+ * Ensures that self-closing flags on HTML void elements are not serialized, to
+ * prevent risk of conflating the flag with unquoted attribute values.
+ *
+ * Example:
+ *
+ * BR element with "class" attribute having value "clear"
+ * <br class="clear"/>
+ *
+ * BR element with "class" attribute having value "clear"
+ * <br class=clear />
+ *
+ * BR element with "class" attribute having value "clear/"
+ * <br class=clear/>
+ *
+ * @ticket 62036
+ */
+ public function test_void_elements_get_no_dangerous_self_closing_flag() {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( '<br class="clear"/>' ),
+ '<br class="clear">',
+ 'Should have removed dangerous self-closing flag on HTML void element.'
+ );
+ }
+
+ /**
+ * Ensures that duplicate attributes are removed upon serialization.
+ *
+ * @ticket 62036
+ */
+ public function test_duplicate_attributes_are_removed() {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( '<div one=1 one="one" one=\'won\' one>' ),
+ '<div one="1"></div>',
+ 'Should have removed all but the first copy of an attribute when duplicates exist.'
+ );
+ }
+
+ /**
+ * Ensures that SCRIPT contents are not escaped, as they are not parsed like text nodes are.
+ *
+ * @ticket 62036
+ */
+ public function test_script_contents_are_not_escaped() {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( "<script>apples > or\x00anges</script>" ),
+ "<script>apples > or\u{FFFD}anges</script>",
+ 'Should have preserved text inside a SCRIPT element, except for replacing NULL bytes.'
+ );
+ }
+
+ /**
+ * Ensures that STYLE contents are not escaped, as they are not parsed like text nodes are.
+ *
+ * @ticket 62036
+ */
+ public function test_style_contents_are_not_escaped() {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( "<style>apples > or\x00anges</style>" ),
+ "<style>apples > or\u{FFFD}anges</style>",
+ 'Should have preserved text inside a STYLE element, except for replacing NULL bytes.'
+ );
+ }
+
+ public function test_unexpected_closing_tags_are_removed() {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( 'one</div>two</span>three' ),
+ 'onetwothree',
+ 'Should have removed unpected closing tags.'
+ );
+ }
+
+ /**
+ * Ensures that self-closing elements in foreign content retain their self-closing flag.
+ *
+ * @ticket 62036
+ */
+ public function test_self_closing_foreign_elements_retain_their_self_closing_flag() {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( '<svg><g><g /></svg>' ),
+ '<svg><g><g /></g></svg>',
+ 'Should have closed unclosed G element, but preserved the self-closing nature of the other G element.'
+ );
+ }
+
+ /**
+ * Ensures that incomplete syntax elements at the end of an HTML string are removed from
+ * the serialization, since these are often vectors of exploits for the successive HTML.
+ *
+ * @ticket 62036
+ *
+ * @dataProvider data_incomplete_syntax_tokens
+ *
+ * @param string $incomplete_token An incomplete HTML syntax token.
+ */
+ public function test_should_remove_incomplete_input_from_end( string $incomplete_token ) {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( "content{$incomplete_token}" ),
+ 'content',
+ 'Should have removed the incomplete token from the end of the input.'
+ );
+ }
+
+ /**
+ * Data provider.
+ *
+ * @return array[]
+ */
+ public static function data_incomplete_syntax_tokens() {
+ return array(
+ 'Comment opener' => array( '<!--' ),
+ 'Bogus comment opener' => array( '<![sneaky[' ),
+ 'Incomplete tag' => array( '<my-custom status="pending"' ),
+ 'SCRIPT opening tag' => array( '<script>' ),
+ );
+ }
+
+ /**
+ * Ensures that presumptuous tag openers are treated as plaintext.
+ *
+ * @ticket 62036
+ */
+ public function test_encodes_presumptuous_opening_tags() {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( '<>' ),
+ '<>',
+ 'Should have encoded the invalid presumptuous opening tag as plaintext.'
+ );
+ }
+
+ /**
+ * Ensures that presumptuous tag closers are skipped in serialization.
+ *
+ * @ticket 62036
+ */
+ public function test_skips_presumptuous_closing_tags() {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( '</>' ),
+ '',
+ 'Should have completely ignored the presumptuous tag closer.'
+ );
+ }
+
+ /**
+ * Ensures that invalid or "bogus" comments in HTML are normalized to their proper normative form.
+ *
+ * @ticket 62036
+ *
+ * @dataProvider data_bogus_comments
+ *
+ * @param string $opening Start of bogus comment, e.g. "<!".
+ * @param string $comment_text Comment content, as reported in a browser.
+ * @param string $closing End of bogus comment, e.g. ">".
+ */
+ public function test_normalizes_bogus_comment_forms( string $opening, string $comment_text, string $closing ) {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( "{$opening}{$comment_text}{$closing}" ),
+ "<!--{$comment_text}-->",
+ 'Should have replaced the invalid comment syntax with normative syntax.'
+ );
+ }
+
+ /**
+ * Data provider.
+ *
+ * @return array[]
+ */
+ public function data_bogus_comments() {
+ return array(
+ 'False DOCTYPE' => array( '<!', 'html', '>' ),
+ 'CDATA look-alike' => array( '<!', '[CDATA[inside]]', '>' ),
+ 'Immediately-closed markup instruction' => array( '<!', '?', '>' ),
+ 'Warning Symbol' => array( '<!', '', '>' ),
+ 'PHP block look-alike' => array( '<', '?php foo(); ?', '>' ),
+ 'Funky comment' => array( '</', '%display-name', '>' ),
+ 'XML Processing Instruction look-alike' => array( '<', '?xml foo ', '>' ),
+ );
+ }
+
+ /**
+ * Ensures that NULL bytes are properly handled.
+ *
+ * @ticket 62036
+ *
+ * @dataProvider data_tokens_with_null_bytes
+ *
+ * @param string $html_with_nulls HTML token containing NULL bytes in various places.
+ * @param string $expected_output Expected parse of HTML after handling NULL bytes.
+ */
+ public function test_replaces_null_bytes_appropriately( string $html_with_nulls, string $expected_output ) {
+ $this->assertSame(
+ WP_HTML_Processor::normalize( $html_with_nulls ),
+ $expected_output,
+ 'Should have properly replaced or removed NULL bytes.'
+ );
+ }
+
+ /**
+ * Data provider.
+ *
+ * @return array[]
+ */
+ public static function data_tokens_with_null_bytes() {
+ return array(
+ 'Tag name' => array( "<img\x00id=5>", "<img\u{FFFD}id=5></img\u{FFFD}id=5>" ),
+ 'Attribute name' => array( "<img/\x00id=5>", "<img \u{FFFD}id=\"5\">" ),
+ 'Attribute value' => array( "<img id='5\x00'>", "<img id=\"5\u{FFFD}\">" ),
+ 'Body text' => array( "one\x00two", 'onetwo' ),
+ 'Foreign content text' => array( "<svg>one\x00two</svg>", "<svg>one\u{FFFD}two</svg>" ),
+ 'SCRIPT content' => array( "<script>alert(\x00)</script>", "<script>alert(\u{FFFD})</script>" ),
+ 'STYLE content' => array( "<style>\x00 {}</style>", "<style>\u{FFFD} {}</style>" ),
+ 'Comment text' => array( "<!-- \x00 -->", "<!-- \u{FFFD} -->" ),
+ );
+ }
+}
</ins><span class="cx" style="display: block; padding: 0 10px">Property changes on: trunk/tests/phpunit/tests/html-api/wpHtmlProcessor-serialize.php
</span><span class="cx" style="display: block; padding: 0 10px">___________________________________________________________________
</span></span></pre></div>
<a id="svneolstyle"></a>
<div class="addfile"><h4 style="background-color: #eee; color: inherit; margin: 1em 0; padding: 1.3em; font-size: 115%">Added: svn:eol-style</h4></div>
<ins style="background-color: #dfd; text-decoration:none; display:block; padding: 0 10px">+native
</ins><span class="cx" style="display: block; padding: 0 10px">\ No newline at end of property
</span></div>
</body>
</html>