[wp-trac] [WordPress Trac] #30130: Normalize characters with combining marks to precomposed characters

[WordPress Trac]

#30130: Normalize characters with combining marks to precomposed characters
------------------------------------+-----------------------------
 Reporter:  zodiac1978              |       Owner:  SergeyBiryukov
     Type:  enhancement             |      Status:  reviewing
 Priority:  normal                  |   Milestone:  5.4
Component:  Formatting              |     Version:
 Severity:  normal                  |  Resolution:
 Keywords:  dev-feedback has-patch  |     Focuses:
------------------------------------+-----------------------------
Changes (by zodiac1978):

 * keywords:  dev-feedback has-patch close => dev-feedback has-patch


Comment:

 As this is milestoned for 5.4 I would like to propose a "roadmap" to
 normalization to get this ticket moving.

 At the moment we have a "solution" in the block editor (Gutenberg) which
 is normalizing via JS in modern browsers:
 https://github.com/WordPress/gutenberg/pull/6880

 There are some problems with this approach. I have shown @soean at
 WordCamp Stuttgart that this is not working reliable. If you copy a NFD
 text (=text with decomposed characters) ''from within'' the block editor,
 the normalization (via JS on paste) does **not** happen.
 Additionally this is **not** normalizing the title and therefore **not**
 the permalink too.
 See this open issue: https://github.com/WordPress/gutenberg/issues/14178

 I would like to propose exploring the idea of my latest patch a little bit
 more:
 https://core.trac.wordpress.org/attachment/ticket/30130/30130.2.diff

 There still some places left as mentioned
 https://core.trac.wordpress.org/ticket/30130#comment:28
 I would like to find those missing filters and maybe this can be a first
 step to be included in core.

 Unfortunately the needed normalizer function is not there by default. PHP
 5.3+ has to be used (which is not a problem anymore), but the needed PHP
 module `intl` is ''optional'' and not every hoster is using it.

 I've added this issue on the health check plugin for checking for the
 module:
 https://github.com/WordPress/health-check/issues/333

 But to get this finally started I propose to add the approach from my last
 patch for those who have the module installed - Fallbacks, polyfills could
 be added later if necessary.

 I have tested this on macOS 10.14 and this is still the case, therefore I
 remove the `close` tag.

 So please leave your feedback on this patch. Any problems with adding this
 function to so many filter hooks? Any performance problems with it?

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/30130#comment:44>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform