[wp-trac] [WordPress Trac] #36280: HTTP API (Streams backend) secured proxy tunnels and chunked and zlib on the fly streaming

[WordPress Trac]

#36280: HTTP API (Streams backend) secured proxy tunnels and chunked and zlib on
the fly streaming
--------------------------+-----------------------------
 Reporter:  bobbywalters  |      Owner:
     Type:  defect (bug)  |     Status:  new
 Priority:  normal        |  Milestone:  Awaiting Review
Component:  HTTP API      |    Version:  trunk
 Severity:  normal        |   Keywords:
  Focuses:                |
--------------------------+-----------------------------
 The purpose of this ticket is to address concerns within the
 WP_Http_Streams class (wp-includes/class-wp-http-streams.php). And supply
 a patch to correct the defects, improve performance, and extend the
 available features as noted below.

 == Proxy Related ==

 * HTTPS requests through a proxy now create a properly secured tunnel with
 certificate validation possible. No longer have to use the Curl backend to
 properly tunnel HTTPS requests through the WP Remote API. No more `501 Not
 Implemented` errors.
 * Created filter `'proxy_tunnel'` to alter which connections may be
 tunneled through a proxy using HTTP CONNECT.

 == SSL/TLS Related ==

 * Added filter `'stream_crypto_method'` to decide which cryptography
 (crypto) method is used for handshake. Default is
 `STREAM_CRYPTO_METHOD_TLS_CLIENT`.
 * Added `'peer_name'` `'ssl'` context parameter to enable peer name
 validation even over tunneled proxy connections.
 * Added `'verify_peer_name'` which is set based on `$ssl_verify` in the
 event verification should be disabled.

 == Transfer-Encoding: chunked ==

 * Using the PHP builtin stream filter `'dechunk'` to process response
 bodies transferred in a chunked fashion.
 * Much faster streaming capability avoids having to read in the entire
 response and then perform logic to distinguish chunks (ie
 `WP_Http::chunkTransferDecode()` is not used).
 * The stream filter also corrects some of the handling found in
 `WP_Http::chunkTransferDecode()` that prevented https://www.google.com
 from correctly being read. As a side note: I'm not sure what was throwing
 chunkTransferDecode off when reading https://www.google.com but it may
 have been the multibyte gzip data along with the 0 padded chunk sizes at
 the beginning. I didn't pursue trying to alter chunkTransferDecode since
 the `'dechunk'` stream filter preforms the reads correctly.

 == Content-Encoding: (deflate | gzip) ==

 * Any content encoding handled by zlib is now done with the stream filter
 `'zlib.inflate'` to handle on the fly decompression.
 * This provides performance and resource utilization improvements
 especially when the `'limit_response_size'` option is set.

 == General ==

 * All options (`'stream'`, `'decompress'`, `'limit_response_size'`, etc)
 are still honored.
 * Reading responses with none, one, or both of `'Transfer-Encoding'` and
 `'Content-Encoding'` work on the fly and streaming.
 * Both HTTP and HTTPS connections with and without a proxy involved work
 now.
 * Redirects continue to work with the improved proxy tunneling even while
 switching between HTTP to HTTPS. As an example http://www.wordpress.org
 performs a few redirects to drop the "www" and get over to https.

 == Testing ==

 * Using a local instance of WordPress with a combination of WP-CLI eval-
 file scenarios along with clearing out the transients and using the admin
 console to search, view details, and download plugins.
 * The previous bullet with a local Squid instance as setup out of the box
 Debian Jessie style. A simple Dockerfile was to get Squid running which
 could be shared. `wp-config.php` was modified to define `WP_PROXY_HOST`
 and `WP_PROXY_PORT` to use the local Squid instance.
 * As of right now I have not run the phpunit tests because I've never
 attempted to run them in WordPress. However, looking over
 https://develop.svn.wordpress.org/trunk/tests/phpunit/tests/http/base.php
 I don't immediately see anything that would be a problem.
 * I could see some additional test cases being useful around HTTP SSL/TLS,
 chunked, and gzip handling. Then all of that with a proxy in play but I'm
 not sure how feasible proxy testing is.

 I may have skipped over some of the finer points but that should be the
 majority of the changes and rationale behind them. Thanks in advance for
 looking over this information and the related code changes.

--
Ticket URL: <https://core.trac.wordpress.org/ticket/36280>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform