[wp-trac] [WordPress Trac] #16236: Method to stream file downloads in HTTP API, to reduce update memory footprint

[WordPress Trac]

#16236: Method to stream file downloads in HTTP API, to reduce update memory
footprint
----------------------------+-----------------------
 Reporter:  markjaquith     |       Owner:  sivel
     Type:  task (blessed)  |      Status:  accepted
 Priority:  normal          |   Milestone:  3.2
Component:  HTTP            |     Version:
 Severity:  normal          |  Resolution:
 Keywords:  has-patch       |
----------------------------+-----------------------

Comment (by dd32):

 Added a unit test to verify the download sizes, seems right, Headers are
 parsed correctly still too.

 Just tested on 2 systems: (Note, this is not a unit test output, a
 functional benchmark output instead)
 {{{
  exthttp had an error: There are no HTTP transports available which can
 complete the requested request.
  curl consumed 7856 Bytes of memory to download 3,096.07kB, took 6
 seconds, @ 516.01 K/s
  streams consumed 584 Bytes of memory to download 3,096.07kB, took 7
 seconds, @ 442.3 K/s
  fsockopen consumed 608 Bytes of memory to download 3,096.07kB, took 6
 seconds, @ 516.01 K/s

  exthttp consumed 14352 Bytes of memory to download 3,096.07kB, took 9
 seconds, @ 344.01 K/s
  curl consumed 544 Bytes of memory to download 3,096.07kB, took 90
 seconds, @ 34.4 K/s (Note: This slow request couldn't be duplicated,
 probably some local bottleneck)
  streams had an error: There are no HTTP transports available which can
 complete the requested request.
  fsockopen consumed 2976 Bytes of memory to download 3,096.07kB, took 6
 seconds, @ 516.01 K/s
 }}}

 Those numbers seem off, So i've re-run it with the 'real_memory' param of
 memory_get_usage() enabled, and all of them aside from HTTP extension
 returned 0 bytes used:
 {{{
 exthttp consumed 262144 Bytes of memory to download 3,096.07kB, took 8
 seconds, @ 387.01 K/s
 }}}

 Overall, we're going to expect the HTTP Extension to use the most memory
 due to the implementation, however my testing systems don't seem to be
 able to give me actual memory usage figures..

 Will test on another system tomorrow and commit.

-- 
Ticket URL: <http://core.trac.wordpress.org/ticket/16236#comment:30>
WordPress Trac <http://core.trac.wordpress.org/>
WordPress blogging software