[wp-trac] [WordPress Trac] #21098: Out of memory errors in XML-RPC

[WordPress Trac]

#21098: Out of memory errors in XML-RPC
---------------------------------+------------------------------
 Reporter:  koke                 |       Owner:
     Type:  defect (bug)         |      Status:  new
 Priority:  normal               |   Milestone:  Awaiting Review
Component:  XML-RPC              |     Version:  3.4
 Severity:  major                |  Resolution:
 Keywords:  mobile dev-feedback  |
---------------------------------+------------------------------

Comment (by koke):

 Replying to [comment:7 redsweater]:
 > When you ask for 300 posts, it often adds up to an awfully big single
 XML response that PHP has to load into memory before it sends down the
 pipe. (I assume - PHP experts disagree?)

 It doesn't have to be that way. What I understand WordPress does
 (simplified):

 * Call wp_get_recent_posts, store the array in memory
 * Create a new array of posts (in memory) with the right output attributes
 for XML-RPC
 * Build the XML-RPC response as a string with those posts (again in
 memory)
 * Output the string

 I don't know much about PHP's memory management and when do objects get
 released, but this is using at least 3x the posts size

 A more memory efficient solution would be:

 * Output the XML-RPC prefix
 * Call wp_get_recent_posts, store the array in memory (we probably can't
 get rid of this without redesigning too much)
 * For each post, output the proper xml representation
 * Output the XML-RPC suffix

 There are problems with this too, like not being able to set Content-Size
 in advance, but temporary files could be used to build the xml response

 > So once any obviously foolish memory usage issues in WP are ruled out, I
 think paging techniques are the way to tackle this. It looks like paging
 was added to wp.getRecentPosts, so you can, instead of asking for 100
 recent posts, ask for e.g. 10 at a time with an increasing offset. It
 means 10 requests instead of one, which means more latency and probably an
 ultimately slower transaction, but at least it's extremely likely not to
 exceed the memory limitations of the server.

 My main concern with paging is to figure out how much we should page. I'm
 not going to ask for every single post on a single request, but how do I
 know how many is too many?

 Also, this memory problems only happen to a subset of our users, so I
 don't feel it's right to do more requests (more bandwidth/latency) for the
 majority of users. I know WordPress.com can handle that amount of data.
 I'm sure most of the other hosts can too, but I have no idea how to tell
 them apart.

-- 
Ticket URL: <http://core.trac.wordpress.org/ticket/21098#comment:8>
WordPress Trac <http://core.trac.wordpress.org/>
WordPress blogging software