[wp-trac] [WordPress Trac] #55206: wp core api memory leaks

[WordPress Trac]

#55206: wp core api memory leaks
--------------------------+-----------------------------
 Reporter:  sllimrovert   |      Owner:  (none)
     Type:  defect (bug)  |     Status:  assigned
 Priority:  normal        |  Milestone:  Awaiting Review
Component:  Database      |    Version:  trunk
 Severity:  normal        |   Keywords:
  Focuses:                |
--------------------------+-----------------------------
 I've experienced the following two memory leaks in WP core.  One involves
 $wpdb when `SAVEQUERIES` is defined truthy, and the other involves
 `$wp_object_cache` growing as a consequence of calling core api functions
 that themselves save to the object cache.  Both have happened for me in
 cases where I'm doing large batch processing involving thousands or tens
 of thousands of posts.  I've had memory usage exceed 512MB and cause
 crashes.

 I'm including unit tests here showing each memory leak and also the fix
 that I've used to prevent the memory leak and keep my batch jobs running.

 {{{#!php
 <?php

 /**
  * Class WP_Memory_Leak_Tests
  *
  * This class tests two cases that cause memory leaks in WordPress that
 could
  * lead to crashes, particularly in CLI jobs that work on larger batches.
 For
  * each of the cases ( one for the wpdb class and one for the global
  * $wp_object_cache ), we perform some seemingly innocuous task many times
 -
  * enough times to require that PHP allocate more memory because of a
 specific
  * action.
  *
  * Neither of the tests here show a particularly large memory increase,
 but I've
  * personally had both occur for me on large jobs hitting WP API
 functions.  The
  * one with $wpdb->queries particularly has a tendency to blow up.
  */
 class WP_Memory_Leak_Tests extends WP_UnitTestCase {

         /**
          * This tests a condition which exposes a memory leak in the WPDB
 class.
          * If 'SAVEQUERIES' is defined as truthy, then the $wpdb->queries
 property
          * can grow indefinitely.
          */
         public function test_WPDB_Memory_Leak() {

                 // Once a constant is defined, it can't be undefined, it's
 often defined in dev or staging environments.
                 define( 'SAVEQUERIES', true );

                 // I'll just start my cron job to read the import file
 I've got.  It's
                 // got a decent number of records.
                 $number_of_records = 1000;

                 global $wpdb;
                 $memory = memory_get_usage( true );
                 $peak   = memory_get_peak_usage( true );
                 foreach ( [ 'first', 'second' ] as $pass ) {
                         // first pass through, we'll apply a fix for this
 memory leak.
                         // second pass through, we'll bypass the fix and
 the tests will fail.
                         for ( $i = 1; $i <= $number_of_records; $i ++ ) {
                                 if ( 'first' === $pass ) {
                                         $wpdb->queries = [];
                                 }

                                 // for this test, we'll do direct calls to
 $wpdb
                                 $wpdb->query( $wpdb->prepare( "SELECT *
 FROM $wpdb->posts WHERE ID = %d", $i ) );
                         }
                         $this->assertEquals( $memory, memory_get_usage(
 true ), "$pass pass" );
                         $this->assertEquals( $peak, memory_get_peak_usage(
 true ), "$pass pass" );
                 }

         }

         /**
          * This tests a condition which exposes a memory leak in wp cache
 API.  If
          * a large batch job attempts to do a lot of something that ends
 up caching
          * things ( like, for example, get_post or wp_insert_post ), then
 unless
          * the cache is flushed regularly, the memory usage grows
 indefinitely.
          */
         public function test_WP_Cache_Memory_Leak() {

                 // I'll just start my cron job to read the import file
 I've got.  It's
                 // got a decent number of records.
                 $number_of_records = 1000;

                 global $wpdb;
                 $memory = memory_get_usage( true );
                 $peak   = memory_get_peak_usage( true );
                 foreach ( [ 'first', 'second' ] as $pass ) {
                         // first pass through, we'll apply a fix for this
 memory leak.
                         // second pass through, we'll bypass the fix and
 the tests will fail.
                         for ( $i = 1; $i <= $number_of_records; $i ++ ) {
                                 if ( 'first' === $pass ) {
                                         wp_cache_flush();
                                 }

                                 // Because our last test defined
 'SAVEQUERIES', we need to
                                 // always apply this fix, otherwise that
 memory leak manifests.
                                 // With us doing a core API function
 `wp_insert_post`, the number
                                 // of queries is quite large and memory
 __really__ grows.
                                 $wpdb->queries = [];

                                 // let's say we're inserting posts, maybe
 from an excel file.
                                 // this caches some things, so
 $wp_object_cache grows.
                                 wp_insert_post([
                                         'post_type' => 'post',
                                         'post_title' => "post $i",
                                         'post_content' => "pass $pass"
                                 ]);
                         }
                         $this->assertEquals( $memory, memory_get_usage(
 true ), "$pass pass" );
                         $this->assertEquals( $peak, memory_get_peak_usage(
 true ), "$pass pass" );
                 }

         }

 }

 }}}

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/55206>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform