[wp-trac] [WordPress Trac] #20057: Media upload for multi-webserver setups introduces a nasty race condition that could corrupt uploaded files

[WordPress Trac]

#20057: Media upload for multi-webserver setups introduces a nasty race condition
that could corrupt uploaded files
--------------------------+-----------------------------
 Reporter:  archon810     |      Owner:
     Type:  defect (bug)  |     Status:  new
 Priority:  normal        |  Milestone:  Awaiting Review
Component:  Media         |    Version:  3.3.1
 Severity:  major         |   Keywords:
--------------------------+-----------------------------
 I am in the process of scaling a Wordpress blog with several million
 monthly pageviews, and I have designed a plan to scale it in the highest
 availability manner possible.

 '''The setup'''
 Each web server that hosts HTML/PHP, static files, and uploads will be
 replicated via rsync from time to time and sitting behind a load balancer.
 Rsync will run every 5min to 1 hour, and to mitigate the 404s in the
 uploads, I put together an nginx setup that automatically tries a
 different upstream server in its configuration when it encounters 404s.
 This allows any web server to go down at any time, and the system to run
 as if nothing happened. This also gives me freedom for rsyncing
 periodically rather than immediately and avoids 404s completely.

 '''The problem'''
 Now, the bug (note, I'm using Windows Live Writer which automatically
 names uploads image.png, but I could see this potentially happen without
 WLW too since WP seems to automatically name files on disk in case of
 collisions).

 Let's say we have server A and B. The site name is foo.com. Let's also say
 B is out of date by an hour and a bunch of files got uploaded to A that
 aren't on B. Say, A has image.png and image2.png and B has only image.png.

 Now, the issue is that if the load balancer directs the new post uploader
 (the new post contains a single image) to server B, the file that it will
 create on disk will be named image2.png rather than image3.png. So now B
 will have a file that's different from A's but is named the same way.

 The main problem is that the file name is given based on what's available
 on the disk rather than according to the database. It's easy for this race
 condition (between rsyncs) to destroy the integrity of the files.
 Furthermore, and I've experienced this first hand), if you delete the
 attachment from the Wordpress UI, it could actually delete the wrong file
 from the wrong server as a result.

 Seeing this, I can't continue with my scaling plan until file names are
 assigned by the database rather than the file system or I figure out how
 to mitigate that. I don't want to force all writers to use only server A
 for uploads using its direct IP, as I want HA (high availability).

 Thank you.

-- 
Ticket URL: <http://core.trac.wordpress.org/ticket/20057>
WordPress Trac <http://core.trac.wordpress.org/>
WordPress blogging software