[wp-trac] [WordPress Trac] #22981: Tweets import plugin tracking ticket

[WordPress Trac]

#22981: Tweets import plugin tracking ticket
----------------------------+----------------------------
 Reporter:  nacin           |       Owner:  PeteMall
     Type:  task (blessed)  |      Status:  accepted
 Priority:  high            |   Milestone:  WordPress.org
Component:  Import          |     Version:
 Severity:  normal          |  Resolution:
 Keywords:                  |     Focuses:
----------------------------+----------------------------

Comment (by Otto42):

 wp-twitter-importer.3.php is my modification of the last version uploaded
 by aaroncampbell.

 - Removed the dependency on the WP_Importer_Cron (for now, the plugin
 wasn't really using it anyway). May add this back later, along with
 checking for duplicates on import and smarter parsing.

 - Some start and end functions to speed up the process by turning off the
 cache invalidation and term/comment counting during the import

 - Speed boost by turning off autocommit and manually committing once every
 1000 tweets. This isn't the safest thing in the world, but it's a start.
 The speed boost is substantial. Importing 11,000 tweets went down from ~6
 minutes to ~30 seconds.

 - Added a filter for the post_type used, in case somebody doesn't want
 them all to be "post".

 Few known problems:

 - It's using the WP_Filesystem slightly incorrectly. I'm not sure how to
 fix this at present because I've never tried to use it with a file upload.
 Think the order of operations needs to be changed around here. It will
 work with "direct" mode only at present.

 - It's still a bit slow and will run into PHP timeouts. There are some
 enhancements that can be made for speed, and it can be put into a
 continuous cron job.

 - Turning off autocommit with 'set autocommit = 0' strikes me as
 potentially bad. Probably need to find a better way.

 - Like the other importers, we should probably make a token effort of some
 kind to download the media "attachments" in the tweets and rejigger the
 URLs to local ones. Might want to limit this to known domains, like
 pic.twitter and the like.

--
Ticket URL: <https://core.trac.wordpress.org/ticket/22981#comment:38>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform