[wp-trac] Re: [WordPress Trac] #8999: Completely New LiveJournal Importer

[WordPress Trac]

#8999: Completely New LiveJournal Importer
-------------------------------------+--------------------------------------
 Reporter:  beaulebens               |        Owner:          
     Type:  task (blessed)           |       Status:  reopened
 Priority:  normal                   |    Milestone:  2.8     
Component:  Import                   |      Version:  2.7     
 Severity:  normal                   |   Resolution:          
 Keywords:  needs-testing has-patch  |  
-------------------------------------+--------------------------------------
Comment (by beaulebens):

 OK - I've just run a full import (3,750 posts, 200,000 comments) and it
 took just over 6 HOURS to import (previous, file-based version too just
 under 3 hours for the same journal). The comments really kill this process
 --  re-threading takes 3 hours because querying the comments is super slow
 since I'm using "spare" fields that aren't indexed.

 Importing a smaller journal goes quickly and without problem.

 2 options here to speed it up for big journals:

  1. Before re-threading, ALTER TABLE ADD INDEX on the 3 fields that are
 used. That seems to reduce things down to literally a couple of minutes,
 then I can DROP INDEX when I'm done, or
  2. Use a temporary table in MySQL that's optimized for what I'm trying to
 do (but then I'd have all sorts of custom code that operated outside the
 normal comments API).

 I'd lean towards the first option, but I don't know how that jives with
 the general approach of core code?

 Comments? Ryan?

-- 
Ticket URL: <http://trac.wordpress.org/ticket/8999#comment:38>
WordPress Trac <http://trac.wordpress.org/>
WordPress blogging software