[wp-trac] Re: [WordPress Trac] #8999: Completely New LiveJournal Importer

[WordPress Trac]

#8999: Completely New LiveJournal Importer
-------------------------------------+--------------------------------------
 Reporter:  beaulebens               |        Owner:          
     Type:  task (blessed)           |       Status:  reopened
 Priority:  normal                   |    Milestone:  2.8     
Component:  Import                   |      Version:  2.7     
 Severity:  normal                   |   Resolution:          
 Keywords:  needs-testing has-patch  |  
-------------------------------------+--------------------------------------
Comment (by westi):

 Replying to [comment:38 beaulebens]:
 > OK - I've just run a full import (3,750 posts, 200,000 comments) and it
 took just over 6 HOURS to import (previous, file-based version too just
 under 3 hours for the same journal). The comments really kill this process
 --  re-threading takes 3 hours because querying the comments is super slow
 since I'm using "spare" fields that aren't indexed.
 >

 Cool. Although 6 hours is a little long ;-)

 > Importing a smaller journal goes quickly and without problem.
 >
 > 2 options here to speed it up for big journals:
 >
 >  1. Before re-threading, ALTER TABLE ADD INDEX on the 3 fields that are
 used. That seems to reduce things down to literally a couple of minutes,
 then I can DROP INDEX when I'm done, or
 >  2. Use a temporary table in MySQL that's optimized for what I'm trying
 to do (but then I'd have all sorts of custom code that operated outside
 the normal comments API).
 >
 > I'd lean towards the first option, but I don't know how that jives with
 the general approach of core code?
 >

 What are the extra fields you are using.

 Is there any reason why they couldn't just be indexed anyway?

 I think we should go with the speediest solution - people won't want to
 wait 6 hours for there import to complete!

-- 
Ticket URL: <http://trac.wordpress.org/ticket/8999#comment:39>
WordPress Trac <http://trac.wordpress.org/>
WordPress blogging software