[wp-trac] [WordPress Trac] #60375: Site Transfer protocol

Tue Jan 30 10:29:08 UTC 2024

#60375: Site Transfer protocol
-------------------------+------------------------------
 Reporter:  zieladam     |       Owner:  (none)
     Type:  enhancement  |      Status:  new
 Priority:  normal       |   Milestone:  Awaiting Review
Component:  Import       |     Version:
 Severity:  normal       |  Resolution:
 Keywords:               |     Focuses:
-------------------------+------------------------------
Description changed by zieladam:

Old description:

> Migrating WordPress sites involves custom, error-prone logic. There are
> no canonical tools and the guidelines seem lacking.
>
> Let's:
>
> 1. Formalize a list of steps involved in transferring a WordPress site
> between hosts
> 2. Build a canonical plugin that implements those steps and enables easy
> site migrations
> 3. Merge it into WordPress core once its stable
>
> This is relevant for:
>
> * Site migrations
> * Creating and restoring site backups
> * Staging and development environments
> * WordPress Playground imports and exports
> * Moving live sites into Playground and vice versa
>
> ... probably a lot more.
>

> == ZIP bundle as the export format
>
> The [https://github.com/WordPress/data-liberation/discussions/53 Data
> Liberation proposal] makes a great argument for a ".zip" bundle as the
> export format. I would love to leverage it here. A wordpress.zip file
> with all the site files and the data in an .sqlite (or plaintext .sql)
> format sounds like the most natural and convenient way of moving
> WordPress sites around.
>
> Moving around a huge archive may be problematic for larger sites, but the
> ZIP format was built with streaming, compression, chunking, checksums,
> and seeking in mind. It is a good fit for handling imports that are many
> GBs large on a host with 64 MB of ram allocated, and not enough hard
> drive space to hold the import file itself.
>
> To support that last point – I’ve built a [https://github.com/WordPress
> /wordpress-playground/pull/880 streaming zip encoder and decoder in
> JavaScript] for Playground. It can cherry-pick a single file from
> https://downloads.wordpress.org/plugin/gutenberg.17.5.2.zip by
> transferring only a few kilobytes and without downloading the entire
> 10+MB archive. It works with zip files, and it would work with a
> Synchronization API endpoint where the zipped fragments are generated on
> demand.
>
> === Differences with WXR
>
> Unlike WXR imports this is looking to transfer a site in its entirely
> with the Transfer Protocol. The export bundle should include every
> database table, every installed plugin, every asset and file in the wp-
> content directory. It must also include meta information such as the
> domain from which the site is being exported and all custom wp-config.php
> settings. This will be necessary in order to automate the transfer.
>
> == Tasks involved in site transfer
>
> * Set IMPORTING constant so things shut down:
>     * Stop sending emails
>     * Database replication
>     * Cleanup jobs/CRON jobs that might filter on post creation
> * Communicate source and destination site domains/base URLs
> * Rewrite URLs in the database to match new site URL
> * Rewrite URLs in all files including wp-config.php, wp-content,
> sunrise.php, mu-plugins, etc.
> * Communicate wp-config.php settings, including things like WP_SITEURL
> and plugins directory, theme directory, content directory, memory limits,
> and other settings.
> * Let the target site set the database credentials.
> * Copy all content from source to destination site, including users, site
> options, database * tables.
> * Bonus if there's no post-processing via tools like `wp search-replace`.
> The transferred data would rewritten as the transfer happens (e.g. to
> adjust the site URL).
> * Bonus if we can cryptographically secure the conduit through which the
> transfer takes place to prevent someone intercepting a transfer (e.g.
> create a private/public keypair, only allow a single transfer at a time,
> use that certificate to authenticate the transfer.
> * Bonus to track transfer state, communicate progress on it, and allow
> for pausing and resuming a transfer.
> * Bonus if we can start a database transaction log via $wpdb or similar
> system when starting a transfer so that the source site can continue to
> serve requests and ensure that the destination site gets a full
> concurrent update to its data.
>
> == Challenges
>
> * This assumes a blank slate on the target site otherwise we risk
> overwriting ids or mismatching ids.
> * The right design could become a foundation for live synchronization
> between WordPress sites.
>
> == Related efforts
>
> * https://github.com/WordPress/playground-tools/pull/124
> * https://github.com/WordPress/data-liberation/discussions/53
> * https://github.com/WordPress/wordpress-playground/pull/880
>
> Co-authored with @dmsnell
>
> cc @dufresnesteven @berislavgrgicak @tellyworth @dd32 @barry @payton
> @peterwilsoncc @swissspidy @miyarakira @matt @youknowriad @mamaduka
> @aristath

New description:

 Migrating WordPress sites involves custom, error-prone logic. There are no
 canonical tools and the guidelines seem lacking.

 Let's:

 1. Formalize a list of steps involved in transferring a WordPress site
 between hosts
 2. Build a canonical plugin that implements those steps and enables easy
 site migrations
 3. Merge it into WordPress core once its stable

 This is relevant for:

 * Site migrations
 * Creating and restoring site backups
 * Staging and development environments
 * WordPress Playground imports and exports
 * Moving live sites into Playground and vice versa

 ... probably a lot more.

 == ZIP bundle as the export format

 The [https://github.com/WordPress/data-liberation/discussions/53 Data
 Liberation proposal] makes a great argument for a ".zip" bundle as the
 export format. I would love to leverage it here. A wordpress.zip file with
 all the site files and the data in an .sqlite (or plaintext .sql) format
 sounds like the most natural and convenient way of moving WordPress sites
 around.

 Large sites may seem problematic at the first glance, as 300GB zip
 archives are difficult to manage. However, the ZIP format was built with
 streaming, compression, chunking, checksums, and seeking in mind. It is a
 good fit for handling imports that are many GBs large even on a host with
 64 MB of ram allocated and not enough hard drive space to hold the import
 file itself.

 To support that last point – I’ve built a [https://github.com/WordPress
 /wordpress-playground/pull/880 streaming zip encoder and decoder in
 JavaScript] for Playground. It can cherry-pick a single file from
 https://downloads.wordpress.org/plugin/gutenberg.17.5.2.zip by
 transferring only a few kilobytes and without downloading the entire 10+MB
 archive. It works with zip files, and it would work with a Synchronization
 API endpoint where the zipped fragments are generated on demand.

 === Differences with WXR

 Unlike WXR imports this is looking to transfer a site in its entirely with
 the Transfer Protocol. The export bundle should include every database
 table, every installed plugin, every asset and file in the wp-content
 directory. It must also include meta information such as the domain from
 which the site is being exported and all custom wp-config.php settings.
 This will be necessary in order to automate the transfer.

 == Tasks involved in site transfer

 * Set IMPORTING constant so things shut down:
     * Stop sending emails
     * Database replication
     * Cleanup jobs/CRON jobs that might filter on post creation
 * Communicate source and destination site domains/base URLs
 * Rewrite URLs in the database to match new site URL
 * Rewrite URLs in all files including wp-config.php, wp-content,
 sunrise.php, mu-plugins, etc.
 * Communicate wp-config.php settings, including things like WP_SITEURL and
 plugins directory, theme directory, content directory, memory limits, and
 other settings.
 * Let the target site set the database credentials.
 * Copy all content from source to destination site, including users, site
 options, database * tables.
 * Bonus if there's no post-processing via tools like `wp search-replace`.
 The transferred data would rewritten as the transfer happens (e.g. to
 adjust the site URL).
 * Bonus if we can cryptographically secure the conduit through which the
 transfer takes place to prevent someone intercepting a transfer (e.g.
 create a private/public keypair, only allow a single transfer at a time,
 use that certificate to authenticate the transfer.
 * Bonus to track transfer state, communicate progress on it, and allow for
 pausing and resuming a transfer.
 * Bonus if we can start a database transaction log via $wpdb or similar
 system when starting a transfer so that the source site can continue to
 serve requests and ensure that the destination site gets a full concurrent
 update to its data.

 == Challenges

 * This assumes a blank slate on the target site otherwise we risk
 overwriting ids or mismatching ids.
 * The right design could become a foundation for live synchronization
 between WordPress sites.

 == Related efforts

 * https://github.com/WordPress/playground-tools/pull/124
 * https://github.com/WordPress/data-liberation/discussions/53
 * https://github.com/WordPress/wordpress-playground/pull/880

 Co-authored with @dmsnell

 cc @dufresnesteven @berislavgrgicak @tellyworth @dd32 @barry @payton
 @peterwilsoncc @swissspidy @miyarakira @matt @youknowriad @mamaduka
 @aristath

--

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/60375#comment:5>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform