[wp-trac] [WordPress Trac] #43711: Let's create a standard development setup for WordPress core.

Wed May 30 22:39:09 UTC 2018

#43711: Let's create a standard development setup for WordPress core.
------------------------------+------------------------------
 Reporter:  omarreiss         |       Owner:  (none)
     Type:  enhancement       |      Status:  new
 Priority:  normal            |   Milestone:  Awaiting Review
Component:  Build/Test Tools  |     Version:  trunk
 Severity:  normal            |  Resolution:
 Keywords:                    |     Focuses:
------------------------------+------------------------------

Comment (by MikeSchinkel):

 Hi everyone,

 I'd like to throw a new contender into the ring; ''(currently named)''
 [https://github.com/wplib/wplib-box WPLib Box] [https://github.com/wplib
 /wplib-box/issues?q=is%3Aopen+is%3Aissue+milestone%3A0.17.0 0.17.0].

 We originally decided to build it because of frustrations with VVV. The
 fact VVV was provisioned locally using Puppet, which often failed and
 required sysadmin level skills to debug meant that it was not a solution
 that developers with few sysadmin skills could trust.  Instead we wanted a
 box that “just worked,” so we built one that was pre-provisioned.  And as
 of `0.10.2` it was really solid. That took us about a year, though not
 full time.
  Then we set our sites higher and decided to build a box that not only
 just worked but what capable of running any stack needed while still
 ensuring the most common use-cases were dead simple. And that lead us to
 planning the use of a set of orchestrated Docker containers.

 But what we found over the past year is that Docker for Mac has
 significant limitations, such as
 [https://stackoverflow.com/questions/46286741/is-there-a-workaround-to-
 use-the-hosts-network-in-docker-for-mac only being able to use the host’s
 network] on Docker for Mac. This issue alone makes a pure implementation
 of orchestrated Docker containers a non-starter, assuming you want to make
 it “just work” for the developer installing it.  That lead us back to
 using a Vagrant and VirtualBox with Docker containers. Thus the current
 version of WPLib Box is `0.16.2` (with `0.16.3` coming later this week,
 hopefully) and the current version is based on Ubunutu 14.04 LTS + Vagrant
 + VirtualBox + individual Docker containers inside.    But in reality
 Vagrant is very problematic, constantly breaking working boxes on updates
 and just persnickety when it comes to installation.  Further, Ubuntu is
 really memory intensive.
 So our plans are to release `0.17.0` in the near future, and it will be
 based on either Tiny Core Linux or Alpine, VirtualBox and
 [https://github.com/wplib?utf8=%E2%9C%93&q=docker many Docker containers].
 It will also come with both a Mac and a Windows installer where the goal
 is for it to be as easy to get working as any application you might
 download and install on your Mac or Windows computer.

 Looking back, we worked on `0.16.0` for over a year and it was a huge
 release which changed the box from a LEMP VM to a VM designed to
 orchestrate Docker containers and it positions us for a relatively quick
 release for `0.17.0`. In order to release sooner or later we will probably
 not implement all of our [https://github.com/wplib/wplib-
 box/issues?q=is%3Aopen+is%3Aissue+milestone%3A0.17.0 current backlog of
 issues tagged for 0.17.0] and instead implement a lot of them in `0.17.x`.
 Longer term, we even plan to get rid of VirtualBox, but that will require
 more than a short-term engineering effort.
  Now on to the requirements mentioned by everyone in this thread:
 Replying to [ticket:43711 omarreiss]:
 > '''''Download and boot'''''
 > - the easiest possible development workflow: just download and boot.
 > - reliable: it should just work.
  From day 1 that has been a mantra for WPLib Box, that is “just works.”
  > == Integrating watch and build behavior ==
 > As we plan to [https://core.trac.wordpress.org/ticket/43055 introduce a
 build step to the development workflow], it becomes more important that
 building is not an afterthought but something that is integrated into the
 development workflow.

 A key aspect of WPLIb Box is its extensible architecture. With that
 architecture we will be able to easily implement and incorporate the build
 solution from #43055.  WPLib’s extensibility features as based on two
 concepts: “Stacks” and “Installables.”  Stacks allow implementing any
 conceivable set of services you need with the default one being
 “WordPress.” But there could be a “WordPress Develop” stack, a “Tide”
 stack, or hell, even a “Drupal” or Node Express” stack.  Stacks can extend
 other stacks and WPLib Box will be able to run multiple stacks.    Stacks
 are comprised of one of more “Installables”, and an Installable can be
 anything that either a “Project” needs or that the Box needs globally.
 Further, Installables can a Docker container, a WordPress plugin, a Theme,
 a SQL data file, or even a collection of those things. or a collections of
 other Installables. An Installable can even be a Stack.

 Stacks and/or Installables required by a Project get written to a
 `project.json` file that can be version controlled and shared so that when
 a team member, or another WordPress core contributor brings up a project
 they will automatically see the exact environment that was configured for
 them by the developer who committed the `project.json` file.  Note WPLib
 Box `0.17.0` will support multiple “Projects,” each being identified by a
 local domain, e.g. `example.local` (we don’t call is a “multi-site” box
 for hopefully obvious reasons.)

 > - fast: reduce build and boot times to a minimum.
  You should download and try the current version to see for yourself.
 We’ve really made it to be very performant, and `0.17.0` should only
 improve that.

 > @pento noted that eventually he'd like this thing to be configurable and
 installable through a GUI in order to reduce the barrier of entry for new
 contributors to the absolute minimum.  Part of our architecture will be to
 have an Electron-based admin console.  It will run based on the API we are
 building into the box.  Assuming something very similar to Local by
 Flywheel.

 Replying to [comment:1 youknowriad]:
 >  - Ideally, the integration and e2e tests (even unit tests since PHP
 unit tests require a full setup) should use the same environment  We are
 [https://github.com/wplib/wplib-box/issues/25#issuecomment-368401449
 adding PHPUnit to the box] by implementing a [https://github.com/wplib
 /wplib-box/issues/363 general-purpose PHP script container approach] that
 will make adding PHP-script based solutions as easy as specifying the
 details in an `installable.json` file.

 From that we should be able to create an Installable adds the integration
 and e2e test and then include them in the “WordPress Develop” stack.

 >  - A dev environment even if it's hidden behind magical scripts should
 stay simple. When bugs happen developers of any level should be able to
 understand, debug and tweak how the environment works.

 Saying this a different way, our goal with WPLib Box is that the common
 case is dead simple, and the edge cases are possible.  We are focused on
 hiding Docker containers completely from everyone except those that want
 to interact with them and/or build their own Docker containers for WPLib
 Box.

 We will be segmenting our UX to multiple levels:
 1.  End-user developer: All they will need to ever see and are the
 installer and Electron-based admin console.
 2.  Power-user developer: In addition to the end-user developer they will
 also be interested in the `box` command line tool that will work both in
 the box and also on the host. They will also be able to build simple
 installables such as a starting point for a website; e.g. WordPress core,
 a collection of plugins and a theme.
 3. Front-end extension developer: Beyond end- and power-user developer
 they will be able to extend the Electron-based admin using Vue.js and the
 Box API.
 4. Box extension developer: Beyond end- and power-user developer these
 people will be able to extend the box itself such as Installable
 containers and even Stacks, as well as adding commands to the `box` CLI.

 From the above segmentation we should be able to directly support hiding
 all complexity and make the end-user developer’s use very simple.
 Replying to [comment:4 Mte90]:
 > One of the problems that has VVV right now is that require a lot of time
 on first start because of installation (and download of a lot of things)
 but the plan for version 3 (https://github.com/Varying-Vagrant-
 Vagrants/VVV/issues/1469) with a box that contain everything and also the
 parallel for the provision will improve a lot this point.  The current
 WPLib Box (`0.16.2`) contains everything and boots in about 30 seconds.
 We expect the next major version  (`0.17.0`) will boot in around 5
 seconds.
 > The other problems that often at contributor day we have is Windows.
 Install virtualbox, vagrant, git on Windows is a big step because people
 do't know anything of them.
  We attend to hide the installation of VirtualBox via our installers.  We
 can also look into installing Git, or we can provide them Git access via
 `box git …` which could be wrapped in a batch or Powershell file as just
 `git`.

 > Often the problems on VVV is because there are different versions of
 powershell on the system (don't ask me why) and Vagrant or Virtualbox
 works with issue based on the version of ps.

 For the most part this just won’t be an issue in `0.17.0` given our
 architecture.  And our box command will be implemented in GoLang so that
 should resolve the cross platform issue.
  > With docker we will have more problems on Windows because we need a
 virtual machine so (virtualbox and git will be required in any case) so
 the only advantage that we don't have anymore the easy vagrant to use but
 docker (that is not so entry level).

 Again, we are addressing all of this with an installer, a VM to contain
 and orchestrate the Docker containers, and a simple admin console to make
 things easy.

 > Sure, there is the VVV USB Generator that prepare all the executables
 for download (maybe also the latest image) but we need to think that at
 contributor day not everyone has an OS dev friendly for *nix stuff.  We
 will support Windows day one.
 > From the other side VVV has the cool part to be used also for not
 contribute and very easy with vagrant up/halt and db backup so usually
 people are interested in that then contribute.

 I would like to have is so that when someone adopts WPLib Box because of
 how easy it is that we make it just as easy for them to contribute to
 WordPress by giving them a menu option to just get started.

 > > Integrating watch and build behavior: As we plan to
 [https://core.trac.wordpress.org/ticket/43055 introduce a build step to
 the development workflow], it becomes more important that building is not
 an afterthought but something that is integrated into the development
 workflow.

 We plan to add watching/building as a default installable for our
 WordPress stack.

 We have another mantra that is
 > ''"If we can simplify things for the end user and make it run in the
 box, we should make it run in the box."''

 So our vision is to minimize the number of things developers have to
 (figure out how to) install and get working on their host machine, and
 that we instead deliver a default solution that "just works," but that is
 also configurable and extensible for their needs.

 > having access to a https://en.wikipedia.org/wiki/Command-line_interface
 (CLI) to run `grunt watch` and `grunt build` tasks will be an important
 part of the WordPress development workflow.

 We will certainly have installables for Grunt and Gulp though we plan to
 have WebPack + NPM scripts be the default build solution for WordPress
 (and WordPress Develop.)

 > * WordPress' CLI commands and scripts will need to be compatible with
 Mac, Windows, and *nix operating systems

 The `box` CLI will be compatible on Mac, Windows and *nix, and the build
 systems will be able to run in the box so the scripts we use will only
 need to run within our purpose-built containers ''(which is the whole
 benefit of containers, right?)''

 > * [https://getcomposer.org/ Composer] is required as this was recently
 added in #43558 for [https://github.com/squizlabs/PHP_CodeSniffer PHPCS]
 support

 We already include support for [https://github.com/wplib/composer-docker
 several versions of Composer] in WPLib Box `0.16.2`.

 > * Support multiple versions of MySQL and MariaDB for testing #30462
 #42811

 We already include support for [https://github.com/wplib/mysql-docker
 multiple versions of MySQL] and [https://github.com/wplib/mariadb-docker
 multiple versions of MariaDB] in WPLib Box `0.16.2`.

 > * Support multiple versions of PHPUnit for testing (Legacy PHP versions
 required legacy PHPUnit versions) #43218

 As noted, we have plans to [https://github.com/wplib/wplib-
 box/issues/25#issuecomment-368401449 support all necessary versions of
 PHPUnit] in `0.17.0`.

 > * Support multiple versions of PHP for testing,

 [https://github.com/wplib/wplib-
 box/issues?q=is%3Aissue+is%3Aopen+PHP+5.+label%3AFeature Check], for
 `0.17.0`.

 > at the moment Gutenberg is using
 [https://github.com/WordPress/gutenberg/commit/3c3427472fadd4696347956b3b3e409bd21d606d
 these] scripts to manually install PHP 5.2 & 5.3 versions for Travis CI
 though they are quite fragile.

 I don't fully understand what you are asking for, but I expect we can
 easily address it given our architecture.

 Replying to [comment:7 TJNowell]:
 > Any solution would require a provisioning script to set up the database
 user and the relevant files, as well as perform checks that relevant tools
 are installed. This would seem to be a more portable approach that needs
 to be taken anyway regardless of what devops gets done.
  WPLib Box currently provisions the database on first load, and a package
 of Installables should support all of this.

 Replying to [comment:8 pento]:
 > One of the most valuable things that Docker has done is to popularise
 the "container" concept: small, disposable, task-oriented VMs that are
 designed to spin up quickly, perform a single task well, and can be
 cheaply reset. This encourages tasks to be performed in the environment
 they most make sense: if something needs disk performance (eg,
 `watch`-type tasks), it can be run on the host. If something requires
 specialised software or configuration (eg, a MySQL database), it can be
 run inside of a container.
 > This directly benefits the contributor experience: isolating the magic
 to little boxes that can't be easily broken (and can be easily reset in
 the event they are) reduces the cost of experimenting, knowing that you
 have a fast and reliable reset mechanism to fall back on. Drawing from my
 personal experience,

 I cannot agree more. That is exactly why we chose to architect WPLib Box
 as a collection of orchestrated containers rather than try and create one
 big behemoth Docker container containing everything.

 > VVV has the opposite experience: provisioning is a slow process, which
 discourages resetting, which makes experimenting a risky proposition.

 Yet again, agreed. That opposite experience is exactly why we first
 started to build WPLib Box.

 > - Lightweight.

 Check. See above.

 > - Disposable.

 WPLib Box is currently designed to be disposable.  In `0.17.0` we plan to
 continue that, if for no other reason than that will make upgrades
 trivial.

 > - Centrally manageable.
 > - Swappable components ''(such as for PHP)''

 Check.  We will be managing containers centrally and will have the open to
 auto-update; currently you can run `box self-update` in version `0.16.2`
 and it does essentially that.  And swapping is just changing the name of
 the PHP container in the `projects.json` file

 Further, given our architecture the WordPress core team could manage their
 own ''"WordPress Develop"'' Stack ''(basically a list of Installable
 containers and other installables)'' and those containers could either
 extend ours ''(to make it quick and simple to get the this Stack
 started)'' or they could replace ours if the core team's needs are more
 urgent than we can respond and/or if the core team needs something that
 makes no sense to include in our standard WPLib Box containers.

 Replying to [comment:10 nbachiyski]:
 > If our end goal is to make it really easy, the same way WordPress made
 website create easy 😜 we should aim higher   Absolutely.  WPLib Box’s
 core mantra is “It just works” and everything we do is filtered with the
 lens of getting us there by version 1.0

 > * https://c9.io/ – a web IDE now run by Amazon AWS (Janitor uses it).
  I really hope we don’t decide to go there. I have tried using C9, and it
 cannot compare to a solid local experience with PHPStorm+XDEBUG.  I don’t
 know about others, but if contributing to WordPress required using c9, I
 would never even consider it.  That said, if c9 were an ''optional''
 approach for newbies, that might make sense.  More on this ''"optional"''
 idea below.

 Replying to [comment:12 pento]:
 > web-based development environments require a continuous, reasonably
 stable internet connection.

 Absolutely. A local solution that requires no internet connection is
 ideal.  Which is how we've configured WPLib Box.

 > - Adding config for popular editors is a good idea. Mandating the editor
 is not.

 Plus+1 for a WPLib Box Installable.

 > Storing GitHub/SVN/whatever credentials in the cloud is... less than
 ideal. 🙃 Requiring re-authentication for every commit is a terrible UX.

 Agreed.  SSH passthru from host is better.  Auto-managing SSH in the box
 for those who do not already have set up is an even more ideal approach
 ''(which we want to implement with a Installable container.)''

 > - Fits existing uses. Primarily, core development, Gutenberg development
 (as a model for future core development), and can be easily setup at
 contributor days.

 Check:  `0.16.2`

 > - Is built with future uses in mind. Either container or full VM
 configurations can be relatively easily translated to online development
 environments, or to testing environments like calypso.live.

 Check: `0.17.0`

 All that said, if you made it to this point I greatly appreciate it.

 What I do notice on this ticket is there are now three (3) solutions vying
 for adoption:  VVV, some abstract Docker solution, and now WPLib Box.

 So my proposal is this: Rather than make a hard and fast decision on one
 solution, why not leverage the competition inherent in the market and
 instead **establish a set of requirements** that any recommended solution
 will need to provide, and then provide a recommended list of solution
 based on which one meets all ''(or most)'' of those objectives?  That way
 you won't be shutting the door on a solution that might end up being
 significantly better simply by agreeing to choose one early one?

 And if the core team later chooses to adopt just one as the ''"official"''
 version then you will have the option of picking the one that evolved to
 be the best and not just the one that was anointed early on?

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/43711#comment:15>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform