Percona Server is an open source project to produce a distribution of the MySQL server with improved performance, scalability and diagnostics.
This process is very much modeled on what is being used by Drizzle. The Drizzle project went through several iterations and refinements before settling on this process. It has been found to both keep trunk in a constant state of stability (allowing for a release at any time) and minimizing wasted time by developers due to broken code from somebody else interfering with their day.
You should also be familiar with our Jenkins setup.
Changes to our software projects could be because of a new feature (blueprint) or fixing a bug (bug). Projects such as refactoring could be classed as a blueprint or a bug depending on the scope of the work.
Blueprints and bugs are targeted to specific milestones (releases). A milestone is part of a series - e.g. 1.6 is a series in Percona XtraBackup and 1.6.1, 1.6.2 and 1.6.3 are milestones in the 1.6 series.
Code is proposed for merging in the form of merge requests on launchpad.
Some software (such as Percona Xtrabackup) we maintain both a development branch and a stable branch. For example: Xtrabackup 1.6 is the current stable series, and changes that should make it into bugfix releases of 1.6 should be proposed for the 1.6 tree. However, most new features or more invasive (or smaller) bug fixes should be targeted to the next release, currently 1.7. If submitting something to 1.6, you should also propose a branch that has these changes merged to the development release (1.7). This way somebody else doesn’t have to attempt to merge your code and we get to run any extra tests that may be in the tree (and check compatibility with all platforms).
For Percona Server, we have two current bzr branches on which development occurs: 5.1 and 5.5. As Percona Server is not a traditional project, instead being a set of patches against an existing product, these two branches are not related. That is, we do not merge from one to the other. To have your changes in both, you must propose two branches: one for 5.1 version of patch and one for 5.5.
In this case we’re going to use percona-xtrabackup as an example. workflow is similar for Percona Server, but patch will need to be modified both in 5.1 and 5.5 branches.
If the change also applies to a stable release (e.g. 1.6) then changes should be made on a branch of 1.6 and merged to a branch of trunk. In this case there should be two branches run through param build and two merge proposals (one for 1.6 and one with the changes merged to trunk). This prevents somebody else having to guess how to merge your changes.
Before code hits trunk, it goes through a “staging” branch, where some extra tests may be run (e.g. valgrind) along with testing that all branches behave well together (build and test) before pushing to trunk.
To ensure quality, DO NOT push directly to trunk! everything must go through adequate testing first. This ensures that at any point trunk is in a releasable state.
Please note that ALL changes must go through staging first This is to ensure that several approved merge requests do not interact badly with each other.
In the event of a merge request being marked as ‘Work In Progress’ due to build/test failures when merging, the developer should fix up the branch, run through param build and then ‘Resubmit’ the merge proposal.
There is a link on launchpad to resubmit the merge proposal, this means it appears in the list of merge requests to review again rather than off in the “work in progress” section.
The same process for Percona Server, but we have different branches (and merge requests) for 5.1 and 5.5 series.
We will need some human processes to ensure that we do not merge extra things during the time when base MySQL version is being updated to avoid making life harder for the person doing the update.
This way anybody can easily check out an old release by just using bzr to branch the specific tag.
Our Jenkins instance uses a mixture of VMs on physical hosts that Percona runs and Virtual Machines in Amazon EC2 that are launched on demand.
We have some jobs that are activated based on source control changes (new commits in a bzr repository). We have some that are “param build” - that is, a user specifies parameters for the build (e.g. the bzr tree). A param-build allows developers to ensure their branch compiles and passes tests on all supported platforms before submitting a merge request. This helps us maintain the quality of the main bzr branches and not block other developers work.
Jenkins is a Master/Slave system and the jenkins master schedules the builds across available machines (and may launch new VMs in EC2 to meet demand).
Most of our jobs are what’s known as “matrix builds”. That is, a job that will be run with several different configurations of the project (e.g. release, debug) across several platforms (e.g. on a host matching the label of “centos5-32” and a host matching label of “ubuntu-natty-32bit”). Matrix builds show a table of lights to indicate their status. Clicking “build now” on one of these queues up builds for all of the combinations.
We have some integration of our regression test suites (currently xtrabackup) with Jenkins ability to parse JUnitXML, presenting a nice user interface to any test failures.
Because building some projects is non-trivial, in order to not duplicate the list of compile instructions for each job, we use template builds. You’ll see builds such as percona-xtrabackup-template which is a disabled job, but all current xtrabackup jobs point to it for the commands to build and run the test suite.
We currently build both xtrabackup 1.6 and xtrabackup trunk (will become 1.7).
There are param-builds for 1.6 and trunk too. These should be run for each merge request (and before any collection of merged branches is pushed to trunk)
We have separate jobs for Percona Server 5.1 and Percona Server 5.5 due to the different build systems that MySQL 5.1 and 5.5 use.
The mysql-test-run.pl test suite is integrated with Jenkins through subunit and subunit2junitxml allowing us to easily see which tests passed/failed on any particular test run.
We have trunk and param jobs. We also have a valgrind job that will run after a successful trunk build.
I’ve set up a few jobs in Jenkins that should help us predict the future for Percona Server. Namely, if upstream MySQL may cause us any problems.
I wanted to see if some test failures were possibly upstream, so I set up two jobs:
both of which ask for a URL to a MySQL source tarball and then do a full build and test across the platforms we have in jenkins.
But my next thought was that we could try and do this before the source tarballs come out - hopefully then being able to have MySQL release source tarballs that do in fact pass build and test everywhere where we’re wanting to support Percona Server.
are scheduled to just try once per week (we can change the frequency if we want to) to build and test from the MySQL bzr trees.
I also have a valgrind build (same configuration as for Percona Server) to help us see if there’s any new valgrind warnings (or missed suppressions).
I’m hoping that these jobs will help us catch any future problems before they become our problem. (e.g. we can easily see that the sporadic test failures we see in Percona Server are actually in upstream MySQL).