Recently, we’ve seen a few people ask us about building TokuMX from scratch. While it’s best if you just use the binaries you can get from us (they have all the right optimizations, we’ve tested them, and we can interpret coredumps they generate), we recognize there are other reasons you might need to do a custom build.

Since we actually build six distinct products all using the Fractal Tree® indexing library (community and enterprise versions of TokuDB for MySQL, TokuDB for MariaDB, and TokuMX), our build process is pretty complicated, compared to software packages that might, for example, just involve one source repository and link against a few standard libraries. Our TokuMX builds involve four git repositories, three separate build stages, two different build tools, and three different compiler spec requirements, and TokuDB is even worse. We want to make sure that if you have to build from scratch, that you get all the optimizations we want you to have, and that you have the fewest steps in between you and a tarball.

I got pretty excited about Vagrant yesterday as I was building out our test infrastructure a little larger, so today I tried something new.

I created a Vagrant config that has all the dependencies you need to build any of the community products, and simplified build scripts that you can use to build TokuMX or TokuDB. Check it out here: Tokutek/vagrant-tokutek-builder.

To get started, clone the repo and provision a machine:

Now, to build TokuMX 1.0.3, just check out the proper revisions of the source, and run the build script:

This builds a tarball in the mongo directory that you can copy to another machine and use.

To build TokuDB 7.0.4, just use the make.mysql.bash script in the ft-engine repo.

This builds a tarball in mysql or mariadb, depending on your choice. There isn’t anything special about this script in the VM, but the VM is set up with all the right dependencies already installed.

Pull requests are welcome. Happy building, and as always, feedback is welcome!

13 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
F Alves

I get this error when I type “vagrant up”
Vagrant could not detect VirtualBox! Make sure VirtualBox is properly installed.
Vagrant uses the VBoxManage binary that ships with VirtualBox, and requires
this to be available on the PATH. If VirtualBox is installed, please find the
VBoxManage binary and add it to the PATH environmental variable.

I do not and cannot use Virtual box. In most enterprises we use server virtualization technologies, not desktop ones. We use KVM or Vmware, some use Hyper-V.
I was under the impression that with this “vagrant” solution we would be able to compile the community version ourselves in any server using any of the modern Linux versions. Did I misunderstand something?
is there a real way to compile this technology?

0
Leif.Walsh

First, this is just meant to be a build machine. Once you build a tarball you can copy it off the virtualbox machine and install and run it anywhere you like.

I did not test anything but virtualbox, but vagrant works well with VMware, so you should feel free to fork it and replace the box in the config with a VMware version of Ubuntu 12.10 and things should just work.

Finally, if you absolutely cannot use this you can always follow the normal build instructions in our documentation, and contact us on one of our support channels if you get really stuck.

0
mxu

When I run vagrant up, I got these, not sure if it is correct.

# vagrant up
Bringing machine ‘default’ up with ‘virtualbox’ provider…
[default] Box ‘quantal64’ was not found. Fetching box from specified URL for
the provider ‘virtualbox’. Note that if the URL does not have
a box for this provider, you should interrupt Vagrant now and add
the box yourself. Otherwise Vagrant will attempt to download the
full box prior to discovering this error.
Downloading or copying the box…

0
Leif.Walsh

This is fine, this is a normal informational message from vagrant.

0
mxu

I get the following files

-rw-rw-r– 1 vagrant vagrant 287M Oct 11 13:45 mariadb-5.5.30-tokudb-7.0.1-linux-x86_64.tar.gz
-rw-rw-r– 1 vagrant vagrant 59M Oct 11 13:14 mariadb-5.5.30-tokudb-7.0.1.tar.gz

How can I get RPM packages, so I can easy copy and install packages on CentOS machines?

0
Leif.Walsh

We only support tarballs because they’re a lot less complex. You can just drop those on your servers, unpack them to something like /opt, and make symlinks to the binaries in /usr/local/bin. This makes upgrading simple.

If you want to try to build RPMs, it’s been done successfully by using cpack -G RPM in the build directory (build.RelWithDebInfo), but if you need debugging help later the first thing we’ll do is ask you to install the official tarballs from our downloads page and use that.

0
mxu

$ sudo apt-get install alien fakeroot
$ cpack -G RPM
$ fakeroot alien mysql-5.5.30-tokudb-7.0.4-linux-x86_64.rpm
$ sudo dpkg -i mysql_5.5.30-2_amd64.deb
$ sudo /usr/bin/mysqld_safe
131012 03:17:17 mysqld_safe Logging to ‘/usr/local/mysql/data/quantal64.err’.
touch: cannot touch /usr/local/mysql/data/quantal64.err': No such file or directory
chmod: cannot access /usr/local/mysql/data/quantal64.err’: No such file or directory
touch: cannot touch /usr/local/mysql/data/quantal64.err': No such file or directory
chown: invalid user: mysql’
131012 03:17:18 mysqld_safe The file /usr/local/mysql/bin/mysqld
does not exist or is not executable. Please cd to the mysql installation
directory and restart this script from there as follows:
./bin/mysqld_safe&
See http://dev.mysql.com/doc/mysql/en/mysqld-safe.html for more information
/usr/bin/mysqld_safe: 128: /usr/bin/mysqld_safe: cannot create /usr/local/mysql/data/quantal64.err: Directory nonexistent

I can get RPM package and DEB package now, but I think it is still not quite easy to deploy tokudb on production environment, no idea why cannot build normal RPM packages just like

mysql-server…
mysql-client…
mysql-devel…
mysql-shared…

thus, we can share our experience on server installation.

0
Leif.Walsh

Like I said, we don’t support RPM or Debian packaging right now. Why don’t you send email to the tokudb-dev mailing list? That’s a better forum for support, and maybe someone from the community will be able to help you.

0
vishnu rao

successfully built tokuMx on the ubuntu vagrant box.

Important note (mentioned above) which one might miss(i did),

“This builds a tarball in the mongo directory that you can copy to another machine and use.”

i copied the tarball to some other location on the vagrant and started mongoD… the process crashes with abort signal.

Solution:
http://stackoverflow.com/questions/16422542/mongodb-crash-on-start-aborted-with-signal-6-centos-6-2-azure

0
Per Johansson

Hi,

TokuDB looks very interesting!
Any plans/chance of making it a “proper” MySQL storage plugin so it can be installed and used without using your MySQL fork?

Best regards,
Per

0
Leif.Walsh

It is a proper storage engine, the reason we ship our own MySQL and MariaDB binaries is that there are changes to the SQL grammar to support some of our features, for example clustering secondary keys. MariaDB has included TokuDB upstream in 5.5.33 with a different grammar for clustering keys and some of our other changes (e.g. online alter table) completely unavailable. Supposedly they will be adding our other features in MariaDB 10.

0
Montrial Harrell

Hi

We are trying to build the TokuMX with ssl option. In the outlined steps, where would we insert –ssl?

0
Leif.Walsh

The build system has changed to be fully CMake as of v1.4.0. If you’re building 1.3.3 or earlier (please don’t), just add ‘–ssl’ anywhere in the scons command in the build script. If you’re building 1.4.0 or later, add ‘-D USE_SSL=ON’ to the cmake command (see docs/building.md).

0