Scheduling Work

Must I Schedule in Advance?

Yes. All work by Percona consultants is via pre-scheduled appointments.

How Long Must I Wait For Help?

Simple tasks often begin in a few hours, complex tasks in a few days. We try to schedule first come, first served. But our availability can vary during our busiest times.

How Do I Schedule An Appointment?

Visit Percona's customer portal and create a new issue -OR- Contact live anytime a 24x7 Shift Captain

Always propose a few alternative times. Include your full contact information (email, phone, Skype) and an outline of what you need done. We’ll get back in touch with you quickly.

Am I Billed for Unused Time?

Yes. We devote our consultant’s time 100% to your needs during scheduled time. Please plan ahead and make complete use of this opportunity.

Am I Billed for Cancellations?

No - if you cancel two full business days (> 48 hours) in advance.
Yes - if you cancel with less advance notice. Contact Percona the same way as when scheduling an appointment.

What Are Tips for Successful Appointments?

Arrange and test online access for Percona in advance. See our Remote Access Checklist.

Keep your technical staff online and accessible. Short, focused interactions minimize delays.

See also our Customer Quickstart.


Billable time

What Tasks Are Billable?

Anything done by Percona's consultants for the customer’s benefit is billable time. This includes:

  • Kickoff calls and general orientation
  • Project management work by Technical Account Managers
  • Establishing logins
  • Monitoring, research, and experimentation
  • Consultations with other experts
  • Composing emails, editing documentation, and drafting reports
  • Status update calls with customers
  • Orienting substitute coverage by another consultant during absences

How Is Billable Time Reported?

Customers receive weekly timesheets detailing all work done for them.

Are Pre-Sales Calls Billable?

No, but pre-sales calls with consultants are limited to 15 minutes.

Is Learning Time Billable?

If a consultant refreshes himself on a topic in which he should be expert, it is not billable. However, time spent learning obscure, infrequently seen, or undocumented topics is billable.

Is On Call Time Billable?

Yes, at a reduced rate.

Are Mistakes by Percona Billable?

No. Our mistakes are few, but when they happen we do not bill customers for wasted time we cause.

Is Billable Time Rounded?

Yes. For any work done outside of pre-scheduled blocks of time, time is billed in 15-minute increments rounded once daily. For example if two units of work were done, one for 5 minutes and one for 35 minutes, 35 + 5 = 40. This would be rounded up to 45 minutes billable that day.

Am I Billed for Unused Appointment Time?

Yes. We devote our consultant’s time 100% to your needs during his scheduled time. Please plan ahead and make complete use of this opportunity.

Am I Billed for Cancelled Appointments?

No - if you cancel two full business days (> 48 hours) in advance. Yes - if you cancel with less advance notice. Contact Percona the same way as when scheduling an appointment.


Roles and definitions

Technical Account Manager (TAM)

This is a consultant focused on fitting your needs into Percona’s consulting team and maintaining a smooth workflow between you and our consultants.

Consulting Projects Coordinator (CPC)

This is an administrator who schedules your appointments with consultants.

Shift Captain

This is a technical expert, one of whom is always on duty to oversee customer issues. You may contact them for any manner of help anytime 24x7 here.

Customer Portal (Zendesk)

Email communications with consultants are managed through our Customer Portal. All customers have accounts and can login to manage your issues.

Login Preparations

Prepare for our consultant’s login using our Remote Access Readiness Checklist.

Timesheets

The work details from the preceding week, emailed on Wednesdays.


Remote Access overview

In normal cases Percona engineers will access your systems by logging in from our bastions servers with their individual SSH key. This is a simple, secure, and fast way for us to reach your servers.

Alternative Remote Access

If you have different access requirements we will gladly comply where possible, however this may cause our work to take longer and hence cost you more. This is particularly the case when we have to use:

  • A proprietary VPN
  • A browser-based or graphical remote access protocol
  • Any tool without a commandline interface
  • A custom virtual machine to satisfy your VPN

Emergency Assistance

If access is restricted or limited to a small number of Percona experts, it is possible that Percona will not be able to service your emergency in a timely manner if remote access is required and those specific Percona experts are unavailable.

Percona Consulting services do not cover emergency service. Emergency service is handled by either Support or Managed Services.

Please note that Percona will not share individually granted accounts among other consultants. Percona Consulting does not operate a 24x7x365 on call rotation. If there is a desire for 24x7x365 availability with remote access, Percona Support or Managed Services can assist. Click here for more information. Time spent on maintaining and documenting individualized accounts is billable.

Known Access Times

Our consultants are instructed to schedule all time on your systems whenever possible . We will endeavor to ensure that you always know in advance who will access your systems, when they will do it and approximately when they should be finished.


Remote access checklist

This checklist will help make the most of your time with Percona by ensuring that our access to your servers is correctly set up before we begin working with you.

Please make sure you've provided us with the items listed below, so that we can log in to your servers. If there's more than one server, please tell us each server’s function (e.g., “master” or “read-only slave”), and all other information that applies to each server. This helps ensure we are working on the intended servers. Be sure to point out critical things we should be careful not to disturb.

Note: Legacy Keys

Engineer Keys

As of Q1 2016, engineer public SSH keys are available from the percona.com domain and can be easily downloaded in similar fashion to percona-toolkit.

For example:

  • https://www.percona.com/get/engineer/KEY/firstname.lastname.pub
  • https://www.percona.com/get/engineer/KEY/firstname.lastname.pub.sha256
  • https://www.percona.com/get/engineer/RPMS/noarch/percona-engineer-firstname.lastname-1-20181211git.noarch.rpm

Where firstname.lastname is the first and last name of the engineer assigned to work with you.

We provide two types of engineer key packages:

  1. Individual packages that create the username for the engineer should it not already exist on the system, as well as the Percona group if it does not already exist, and installs the public SSH keys into /home/username/.ssh/authorized_keys
  2. A shared package that does not perform any group or user setup, but instead installs all firstname.lastname.pub files into /usr/share/percona/engineer

The shared package is available from:

This package installs all engineer public SSH keys into /usr/share/percona/engineer/keys

No user management is performed. You need to perform this yourself. The files are used only for reference purposes.

As an example, to add all engineer keys to shared Percona users after installing the shared package:

mkdir -p /home/percona/.ssh/
chmod 700 /home/percona/.ssh/
cat /usr/share/percona/engineer/keys/*.pub > /home/percona/.ssh/authorized_keys
chmod 600 /home/percona/.ssh/authorized_keys

Please note that old keys are not removed or managed in any way by this method. You must remove them yourself.

RPM packages available via percona.com/get/engineer are not at this time GPG signed. We plan to add this in the future. At the time of this writing, percona.com’s SSL configuration gains an A+ grade on qualys SSL Labs, and we support Forward Secrecy.

Engineers connect from service bastions with public IPs 54.214.47.252 (bastion.percona.com) and 54.214.47.254 (bastion2.percona.com). Please configure your firewall to allow access from these IP addresses to relevant services within your network.

MySQL Percona User

Create a MySQL user named percona, grant all privileges on *.* and give us the password (or leave it in a file in Percona’s engineer home directory, preferably named .my.cnf).

Extended information about the required MySQL account on your database server that can also be provided:

  • What is the username and password of the MySQL account we should use?
  • Have you granted SELECT, SUPER, PROCESS, RELOAD ON *.* to the user?
  • What is the IP address or hostname of the MySQL server?
  • On what IP address does MySQL listen, if bind_address is set in its configuration?
  • What is the port or socket, if they are non-standard?
  • If possible, create a “percona” database and give us all privileges to it (to create temporary tables while optimizing queries). This is completely optional, but it's helpful to us. You can do it by running: CREATE DATABASE percona; GRANT ALL ON percona.* TO 'user'@'host';

Security Policies

When you exchange information with us, or give us access to your servers, please make sure you do it securely. If you need to upload sensitive data to our servers securely for data recovery purposes, please request our GPG key. This lets you encrypt large files that only we can decrypt.

For security and accountability reasons, please do not do any of the following:

  • Don't permit Percona to log into a shared account. Create a dedicated Percona user.
  • Don't install our SSH keys in the root's home directory.
  • Don't permit Percona to use a common MySQL account such as root. Create a dedicated Percona user in MySQL.
  • Don't permit our SSH key to be forwarded to another host. If we must access your systems through an intermediary server (ex: jump-host), please create an SSH key on that server and install the matching public key on the systems we should be able to access.

Other Types of Access

We can work with nearly any type of access. Direct SSH access to the server is most efficient, however, any more elaborate types of access (such as proprietary VPN programs, platform-dependent technologies such as GoToMyPC, etc.,) can be less efficient – sometimes significantly less efficient.

Many of these applications cannot co-exist, so they require elaborate tricks with virtual machines, etc. Multiple hops are also less efficient, as keystroke latency becomes a problem. The use of a graphical desktop on a remote system is also difficult.

In general, Cisco VPN and OpenVPN are the most efficient VPNs for Percona to use. We can also support VPN termination via our bastion host. If you're using VPN, please tell us all the necessary information:

  • What type of VPN system are we connecting? Do we need to download a client?
  • Please send us the configuration file; e.g., the .PCF file (if you're using a Cisco VPN), OpenVPN, or other configuration information
  • Otherwise, please tell us all of the following that apply: the domain server's hostname and IP address, username, password, group name, group password
  • VPN might not work well with DNS, so please tell us the IP addresses for the servers

If you're using any other type of connectivity such as VNC, Remote Desktop, GoToMyPC, or similar, please provide the details.

Troubleshooting

The most common reason we can't log in is that the installed SSH key is for a different user than the one Percona was assigned. The next most common reason is that there are line breaks, incorrect permissions or other problems with the key file. After you've set the key up, please let us know so we can test it.

If you are using VPN, consultants will login from their own machines using your VPN client, so do not restrict access to any specific range of IP addresses. Also, be sure to use the engineer’s own public SSH key, provide password authentication or provide an SSH key of your choosing communicate securely.

Performance Audit Checklist

A performance audit is our most popular service. In order to expedite a performance audit, please ensure that the following tools are installed on each machine:

  • Perl DBI and DBD::mysql
  • MySQL client, vmstat, iostat, mpstat and optionally numactl, innotop, and dstat
  • If there is no outside network access, please install the Percona Toolkit

These tools are very helpful for observing what the machine is doing, so we highly recommend you install them. If you can't install any of the above for some reason, we can still perform a limited audit. It may result in less information being analyzed by the audit.

For our analysis, we need the slow query log. Please note the original values so that you can change them back:

SHOW VARIABLES LIKE '%slow%';
SHOW VARIABLES LIKE '%long%';
SHOW VARIABLES LIKE 'log_output';

Then, rotate your logs (you can move the current one and issue FLUSH LOGS;) and set these values:

SET GLOBAL log_output=FILE;
SET GLOBAL slow_query_log_use_global_control='log_slow_verbosity,long_query_time,log_slow_rate_limit​';
SET GLOBAL log_slow_verbosity='full';
SET GLOBAL slow_query_log=1;
SET GLOBAL long_query_time=0;
SET GLOBAL log_slow_slave_statements=1;
SET GLOBAL log_slow_rate_limit=100;

Some of these variables are only available in Percona Server, so it is fine if they don't get set.

Once you have one hour or 1GB (whichever is first) of data, set the values back to their defaults and save the file off. Please upload the log to our sftp or another secure location, after compression, and provide it to our consultants.

It may be good to do this on both a master and a read slave.

Note: When you set long query time to 0 it may have some io impact as all queries will be collected. In that case, you may want to use 0.1 or 1.

A short architectural overview can go a long way towards giving us a “head start” on the performance audit. Please provide brief documentation that includes:

  • Programming language(s) used
  • Deployment diagrams (if any)
  • Components used (load balancers, caching systems, etc...)
  • Database replication/sharding Information
  • Planned upgrades/changes – alerting us in advance of any new features or changes planned during the time of the audit can save significant time and rework
  • Anything else you feel is worth mentioning

You don't have to go to extremes with any of this. Too much information can cause extra work, too (lists of tables and columns are not necessary). Keep in mind that we have probably seen dozens of systems like yours, so just giving us “hints” about what's unique or noteworthy about your systems can be very effective. For example, “it's a social networking application using Ruby On Rails and we are not sharded yet, but we do a lot of caching” tells us a great deal.
We highly recommend that all of our customers have alerting and monitoring systems in place.

We have a comparison of a few recommended monitoring solutions on our blog here: https://www.percona.com/blog/2017/03/16/monitoring-databases-a-product-comparison/. If you need help installing or configuring them, please let us know. We are glad to assist.

Amazon RDS - Relational Database Service

Ideally you can provide Percona remote SSH access to a secured Linux server in your own controlled VPC subnet in AWS. You will securely (by way of GPG or other encryption medium) provide a MySQL account for Percona against your RDS instance so that all communication between MySQL client and RDS happens on Amazon's private internal network. Any auditing or compliance requirements are to be made known to Percona prior to engagement.


Time limits

Sometimes customers ask us to limit or cap the hours we spend on a task. We are glad to do this, but ask customers to understand the ramifications.

  1. Percona's acceptance of a cap is NOT a promise to complete your task in the amount of time you’ve allowed us. Rather we are agreeing to do everything we reasonably can within your limit.

  2. The time limit is applied to the entire process. Time spent communicating with you, arranging logins, studying your situation, and documenting outcomes all applies against your cap.

  3. In some situations we cannot know the scope of the work needed until we actually begin, so a cap would be like asking a surgeon to remove a brain tumor in a fixed amount of time. The surgeon may encounter things after the operation has begun that simply cannot be skipped. In such cases Percona's initial "exploratory surgery" might be capped, but the "actual surgery" cannot be capped.

  4. All caps represent Percona's good faith intent but are not strictly binding. For example, Percona will disregard any cap in situations where stopping work would render your production system non-functional. We will finish the job despite the cap and expect full payment for it.

Minimum Limit Allowed

The minimum hourly limit or cap allowed by Percona is four hours.

Managing Time Limits

You can specify a limit or cap to Percona's sales team at the outset of your contract. You can also instruct your consultant later on to impose a cap, but this request must be clearly stated in writing via an email in our Customer Support Portal (CSP).

The CSP sends automated notices when caps are nearly exhausted. You can then decide whether to extend your cap or to discontinue work. Again all instructions regarding a cap must be in writing via the CSP.


How Percona Can Help

Percona's experts can maximize your application performance with our open source database support, managed services or consulting. For immediate database help, contact us at +1-888-316-9775 (USA), +44 203 608 6727 (Europe) or have us reach out to you:

Contact me!