Configuring a Read-Only Web Interface for Orchestrator

Configure read only interface orchestratorIn the MySQL ecosystem, orchestrator is the most popular and well-respected high availability and topology management tool, integrating well with other solutions such as ProxySQL. It facilitates automatic (or manual) discovery, refactoring and recovery of a replicated MySQL environment, and comes complete with both command-line (CLI) and web interfaces for both humans and machines to interact with.

As we all know, humans are prone to errors and as such accidents can happen, particularly when humans and computers interact with each other! Recently, one of these situations related to the web interface of orchestrator during topology refactoring with its drag-and-drop capabilities, where a drop occurred unintentionally and thus had an impact on replication.

When using orchestrator in CLI mode it is possible to set the ReadOnly flag to make the web interface and API unable to make changes on the user’s behalf, which would avoid such accidents. However, what if you want to maintain access to the API for management? Thankfully, there are a number of settings that already exist that when combined together with a reverse proxy (such as NGINX) make this possible.

Configuration of Orchestrator to Support a Read-Only Web Interface

The configuration of orchestrator is well documented, so if you are unfamiliar with configuring it then its documentation is the best place to start.

In order to make the web interface read-only you will need to use the following configuration options:

  • AuthenticationMethod
  • AuthUserHeader

There are a number of choices for the AuthenticationMethod and to achieve the goal of a read-only interface we will set this to:

This will instruct orchestrator to expect authentication to occur via HTTP headers, which brings us to the second setting option, AuthUserHeader. For the purpose of this post, the header name will be set to X-Auth-User, although you are able to set this to whatever makes sense to you, so long as it can be used as a header name.

For the sake of simplicity, we will also set the PowerAuthUsers option to match only the HTTPAuthUser, which allows us to easily revert back to direct full access with "AuthenticationMethod": "basic" should we wish to do so.

There are some other options that we will now set, not relating to authentication, so that we can choose the path via which the web and API interfaces of orchestrator will be accessible in a browser, etc:

To help with configuration of NGINX we will make orchestrator bind to a local address:

Configuration of NGINX to Support Orchestrator

We will assume that we already have a working NGINX configuration and just need to make the necessary changes to provide access to orchestrator. The ngx_http_core_module enables configuration of a location rule to match the value of the URLPrefix that we chose for orchestrator. Here is the full snippet that we will use:

This configuration instructs NGINX to use:

  • The Basic HTTP Authentication Scheme using the contents of /etc/nginx/orchestrator.htpasswd as the source for credentials.
  • Forward request headers to orchestrator, except for the Authorization header that would normally be sent as a result of the client authenticating.
  • Dynamically set the value of a variable ($orchestrator_auth chosen for the example) to either the authenticated user, or force it to readonly for requests to the web interface (/web/).
  • Set an HTTP header named X-Auth-User (our AuthUserHeader) to have a value of $orchestrator_auth.
  • Pass all requests on to the service at 127.0.1.1:3000 (our ListenAddress).

Create the auth_basic_user_file

In order to be able to use authentication, it will be necessary to add at least one user to the auth_basic_user_file. The easiest way to manage authentication user files is with the htpaswd tool.

The following commands will create the file (-c) and add the power user “orchestrator” and then add another user, “readonly“, both using the more secure bcrypt encryption: (-B).

Apply Configuration and Test

In order to activate this, we will need to either reload the configuration of each service that was running or start those that were not. We will presume that both orchestrator and NGINX were already running, so apply the configs to the running instances:

We will expect requests going directly to orchestrator to fail, while requests going via NGINX will succeed.

If you were to now use the web interface via NGINX and authenticate using the power user, you would see that the user would be displayed as “readonly” and the icon indicating that the interface is in read-only mode would show up.

orchestrator header bar

Header navigation bar from orchestrator web interface showing readonly access

We Still Have an Issue

This is not 100% fool-proof as there is still a way to circumvent it, although it would require access to the server that is hosting orchestrator and NGINX. However, it prevents accidents from happening as you would have to be rather determined to bypass the reverse proxy to gain write access for the web interface.

I have sent a pull request to add a new option to orchestrator that would simplify the whole process and also reach the 100% read-only goal, but for now, it is necessary to take these extra steps.

Share this post

Comments (5)

  • rautamiekka Reply

    1 problem: not everyone uses NGINX.

    March 6, 2020 at 10:47 am
    • perconaceri Reply

      You should be able to translate the approach to whichever software you are using, it is not NGINX-specific other than the syntax used. Unfortunately, testing each web server is out of scope.

      For example, information relating to configuring Apache to proxy requests is already in the orchestrator documentation, so you would just need to set the rules to apply to the specific requests:
      https://github.com/openark/orchestrator/blob/master/docs/security.md

      March 6, 2020 at 12:09 pm
      • Alex Reply

        Hello. Can you help solve this problem?
        ClusterAlias unpredictably moves from one cluster to another #1132
        https://github.com/openark/orchestrator/issues/1132

        April 29, 2020 at 6:03 am
        • perconaceri Reply

          Sorry Alex, you will need to continue discussing with Shlomi in terms of a change in behaviour. Just be patient for replies from him.

          If you know all of your instances in advance, you can assign all aliases in the config with ClusterNameToAlias, adding the same cluster name for each of its members.

          Alternatively, you should be able to use DetectClusterAliasQuery and create a query that reads from a table in the database.

          Hope that helps.

          April 29, 2020 at 6:36 am
          • Alex

            We have a DetectClusterAliasQuery query. And we also have the answer to it in MySQL.

            ClusterNameToAlias will not help in this case, since ClusterAlias is defined in the MySQL database and successfully retrieved using DetectClusterAliasQuery.

            In this case, ClusterNameToAlias assigns the specified ClusterAlias to only those hosts that were excluded from the main cluster in the FailoverProcess, for example, the OldMaster host.

            And our problem is that ClusterAlias of the main cluster can be suddenly assigned, for example, to the same OldMaster.

            April 29, 2020 at 1:00 pm

Leave a Reply