Using replicaSetHorizons in MongoDB

When running MongoDB replica sets in containerized environments like Docker or Kubernetes, making nodes reachable from inside the cluster as well as from external clients can be a challenge. To solve this problem, this post will explain the Horizons feature of Percona Server for MongoDB.

Let’s start by looking at what happens behind the scenes when you connect to a replicaset URI.

Node auto-discovery

After connecting with a replset URI, the driver discovers the list of actual members by running the db.hello() command:

The list of hosts returned contains each member’s name as you provided it to the rs.initialize() command.

The node identity crisis

The names are resolvable inside the same network, so all is well in this case. But what happens when connecting from outside?

Typically, you would use names like mongo1-external.mydomain.com that correctly point to the external IP addresses of the members. The problem is that after the initial connection is made, the driver will perform auto-discovery and try to connect to the names as reported by DB.hello(). These are not resolvable from outside.

What if we connect by IP address directly? Again, the driver will get the names from the list above, try to reach those, and fail after the initial connection is made:

Even though mongo1-internal is not part of the connection string, the driver tries to reach it. So if the replica set members advertise their internal IPs or DNS names, clients outside can’t connect unless they can resolve that same name. We could work around that, but there’s another issue: the ports.

The port issue

In the containerized world, it is likely that you set up your containers to use default port 27017. However they might be mapped to a different external port since you have to avoid port collisions (think about the case where containers are co-located in the same host).

We need a way for replica set members to identify themselves with different names and ports, depending on whether the client is in the same network or outside. A concept similar to split-brain DNS.

What is Horizons?

Horizons is a MongoDB feature that allows replica set members to advertise different identities depending on the client’s access context, such as internal versus external networks.

With this, you can make the same MongoDB replica set usable from:

• Internal container network (using internal hostnames/IPs)
• External applications (using public IPs or DNS names)

MongoDB’s horizons rely on Server Name Indication (SNI) during the TLS handshake to determine which hostname and port to advertise. At connection time, clients present the hostname they used, and MongoDB uses that to return the proper set of endpoints. For that reason TLS is required in order for horizons to work.

Let’s walk through an example.

Example scenario: MongoDB Replica Set in Docker

You can run the following steps on your local machine to test the feature.

Step 1: Get your certificates ready

Let’s start by creating the required CA and certificates using Cloudflare’s PKI and TLS toolkit.

1: Create ca-csr.json

Generate the CA:

This creates:

• ca.pem — CA certificate
• ca-key.pem — CA private key

2: Create server-csr.json for each server, specifying both internal and external names in the “hosts” section so that our certificate is valid for everything.

3: Generate certificates using CFSSL

Resulting files:

• mongo{1,2,3}.pem — cert for server
• mongo{1,2,3}-key.pem, key for server
• mongo{1,2,3}-combined.pem, both in a single file as expected by mongo

Step 2: Docker compose setup

Create a file with docker compose configuration:

Here we are mapping our containers to ports 27017, 27018, and 27109 externally.

Now, start the services:

Step 3: Initiate the replica set with Horizons

Now let’s initiate the replica set with different host names and ports for external access.

Launch a shell into one of the containers:

Authenticate and initialize the replica set with this config:

Note: The “horizon” field here maps the external context to a different address than the internal one. Since we are going to test connecting from the local machine directly to the containers, set the horizons to localhost and the mapped ports.

Step 4: Connect from inside Docker

Spin up a new containerized client, or use one of the existing MongoDB containers:

It connects using internal Docker hostnames.

Step 5: Connect from outside Docker

From your local machine:

Step 6: Check the identities returned

As we have seen, MongoDB will resolve the external horizon names and connect successfully in both cases. You can verify the advertised hostnames and ports for the external connection:

Versus the internal case:

Conclusion

The horizons feature in MongoDB is a powerful tool to bridge the gap between internal and external connectivity, especially in containerized or multi-network deployments.

Horizon also has the following limitations:

• Using Horizons is only possible with TLS connections
• Duplicating domain names in Horizons is not allowed by MongoDB
• Using IP addresses in Horizon’s definitions is not allowed by MongoDB
• Horizons should be set for all members of a replica set, or not set at all

For some reason, this feature is not listed in the official MongoDB documentation; however, it is available in both Percona Server for MongoDB and MongoDB Community Edition. Also, Kubernetes users rejoice! Percona Operator for MongoDB supports Horizons since version 1.16.