Using Docker To Deploy Neon Serverless PostgreSQL

Announcement

I will be speaking at Percona Live 2023 about serverless PostgreSQL. Join us at this event if you are interested!

Introduction

Recently, Percona introduced Percona Builds for Neon (Introducing Percona Builds for Serverless PostgreSQL), which makes it easy to install and experiment with serverless PostgreSQL. But now, there’s an even more convenient way to explore the capabilities of serverless PostgreSQL — Docker images. In this article, we’ll explore the benefits of using Docker images for experimenting with serverless PostgreSQL and show you how to get started with them.

Architecture

To understand more about deployment procedures, we need to look a little more at Neon architecture.

There is a section in our Documentation (Introduction to Serverless PostgreSQL) and a short overview of the primary components:

• Page Server

  • The storage server with the primary goal of storing all data pages and WAL records

• Safe Keeper

  • A component to store WAL records in memory (to reduce latency). Because of the critical importance of these components, the recommendation is to deploy multiple of them. They will connect in Paxos group to guarantee data consistency.
  • Later the WAL records will be shipped to Pageserver, where it will use them to update data pages

• Compute Nodes

  • The component to take and handle user queries. Basically, you can view this as a PostgreSQL instance but without a storage layer

• Storage Broker

Storage Broker is a coordination component between WAL Service and Pageserver.

The Pageserver listens for GetPage@LSN requests from the Compute Nodes and responds with pages from the repository.

Deployment with Docker

We published a Docker image with all components in Docker Hub:

perconalab/neon Tags | Docker Hub

And the source code is located in our build repository:

Percona-Lab/serverless-postgresql-build (github.com)

Deployment steps

For my experiments, I follow these steps. I prefer to test a distributed deployment where each component is placed on different servers or virtual machines, that’s why I do not put it into docker-compose.

The assumption is that I use a reachable server with IP address 172.16.0.9. Unfortunately, I have to use physical IP addresses for connectivity in multiple servers docker deployment.

1. Deploy storage broker

2. Deploy safekeeper (or several of them for redundancy)

3. Deploy pageserver

4. Most interesting part – deploy compute nodes (components to handle client requests).

The most interesting part is deploying compute nodes, which are the components that handle client requests. Before we dive in, I need to introduce some new concepts that I’ll describe in more detail in future blog posts, so bear with me for now.

Concept 1: Tenants – Neon has the capability to serve multiple tenants, all located on the same pageserver but visible as separate PostgreSQL instances to clients.

Concept 2: Timelines – Each tenant can have multiple timelines and the ability to branch the current state into a new timeline. At a minimum, a tenant must have one timeline.

Now, let’s get back to deploying compute nodes.

Deploying compute node with creating new tenant and timeline:

This will create a lightweight PostgreSQL instance (compute node), accessible by port 55432. And you can connect to the instance with a normal PSQL client as:

Deploying compute node with existing tenant and timeline:

Assume we already have a tenant and timeline, and we want to attach a lightweight PostgreSQL instance (compute node):

To get timeline and tenant from the previous start, you can find them in docker logs ( docker logs compute) identified as:

Deploying compute node with branching from existing tenant and timeline:

This is the most exciting capability: we can fork (branch) existing data into a new timeline (more in the following blog posts):

Interested?

You are welcome to experiment with Neon using our Docker images.

For feedback and questions, please use our Forums: (Latest Percona Labs/Percona Build for Serverless PostgreSQL topics – Percona Community Forum)

Follow our blog for more information about setups, performance, and unique features of Serverless PostgreSQL.

If you would like to talk with us about builds for Serverless PostgreSQL or stay in touch for future updates – leave your contact details below.

Contact form

Also, if you are interested in possible Support for Serverless PostgreSQL, fill out the Contact form, and I will get in touch with you.