Easier Upgrades and Image Management for Postgres in Kubernetes

Lukas Fittl recently posted one of his 5 minutes of Postgres videos about his experimentation with different Kubernetes Postgres Operators: Postgres on Kubernetes, choosing the right operator, and handling major version upgrades. One passage about version updates caught my interest:

The other article I found interesting was this post by Andrew from the Crunchy Data team, where he describes how the PGO operator now makes it easy to do major version upgrades. This is actually pretty cool. I think this shows pretty well why an operator can be a lot more sophisticated than a simple pod.

While Postgres version upgrades are infrequent, they are very necessary. In order for our users and customers to be able to accomplish these vital maintenance operations we have invested considerable thought and work in how to streamline this process. Out of this has come the Minor Versions upgrade process, as well as the Major Version upgrade process that Lukas spoke about.

Our goal has been to make the life of someone running Postgres on Kubernetes much easier. The upgrade process that we have built in to the Operator also helps our users and customers keep their systems up to date with patches and updates with out breaking a sweat.

We would like to highlight some capabilities within the Operator to further that goal, including the new Pause feature that was introduced in PGO v5.2.0.

Minor Version Upgrades

As shown in the Crunchy Postgres for Kubernetes documentation, performing a minor upgrade of PostgreSQL is as simple as swapping out images in the PostgresCluster spec.

The Postgres image is referenced using the spec.image and looks similar to the below:

spec:
  image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.2-0

Diving into the tag a bit further, you will notice the 14.2-0 portion. This represents the Postgres minor version (14.2) and the patch number of the release 0. If the patch number is incremented (e.g. 14.2-1), this means that the container is rebuilt, but there are no changes to the Postgres version. If the minor version is incremented (e.g. 14.3-0), this means that there is a newer bug fix release of Postgres within the container.

To update the image, you just need to modify the spec.image field with the new image reference, e.g.

spec:
  image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.2-1

Yet, as simple as this process is, what if it could be even easier? And not only that, but what if you could simplify image configuration for all users, while also streamlining operator upgrades?

This post will explore how this can be achieved by using the “related images” feature within Crunchy Postgres for Kubernetes. As you will see, related images can make image management for your various PostgresCluster’s easier than ever before!

Defining Related Images

When exploring the PostgresCluster API included with Crunchy Postgres for Kubernetes (e.g. via the CRD reference), you might have noticed something similar to the following for the various image fields defined within the spec:

The example shown above describes the spec.image field that is used to define what image Crunchy Postgres for Kubernetes should use to run PostgreSQL. And starting with the “REQUIRED” column all the way to the right of this table, you’ll notice that image is actually optional.

If you’re asking how that’s even possible - i.e., obviously Crunchy Postgres for Kubernetes needs to know what image to use for PostgreSQL! - the description actually provides a clue. Specifically, by defining a RELATED_IMAGE_POSTGRES_ environment variable in the Crunchy Postgres Operator (PGO) deployment, it is possible to tell PGO itself exactly what images to use!

In the case of the image field for PostgreSQL shown above, this means anyone creating a PostgresCluster now simply needs to define a PostgreSQL version using spec.postgresVersion, and PGO will handle the rest. However, note that image still remains available for use in the PostgresCluster spec if/when it is needed (e.g. to override the related image value).

Available RELATED_IMAGE_ Environment Variables

While the above example demonstrates how related images can be used for PostgreSQL images, please note that all image fields in the PostgresCluster API have a corresponding RELATED_IMAGE_ environment variable.

For instance, if you look at the PGO deployment within the v5.2.0 Kustomize installer, you will see the following:

- name: RELATED_IMAGE_POSTGRES_13
  value: 'registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-13.8-1'
- name: RELATED_IMAGE_POSTGRES_13_GIS_3.0
  value: 'registry.developers.crunchydata.com/crunchydata/crunchy-postgres-gis:ubi8-13.8-3.0-1'
- name: RELATED_IMAGE_POSTGRES_13_GIS_3.1
  value: 'registry.developers.crunchydata.com/crunchydata/crunchy-postgres-gis:ubi8-13.8-3.1-1'
- name: RELATED_IMAGE_POSTGRES_14
  value: 'registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.5-1'
- name: RELATED_IMAGE_POSTGRES_14_GIS_3.1
  value: 'registry.developers.crunchydata.com/crunchydata/crunchy-postgres-gis:ubi8-14.5-3.1-1'
- name: RELATED_IMAGE_POSTGRES_14_GIS_3.2
  value: 'registry.developers.crunchydata.com/crunchydata/crunchy-postgres-gis:ubi8-14.5-3.2-1'
- name: RELATED_IMAGE_PGADMIN
  value: 'registry.developers.crunchydata.com/crunchydata/crunchy-pgadmin4:ubi8-4.30-4'
- name: RELATED_IMAGE_PGBACKREST
  value: 'registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.40-1'
- name: RELATED_IMAGE_PGBOUNCER
  value: 'registry.developers.crunchydata.com/crunchydata/crunchy-pgbouncer:ubi8-1.17-1'
- name: RELATED_IMAGE_PGEXPORTER
  value: 'registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter:ubi8-5.2.0-0'

Smooth Updates & Upgrades Using Related Images

Let’s now get back to the minor PostgreSQL upgrade use case discussed in the introduction. As you might recall, a minor PostgreSQL image typically involves updating the spec.image field in the PostgresCluster spec.

However, with the proper RELATED_IMAGE_ environment variable defined within the PGO deployment (specifically for the latest patch version of PostgreSQL available), this is no longer needed! For instance, if RELATED_IMAGE_POSTGRES_14 to configured for PostgreSQL 14.4 image, it can be updated to v14.5 as follows:

kubectl set env -n postgres-operator deployment/pgo \
    RELATED_IMAGE_POSTGRES_14=registry.developers.crunchydata.com/crunchydata/crunchy-postgres:centos8-14.5-0

And as soon as the above patch is applied and the PGO deployment is restarted, PGO will safely rollout the the minor upgrade to all PostgresCluster’s using PostgreSQL 14. This means PGO administrators can now easily ensure all PostgreSQL clusters using v14 are updated to the latest patched version of PostgreSQL (which includes the latest bug fixes, security fixes, etc.).

And while this demonstrates a clear benefit that related images brings to the PostgreSQL minor upgrade process, there are other benefits as well. To provide one more great example, you might have noticed the following in the PGO upgrade documentation when upgrading to PGO v5.1.0+:

Relatedly, if you are instead using the RELATED_IMAGE environment variables to set the image values, you would instead check and update these as needed before redeploying PGO.

What this is effectively saying is that by using related images, the pre-requisite to manually update all crunchy-postgres and crunchy-pgbackrest images prior to the PGO upgrade is no longer even applicable. Simply upgrade PGO, and since all installers come pre-defined with RELATED_IMAGE_ environment variables that included latest component images available, PGO will simply handle the rest!

There are many scenarios in which related images are useful. Those managing PostgresCluster’s can ensure the latest patches/fixes, security updates, etc. are applied and managed via a central/single configuration. And those who are provisioning PostgresClusters no longer even need to worry/think about what image tags to use.

Controlling Updates Using “Pause”

Since the RELATED_IMAGE_ environment variables in the PGO deployment itself controls the various images utilized for all PostgresCluster’s, this does mean the a RELATED_IMAGE_ update could cause many clusters within your environment to rollout the change concurrently, which might not always be desirable. Fortunately the Crunchy Postgres for Kubernetes v5.2.0 release provided a way to control rollouts for individual PostgresClusters in the form of a new “pause” capability.

Specifically, by setting the paused field in the PostgresCluster spec to true, it is possible to tell PGO to temporarily stand-down from reconciling any changes to the PostgresCluster:

kubectl patch postgrescluster/hippo -n postgres-operator --type merge \
  --patch '{"spec":{"paused": true}}'

This means a new image value set via a RELATED_IMAGE_ environment variable will not be applied and rolled out until paused is either removed, or set to false. Therefore, using this capability those administering the PGO deployment can control the images utilized via related images, while those managing the PostgresCluster’s can control exactly when those updates are applied.

For more information on pausing PostgresCluster reconciliation and rollouts, please see the Administrative Tasks section of the latest PGO documentation.

Conclusion

Lukas noted that,

The Crunchy Operator clearly has a lot of sophistication built into it, which would give me confidence if I were to deploy it in production.

We are thrilled to see our hard work around streamlining upgrades out in the wild and are excited for the future of this feature. If you have any questions about our Operator and getting it up and running for yourself please feel free to reach out.