External Metadata Database Configuration
Audience: System Administrators
Content Summary: This page outlines how to configure an external metadata database for Immuta instead of using Immuta's built-in PostgreSQL Metadata Database that runs in Kubernetes.
External Metadata Database Only Supported in Kubernetes Deployments
Configuring an external metadata database is not supported in the Single Node Docker deployment method.
You can replace Immuta's built-in PostgreSQL Metadata Database with an external PostgreSQL instance when deploying Immuta in Kubernetes. Running the metadata database outside of Kubernetes eliminates the variability introduced by Kubernetes scaling and scheduling events, and infrastructure administrators have greater control over the configuration of the database overall, such as options for high availability and disaster recovery.
PostgreSQL Version 12
Use PostgreSQL version 12 for your external database to match the version used by Immuta. Immuta only supports PostgreSQL, not PostgreSQL abstraction layers such as AWS Aurora.
When configuring your Helm values, enable an external metadata database by following these steps:
- Enable an external metadata database by setting
immuta-values.yamlfile and passing the connection information for the PostgreSQL instance under the key
- If migrating from Immuta's built-in metadata database to your external PostgreSQL database,
database: enabled: false externalDatabase: host: replace-with-your-external-postgres.database.hostname password: replace-with-your-bometauserpassword superuser: username: postgres password: postgrespassword backup: enabled: true restore: enabled: true
Once set up and working, an in-place upgrade or a backup and restore will create a hook job. This hook job will perform upgrades, migrations, and restores without needing to touch the external database.
External Database Restore
The following process will destroy the database and recreate it for a restore. A backup is necessary for this action to be completed.
In the event that the external database is no longer working and a fresh restore is needed, you can connect to the external database either via a tool or the platform UI and run the following:
drop database bometadata; create database bometadata;
External Database Object
externalDatabase object is detailed below and in the
Immuta Helm Chart Options.
||Hostname of the external PostgreSQL database instance.||
||Port of the external PostgreSQL database instance.||
||The mode for the database connection. Supported values are
||Username for the superuser used to initialize the PostgreSQL instance.||
||Password for the superuser used to initialize the PostgreSQL instance.||
||Username that Immuta creates and uses for the application.||
||Password associated with
||Database name that Immuta uses.||
||Enable flag for external database backups. Only used when
||Enable flag for the external database restore. Only used when
Additionally, it is possible to use
existingSecret instead of setting
externalDatabase.superuser.password in the Helm values. These passwords map to the same keys that are used for the
built-in database. For example,
apiVersion: v1 kind: Secret metadata: name: immuta-secret data: databasePassword: <password for externalDatabase.password> databaseSuperuserPassword: <password for externalDatabase.superuser.password>
Superuser User Requirements
There are two options for the user used for
- a user with superuser role attribute (such as the
- a user with reduced privileges (this option requires preliminary setup).
A user with superuser permission is required to use the built-in backup functionality. It is possible to disable built-in backups for the external PostgreSQL instance when using a less-privileged user; however, care must be used to ensure that the Query Engine is in-sync with the external database if backups are restored. One situation in which built-in backups might be disabled for the external database is when database provider backups are used. For example, an AWS RDS instance might be configured to take backups, and the user would rely on those backups for the metadata database.
If using a less-privileged user, the
bometadata database must exist ahead of time, and the built-in
extension must be created in it. The user must have
CREATEROLE permissions and have been granted
TEMPORARY (ALL) on the
CREATE DATABASE bometadata; \connect bometadata CREATE EXTENSION pgcrypto; CREATE USER example_immuta_admin WITH CREATEROLE ENCRYPTED PASSWORD ''; GRANT ALL ON DATABASE bometadata TO example_immuta_admin; CREATE ROLE bometa WITH CREATEROlE LOGIN ENCRYPTED PASSWORD ''; GRANT ALL ON DATABASE bometadata TO bometa; ALTER ROLE bometa SET search_path TO bometadata,public;
Migrating from Immuta's Built-In PostgreSQL Database to External Metadata Database
For existing deployments, you can migrate from the built-in database to an external database. To migrate, backups must be configured.
- Follow step 1 (Capture a Complete Current Backup) of this process to back up your instance. Do not complete the other steps in that tutorial.
- Set the external database Helm values outlined above and