Database as a Service in Gcloud

Postgresql 11 Self Provisioning

Version 1.0 25 Aug 2020

Authors : Julien Lefevre DBA

Els Uyttenhove DBA
Michel Stevelinck DBA

Department: Database Team

 Postgresql 11 Gcloud Self Provisioning Version 1.0

Contents
Executive summary .................................................................................................................................................................................. 3
Safe harbor statement................................................................................................................................................................................ 3
Versions history ........................................................................................................................................................................................ 6
Introduction .............................................................................................................................................................................................. 7
The Gcloud request page .......................................................................................................................................................................... 8
Accessing the Gcloud request page ....................................................................................................................................................... 8
Select your Group/Project ..................................................................................................................................................................... 8
Selecting a Postgresql template to create a Database (select 11 or 9.6) .................................................................................................... 9
Mail notifications .................................................................................................................................................................................... 12
Connecting to your database using DBeaver .......................................................................................................................................... 13
Stored procedures for developers ............................................................................................................................................................ 16
dbaas_add_database(v_dbname varchar, v_password varchar) .......................................................................................................... 16
dbaas_add_schema (v_dbname varchar, v_schema varchar, v_password varchar) ............................................................................. 17
dbaas_add_grants(v_dbname varchar) ................................................................................................................................................ 18
dbaas_change_password (v_user varchar,v_new_password varchar) ................................................................................................. 18
dbaas_list_schema()............................................................................................................................................................................ 19
dbaas_list_diskspace ........................................................................................................................................................................... 19
dbaas_list_dumpfiles .......................................................................................................................................................................... 19
dbaas_remove_dumpfile (v_dumpfile varchar) .................................................................................................................................. 20
dbaas_take_database_dump(v_dbname varchar) ................................................................................................................................ 21
dbaas_restore_database(v_dumpfile varchar) ..................................................................................................................................... 21
dbaas_duplicate_database(v_source_dbname varchar, v_target_dbname varchar) ............................................................................. 22
Monitoring .............................................................................................................................................................................................. 24
Connection to monitoring ................................................................................................................................................................... 24
Server level monitoring....................................................................................................................................................................... 24
Database level monitoring .................................................................................................................................................................. 25
Monitoring with multiple databases .................................................................................................................................................... 25
The Ares database ............................................................................................................................................................................... 26
Postgresql extensions .............................................................................................................................................................................. 27
Extensions added by the DBA Team .................................................................................................................................................. 27
Supplemental extensions required for your project ............................................................................................................................. 27
Logical backup ....................................................................................................................................................................................... 28
Daily backup of the different schemas ................................................................................................................................................ 28
Manual backup of schemas ................................................................................................................................................................. 29
Support and SLA for the Gcloud environment ....................................................................................................................................... 30
EnterpriseDB Postgresql documentation ................................................................................................................................................ 31

2|P a ge
Postgresql 11 Gcloud Self Provisioning Version 1.0

Executive summary

This document provides information to enable the self-provisioning of Postgresql 11 databases in the Gcloud
environment. This kind of service is also named “Database as a Service” (DBaaS).

VMaaS is a Smals ‘specific self-provisioning of Virtual Machine (VM) on which the Postgresql 11 (or 9.6) database
can be selected for installation and configuration. This database is typically used for developers to perform testing
and feasibility studies.

The purpose of this document is to provide all needed information to developers so that they could start testing
Postgresql 11 in no time.

Safe harbor statement

The following is intended to outline our general services direction. It is intended for information purposes only, and
may not be incorporated into any contract. It is not a commitment to deliver any material, service, or functionality,
and should not be relied upon in making purchasing decisions. The development, release, and timing of any features
or functionality described for DBaaS services remain at the sole discretion of the Smals Database Team.

Information in this document is subject to change without notice, please check regularly for updates!

3|P a ge
Postgresql 11 Gcloud Self Provisioning Version 1.0

Versions history

Contributors Release Comments

Julien Lefevre 1.0 Initial redaction – based on doc for 96

6|P a ge
Postgresql 11 Gcloud Self Provisioning Version 1.0

Introduction

This document provides instructions and configuration to enable the self-provisioning of a Postgresql 11 database in
the Gcloud environment.

This document also provides important information regarding responsibilities and alignment on procedures for the
developers to enable a smooth transition from the Gcloud development to a real production environment.

7|P a ge
 Postgresql 11 Gcloud Self Provisioning Version 1.0

The Gcloud request page

Accessing the Gcloud request page

To access the Gcloud catalog page, go to the following intranet URL:

https://cloudforms.gcloud.belgium.be/catalog/explorer

Then enter your windows credentials and click OK

Select your Group/Project

Then check if you are in the Group/Project for which you want a new asset (this associates asset
costs to a project)

In Top right corner :

8|P a ge
Postgresql 11 Gcloud Self Provisioning Version 1.0

Selecting a Postgresql template to create a Database (select 11 or 9.6)

Take care that the “current standard release” contains more stored procedures than other releases.

Enter the tag for the service name (same screen for all releases)

9|P a ge
Postgresql 11 Gcloud Self Provisioning Version 1.0

Then specify the estimated size for your Data

This is the size for your Data only. Other disk storage will be allocated accordingly:

Storage Size VM Mount point

Data Between 20 Gb-200Gb /opt/pgsql/data/2001
Backup 2x Data size /opt/pgsql/backup/200
Temp 25% Data with max 50 1
/opt/pgsql/tmp/2001
Wal GbData size
1x /opt/pgsql/wal/2001
System 50 Gb /

Enter the Contact email addresses

WARNING Those email addresses are very important as they will be used to provide the credentials and
connectivity information to access your database.

Then click on submit

10 | P a g e
Postgresql 11 Gcloud Self Provisioning Version 1.0

The following screen will appear. It confirms your order.

11 | P a g e
Postgresql 11 Gcloud Self Provisioning Version 1.0

Mail notifications

You will receive 2 mails.

1. One mail for the approval/confirmation of your order, similar to this one :

Once you order is auto-approved, the VM will be created, software installed, Database created, …etc. This
entire process takes more or less 20 minutes.

2. Once all the process is completed, you will receive a second mail which provides you all information to
connect to the database:

You can now connect to your database using an SQL client ( i.e. DBeaver).

INFO Should you need to contact the DBA team to have support, please provide us the following info:
Hostname, IP, Port and Database name.

12 | P a g e
Postgresql 11 Gcloud Self Provisioning Version 1.0

Connecting to your database using DBeaver

We recommend version > 4.0

We assume that your DBeaver software is installed. You can have the latest release here:

http://dbeaver.jkiss.org/

Now that you have received the credentials and connection info for your database, you can enter those in a new
session of DBeaver.

In DBeaver click on new connection

The following windows will open. Select PostgreSQL and click on next.

13 | P a g e
Postgresql 11 Gcloud Self Provisioning Version 1.0

Fill in the connection info accordingly

The test connection should return you this

Click Next. Leave network unchanged and click next.

14 | P a g e
Postgresql 11 Gcloud Self Provisioning Version 1.0

On the last screen for the connection creation, adjust the name of your connection

You now have a new connection in your DBeaver Database Navigator. Double click on it to access your database.

You can now execute SQL queries and build or import your database objects.

15 | P a g e
 Postgresql 11 Gcloud Self Provisioning Version 1.0

Stored procedures for developers

WARNING
In version 11, the procedures can only by executed as user sysadmin_prd connected to database sysadmin.
The password for the user sysadmin_prd is sysadminpg11.

WARNING
In postgresql, max identifier length is 63 bytes. Identifiers longer than 63 characters will be truncated to the
allowed length of 63. Think of it when choosing a name for a database or schema, especially when you plan to
duplicate the database

dbaas_add_database(v_dbname varchar, v_password varchar)

This procedure

creates a new database <v_dbname>

adds a schema schema_<v_dbname>_adm
adds users <v_dbname>_adm/prd/clt with password <v_password>
adds application roles rl_<v_dbname>_prd/clt

INFO The new created user <v_dbname>_adm/prd/clt can only connect to the associated database
<v_dbname>.
A schema can only exist in 1 database. You can use the procedure dbaas_list_schema to display the
existing schemas per database.
If you need another schema in an existing database, please use the procedure dbaas_add_schema.

Example:

16 | P a g e
 Postgresql 11 Gcloud Self Provisioning Version 1.0

Result:

dbaas_add_schema (v_dbname varchar, v_schema varchar, v_password varchar)

This procedure

adds a schema schema_<v_schema>_adm in <v_dbname>

adds users <v_schema>_adm/prd/clt with password <v_password>
adds application roles rl_<v_schema>_prd/clt

INFO The new created user <v_schema>_adm/prd/clt can only connect to the database <v_dbname>,
containing the associated schema schema_<v_schema>_adm
A schema can only exist in 1 database. You can use the procedure dbaas_list_schema to display the
existing schemas per database.

Example:

Result:

17 | P a g e
 Postgresql 11 Gcloud Self Provisioning Version 1.0

dbaas_add_grants(v_dbname varchar)

This procedure adds grants for all tables, views and sequences of all schemas in database <v_dbname>.

for tables:

grant select, update, delete, insert on schema_<app1>_adm.<table_name> to rl_app1_prd

grant select on schema_<app1>_adm.<table_name> to rl_app1_clt

for views:

grant select on schema_<app1>_adm.<view_name> to rl_app1_clt

for sequences:

grant select,insert on schema_<app1>_adm.<sequence_name> to rl_app1_prd

dbaas_change_password (v_user varchar,v_new_password varchar)

This procedure changes the password of a user <v_user> to <v_password>.

Example:

Result:

18 | P a g e
 Postgresql 11 Gcloud Self Provisioning Version 1.0

dbaas_list_schema()

This procedure lists all user schemas in all databases on your VM.

INFO for better output, execute it as “select * from dbaas_list_schema();”

dbaas_list_diskspace

This procedure shows the diskspace of the data and backup mount point on your VM.

dbaas_list_dumpfiles

This procedure lists all available backup dumpfiles.

There are the daily backup files, generated by the scheduled job (see “Daily backup of the Postgresql Cluster”) and
the backups taken by yourself with the procedure “dbaas_take_database_dump”.

The backup dumpfiles are of the format dump_database_<dbname>_YYYYMMDD_HH24MISS.sql

Example:

19 | P a g e
 Postgresql 11 Gcloud Self Provisioning Version 1.0

Result:

dbaas_remove_dumpfile (v_dumpfile varchar)

With this procedure, you can remove dumpfiles , if you don’t need them anymore or you are limited in diskspace.

INFO You need to specify the dumpfile the same way as displayed in the output from the procedure
dbaas_list_dumpfiles

Example:

Result

20 | P a g e
 Postgresql 11 Gcloud Self Provisioning Version 1.0

dbaas_take_database_dump(v_dbname varchar)

This procedure will take a dump of the database <v_dbname>.

The name of the dumpfile is of the format dump_database_<v_dbname>_YYYYMMDD_HH24MISS.sql.

Example:

Result:

dbaas_restore_database(v_dumpfile varchar)
This procedure

deletes all data from your database; users and application roles are not removed.
restores the database with the content from the dumpfile

INFO The database to be restored, is the one specified in the name of the dumpfile.
All your data will be replaced. If you do not want this, please use the procedure
dbaas_duplicate_database.
You can take a backup with procedure dbaas_take_database_dump before starting your restore.
You need to specify the dumpfile the same way as displayed in the output from the procedure
dbaas_list_dumpfiles.
If a new schema is added (as part of the restore), associated users and application roles will be
created. The password for the “restored” users is per default “pg11”. You can change the password
with the procedure dbaas_change_password.

21 | P a g e
 Postgresql 11 Gcloud Self Provisioning Version 1.0

Example:

With this dumpfile, you will do a restore of database ares02.

Result:

dbaas_duplicate_database(v_source_dbname varchar, v_target_dbname varchar)

This procedure

takes a backup from database <v_source_dbname>

restores this backup in a new database <v_target-dbname>

INFO All schemas will be duplicated in the target database with a new name
schema_<v_target_dbname>_xxx_adm (xxx is the schema name from the source database).
For each schema in the duplicated database, associated users and roles will.be added with a default
password “pg96”. You can change the password with the procedure dbaas_change_password.
Grants from the source database will be transferred.

22 | P a g e
Postgresql 11 Gcloud Self Provisioning Version 1.0

Example:

In database ares, suppose you have 3 schemas: schema_ares01_adm, schema_ares02_adm, schema_ares03_adm

The procedure dbaas_duplicate_database (‘ares’,’db02’) will duplicate all the content of database ares into database
db02.

Schemas will be renamed as follows:

schema_ares01_adm ==> schema_db02_ares01_adm

schema_ares02_adm ==> schema_db02_ares02_adm

schema_ares03_adm ==> schema_db02_ares03_adm

Associated roles and users will be added as follows:

roles: rl_db02_ares01_prd/clt, rl_db02_ares02_prd/clt, rl_db02_ares03_prd/clt

users: db02_ares01_adm/prd/clt, db02_ares02_adm/prd/clt, db02_ares03_adm/prd/clt

Grants will also be duplicated to the corresponding roles rl_db02_ares01_prd/clt,...

23 | P a g e
 Postgresql 11 Gcloud Self Provisioning Version 1.0

Monitoring

Connection to monitoring
A useful monitoring is available for each Database service created. This monitoring is available using a web browser
(we recommend Google Chrome).

The URL is the hostname directly followed by “:2005”. User is monitoring … and password is monitoring

In my example, my monitoring URL will be dbaas-pg-4755.cloud.smals.be:2005

Server level monitoring

Once connected, you will receive the Server level

24 | P a g e
 Postgresql 11 Gcloud Self Provisioning Version 1.0

Database level monitoring

Monitoring with multiple databases

Should you have multiple databases, you can select the database for monitoring in the following list

I only have one database here.

Of course, if you select another Database, it will only affect the database monitoring, not the server monitoring.

25 | P a g e
 Postgresql 11 Gcloud Self Provisioning Version 1.0

The Ares database

The ARES database database is created for the DEV environment.

The Ares database has been created based on template1 and the template1 contains all extensions as agreed by the
DBA Team.

The Ares database contains different schemas: ares_adm, ares_prd and ares_clt

INFO The ARES Database, like any new EDB Postgresql database has the following custom setting:
edb_redwood_date = off
autovacuum =on

26 | P a g e
 Postgresql 11 Gcloud Self Provisioning Version 1.0

Postgresql extensions

Extensions added by the DBA Team

Some extensions have been added to the default EDB Postgres 9.4 by the DBA Team.

Extensions
plpython3u
pg_prewarm
Pgstattuple
btree_gist
unaccent

Supplemental extensions required for your project

It is possible that your project requires specific extensions which are currently not installed. As you have all
privileges, you may be tempted to add those extensions yourself.

WARNING
The DBA Team strongly recommend that you contact them should you need supplemental extensions.
Indeed some extensions require specific Unix packages (rpm) or may conflict other extensions already
present. You need to work with the DBA Team to align your needs with the Postgresql standards defined
by the DBA Team. The risk is that some extensions may not be available in production or that extensions’
installation delays your deployment schedule.

Take care that some extensions are available from Postgres Community and other from EnterpriseDB.
Mixing extensions may lead to unpredictable results or unstable environment.
Keep in mind that our Postgres Databases are supported by EnterpriseDB.

In any case, please contact the DBA Team before installing new extensions.

27 | P a g e
 Postgresql 11 Gcloud Self Provisioning Version 1.0

Logical backup

Daily backup of the different schemas

Once your ARES database is created, a logical backup (dump) is scheduled on a daily basis at 19:00 with retention of
7 days. The backup is performed using the stored procedures for backup:

The result backup are located in a separated mount point so that they don’t block the evolution of other files (data,
wal).

WARNING The scheduled daily backups are purged with a retention of 3 days using the following pattern:
dump_database_*.sql

28 | P a g e
Postgresql 11 Gcloud Self Provisioning Version 1.0

Manual backup of schemas

In addition to scheduled backup, you can perform custom backup on demand. You can use the provided stored
procedures or external tools to export/import your data.

Example: tools from DBeaver

HINT It is up to you to check that the backup is running or to take a manual backup at the right time.
You could also extend the data protection by exporting the backup file to your PC or to a shared drive. In case of
an accidental remove of your VM, you can rebuild it pretty rapidly.

WARNING Take care that manual and scheduled backups are logical backups only (dump) and therefore
could not perform PITR (Point In Time Recovery). The logical backups are taken at a “database” level without
any knowledge of transactions (unless physical backups). The recovery operation for such backup will put back
the database as it was when the backup was taken.

29 | P a g e
Postgresql 11 Gcloud Self Provisioning Version 1.0

Support and SLA for the Gcloud environment

The support level for the Gcloud DEV environment is “BRONZE”.

We provide a “best effort” support to provide assistance when required especially if the Gcloud provisioning is not
working at all.

We are also please to assist you for defining the best strategy in term of schema and data structure.

WARNING It is extremely risky for your project to start it in the Gcloud DEV and to continue going live in
the same dev/sandbox infrastructure. You will have no support whatsoever! No resilient backups, no proactive
monitoring, no patching, …!

30 | P a g e
Postgresql 11 Gcloud Self Provisioning Version 1.0

EnterpriseDB Postgresql documentation

The documentation for EDB Postgresql is available here : (select “Enterprise”).

http://www.enterprisedb.com/products-services-training/products/documentation

You can also access older versions (the latest version is displayed by default).

31 | P a g e