Administrator guide for the HQbird replication cluster: hqclusternode and hqbirdrcm (RCM). Installers: HQbird 2027 Download. Русская версия: на русском.

Contents


0. Introduction

This guide describes how to install and use the HQbird replication cluster: the hqclusternode nodes and the cluster manager hqbirdrcm (RCM).

This is a guide for the Firebird administrator. We assume that you know Firebird, gfix and nbackup. You do not need to know the internal workings of a node.

What the cluster consists of

Component How many Where it runs What it does
Firebird (HQbird) one or more instances on each database server stores databases, writes replication segments
fbagent one per Firebird instance alongside Firebird stops and starts Firebird at the request of a node
hqclusternode (node) one per instance and role alongside Firebird configures replication, transfers segments, performs replica initialisation
hqbirdrcm (RCM) one per cluster on any host that can see all nodes web interface, commands, alerts, statistics

A node has a role: master or replica. Two nodes can run on a single Firebird instance: a master node and a replica node. In that case some databases of this Firebird are masters and others are replicas.

How it works

  1. On the master, Firebird writes database changes to segments (replication log files).
  2. The master node sends the segments to the replica node over a secured channel (mTLS).
  3. The replica node places the segments in the incoming segments folder. The replica’s Firebird applies them to its copy of the database.
  4. The master node itself transfers the first copy of the database to the replica. This is called initialisation (reinit). It goes through nbackup, and the master keeps running during this time.
  5. The nodes send statistics to RCM. RCM shows the state of the cluster and sends operator commands to the nodes.
How the cluster works: segments, initialisation, statistics and operator commands
How the cluster works: segments, initialisation, statistics and operator commands

Supported versions

  • Firebird / HQbird 5.0 and 4.0 — all features.
  • HQbird 3.0 and 2.5 — replication via the replconf plugin. There are no publications; the set of tables is defined by filters. The differences are described in the “HQbird 2.5/3.0” boxes.

The major version of Firebird on the master and on the replica must match. Otherwise the replica node will refuse initialisation.

Ports

Port Who listens Who connects What for
7051 node other nodes and RCM node API, transfer of segments and database copies
7443 RCM all nodes receiving statistics from nodes
7444 RCM operator web interface and operator API. Installation via RCM (chapter 1.3): HTTPS on all addresses. Installation via HQCenter Admin or manually: HTTP, 127.0.0.1 only
7450–7459 fbagent, while the linking window is open RCM linking hosts (chapter 1.3)
13050 fbagent, 127.0.0.1 only node on the same host Firebird management
3050 Firebird applications normal database operation

If there are two nodes on one host (master and replica), the second node has a different port. When installing via RCM, the second node gets port 7061. In other cases you set the port yourself, for example 7052.

Port 13050 must not be exposed to the network. Port 7444 with HTTP (installation via HQCenter Admin or manually) must not be either: connect to the RCM web interface through an SSH tunnel (chapter 1). When installing via RCM, port 7444 runs over HTTPS, and the installer opens it in the firewall.

The main rule: no fbagent, no changes

A node changes the replication configuration only through fbagent: it needs to restart Firebird. If fbagent is unavailable, the node refuses any action that changes the configuration. It does not work “partially”.

Terms

Term Meaning
Segment a replication log file with database changes
Publication the list of master tables whose changes go into segments (Firebird 4/5)
Initialisation (reinit) transfer of a copy of the database from the master to the replica via nbackup
Database GUID the Firebird database identifier; RCM uses it to link the master and the replica
Peer partner node: for a master, its replicas
Companion the second node on the same Firebird instance (master and replica on one host)
Enroll registration of master databases in the node

How to read the guide

Chapter Topic
1. Installation installation of nodes and RCM via HQCenter Admin, via RCM without HQCenter Admin and manually, Linux and Windows
2. Cluster configuration master → replica links, node settings, RCM users
3. Adding a database to the master Enroll, restarting Firebird, verification
4. How to make a database a replica stages, preparing the replica, disk space
5. Initialisation reinit steps, impact on operation, long transactions, speed, errors
6. Configuring the publication Firebird 4/5 publication, tables, new tables, HQbird 2.5/3.0
7. Other replication options replication.conf parameters, archive, transfer, alerts, pause, logs
8. Converting a replica into a regular database Make regular, Promote to master, new GUID, services (switchover, failover), switchover checklist

All chapters use one example:

What Value
node-a master node, databases in /databases/master
node-b replica node, databases in /databases/replica
RCM a separate host (may also be one of the database servers)
database employee.fdb

1. Installing nodes and hqbirdrcm

There are three installation methods:

Method When to choose it What you do yourself
Via HQCenter Admin (1.2) the usual path: the HQCenter Admin update server is reachable, auto-updates are needed you run the fbagent installation script, approve the request, describe the cluster
Via RCM, without HQCenter Admin (1.3) there is no HQCenter Admin, but you need certificates, configuration and updates without manual work; up to 3 database servers you install HQbird on each host, log in to RCM, and in Settings → Link hosts specify the host addresses
Manually (1.4) no access to HQCenter Admin, closed network, special requirements you install the files, create certificates, write node.json and rcm.json, register the services

Via HQCenter Admin and via RCM you do not install the nodes and RCM yourself. They are installed by fbagent when you add it to the cluster (chapter 2). Certificates and configuration are also created automatically.

Linux and Windows are described in a single chapter. The Windows differences are in the “On Windows” boxes.

1.1. Requirements

On every database server:

  • Firebird / HQbird 5.0, 4.0, 3.0 or 2.5 (2.5 — Linux only).
  • fbagent alongside every Firebird instance. Without fbagent the node does not work.
  • Time synchronisation (NTP, chrony, w32time) on all servers and on the RCM host. Restart and transfer windows, lag estimation and the “node not responding” check depend on the clocks.
  • Free space for the segment archive and for replica initialisation (chapter 4).
  • Network per the port table (chapter 0).

Linux: root privileges, x86_64, systemd, curl. Supported: Ubuntu 20–26, Debian 11–13, Astra, CentOS 7–10, AlmaLinux, Rocky, Oracle Linux 8–10, RED OS, openSUSE 15.6, Fedora 44+.

Windows: administrator privileges, x64, Windows 10 / Server 2016 or newer, 64-bit Windows PowerShell 5.1. On Windows 10 and Server 2016–2022, 7-Zip is required.

Where RCM will run. One RCM is needed per cluster. It can be installed on any server that can see all nodes, including on one of the database servers.

1.2. Installation via HQCenter Admin

Step 1. Run the fbagent installation script

The script installs Firebird (the HQbird build), fbagent, and registers fbagent on the HQCenter Admin server. Choose the script matching your Firebird version: fb25, fb30, fb40, fb50.

Linux (as root):

Command
sudo sh fbagent-fb50_known.sh --goafts <HQCenter-Admin-server> --enroll --cluster

On Windows (PowerShell as administrator):

Command
powershell -NoProfile -ExecutionPolicy Bypass -File .\fbagent-fb50_known.ps1 -Enroll

There is no -Cluster parameter on Windows: fbagent runs as LocalSystem and creates the cluster folders itself.

Main parameters:

Linux Windows Meaning
--goafts <name> -Goafts HQCenter Admin server to download from; the name is provided by your supplier
--enroll -Enroll register fbagent in HQCenter Admin (certificate request)
--cluster — create the /opt/hqclusternode and /opt/hqbirdrcm folders (only needed for fbagent 2.55.2 and older; harmless for newer versions)
--channel -Channel fbagent update channel, stable by default
--token -Token licence token: HQCenter Admin will approve the request automatically
--port -Port Firebird port
--yes -Yes do not ask questions (needed when there is no terminal)

For the full list of parameters, see the README of the fbagent installation script.

Step 2. Approve the fbagent request

With the --enroll parameter, fbagent sends a certificate request (CSR) to HQCenter Admin and waits up to 30 minutes for approval. You can approve it as follows:

  • automatically — if the installation was run with a licence token (--token) and the licence has a free slot;
  • in the HQCenter Admin panel — the CSR requests section, this host’s request, the approve and sign button;
  • via the HQCenter Admin API — POST /v1/admin/csr-requests/{id}/approve.

If registration did not succeed (for example, the waiting time ran out), fbagent still works, but the script exits with code 2. Run the script again with the same parameters: only the registration will be repeated.

A few minutes after registration, fbagent installs HQ Monitor itself.

Step 3. Describe the cluster

Nodes and RCM appear once you add fbagent to the HQCenter Admin cluster. How to describe the cluster is covered in chapter 2.

After saving the cluster:

  1. Within 10 minutes fbagent receives the cluster from HQCenter Admin (after that it checks for changes every 60 seconds).
  2. The fbagent on the RCM host creates the cluster certificate authority (CA) and signs the certificates of all nodes.
  3. Each fbagent writes node.json or rcm.json for its cluster members.
  4. On the next product check (every 3 minutes) fbagent installs and starts the services.

What is installed and where:

What Linux Windows
Master node /opt/hqclusternode/master/, service hqclusternode-master-p<port> <HQbird>\hqclusternode\master\, service HQClusterNode-master-p<port>
Replica node /opt/hqclusternode/replica/, service hqclusternode-replica-p<port> <HQbird>\hqclusternode\replica\, service HQClusterNode-replica-p<port>
RCM /opt/hqbirdrcm/, service hqbirdrcm <HQbird>\hqbirdrcm\, service hqbirdrcm

<port> is the Firebird port, for example 3050. A single fbagent can host no more than one master node, one replica node, one RCM and one router (hqclusternode router, from HQbird 2027.2.30, chapter 8.7).

On Windows. fbagent versions before 2.83.0 installed nodes in C:\hqclusternode and RCM in C:\hqbirdrcm. If these folders exist, fbagent continues to use them.

Updates

  • fbagent checks for new versions of the node and RCM every 3 minutes; the default channel is stable. The replica node is updated first, then the master node.
  • If, after an update, the service fails to start or reports the wrong version, fbagent rolls back to the previous version and blocks the new one. The list of blocks: <fbagent folder>/update/<product>/blocked-versions.json. To try the version again, delete its line from this file.
  • Product status: fbagent --product-status hqclusternode (or hqbirdrcm).

1.3. Installation via RCM (without HQCenter Admin)

The cluster is managed by RCM. Each host’s fbagent receives the cluster description from RCM, the fbagent on the RCM host signs certificates, and the node and RCM are updated from releases stored by RCM.

Limitations:

  • One cluster per RCM.
  • No more than 3 seats. A seat is a host that runs a master or replica node. A master and replica on the same host count as one seat. A host that runs only RCM does not take a seat. A fourth host produces the error seats_exceeded.
  • RCM does not update fbagent or Firebird. fbagent updates them itself and does not touch replication.conf, node.json, rcm.json or certificates.

There are two methods:

  • Link hosts in RCM — the usual method. Three actions: the installer on each host (RCM on one host, discovery mode on the others), logging in to RCM, the Link hosts form.
  • Manually from the command line — join tokens and fbagent --cluster-join. Needed when port 7450 is closed between hosts or the pairing window has already closed.

Licence. The trial licence includes the cluster feature. After the trial period, fbagent does not install a node or RCM if the licence lacks cluster. It never blocks updates of already installed nodes and RCM. Without HQCenter Admin, a floating licence stops working after online_check_days plus grace days. Therefore install a bound licence on each fbagent:

Command
fbagent --license-request --out license_request.json
# send the file to the vendor and receive license.json
fbagent --license-install license.json

Without a licence, the master stops sending segments (chapter 2.8).

Ports:

Port Where Who connects
7443 RCM host fbagent and nodes of all cluster hosts
7444 RCM host operator’s browser (RCM console, HTTPS)
7450 (on the second and subsequent fbagent host — 7451…7459) host in discovery mode, while the pairing window is open RCM on the RCM host
7051 (second node of the same fbagent — 7061) node hosts other nodes and RCM (1.4, step 7)

Example from scratch. Two Linux servers, the .run installer, linking, registration, publication, initialisation and reinitialisation under load — with screenshots and measurements: the guide hqbird2027-linux-cluster-guide.ru.md and the runbook hqbird2027-linux-cluster-runbook.ru.md (goafts repository, documentation folder).

Step 1. Install HQbird on each host

Install HQbird with the full installer on each database server. Do not register with HQCenter Admin (--enroll).

By default the installer does not configure a cluster. It places the node and RCM programs in the fbagent folder (update/<product>/staged/), but does not install RCM, does not open the pairing window and does not open the cluster ports in the firewall. The cluster is enabled by two parameters:

Command
sudo sh hqbird2027-fb50-linux-<version>.run --rcm on               # RCM host
sudo sh hqbird2027-fb50-linux-<version>.run --cluster-pairing on   # other hosts

--rcm on — on one host, where the cluster’s RCM will run. The installer:

  1. creates this host’s cluster (fbagent --cluster-init), the cluster name is the host name;
  2. installs and starts RCM, the RCM console runs over HTTPS on port 7444;
  3. creates the first RCM administrator rcmadmin with a random password;
  4. opens ports 7443, 7444 and node port 7051 in the firewall.

--cluster-pairing on (discovery mode) — on each other host. The installer opens fbagent’s pairing window for 24 hours: port 7450 and node port 7051. While the window is open, an RCM from the network can invite this host into its cluster. fbagent starts listening on port 7450 within a minute.

Without these parameters the installer asks itself: in the text interface this is the Cluster page, with --no-tui — two questions. The default answer is “no”. --yes without cluster parameters also answers “no” to both questions.

At the end the installer prints the lines RCM: and Pairing:. On the RCM host:

text
RCM:        https://10.0.0.7:7444 (cluster db1)
CA hash:    sha256:7cfc1d4b…032d2582
Login:      rcmadmin
Password:   in /opt/hqbird/fbagent-fb50/clusterca/db1/rcm-initial-password (read it with sudo; RCM asks for a new one at the first login)
Pairing:    not needed: this agent runs the RCM; other agents link through it

On another host:

text
RCM:        off; turn it on in HQbird Monitor or with fbagent --cluster-init <id>
Pairing:    port 7450 open until 2026-11-17T17:49:18Z (fingerprint sha256:0439e44a…91a2f9b9); an RCM may invite this host

There is no password in the output: it is in the file from the Password: line. Write down the fingerprint of each host in discovery mode.

Cluster parameters in the Linux installer:

Parameter What it does
--rcm on|off install RCM and create a cluster on this host; default off
--cluster-pairing on|off discovery mode: open the pairing window for 24 hours; default off
--pairing-port N pairing port of the new fbagent, 7450–7459; default first free
--rcm-password-file F RCM administrator password from the first line of file F, instead of random
--cluster-id ID cluster name ID instead of the host name (with --rcm on)
--cluster-node on|off install the node program; default on
--rcm-allow A,B only these RCMs may invite this host: address, CIDR network or name

The first fbagent on a host gets pairing port 7450 and node ports 7051 and 7061. The second instance on the same host — 7451, 7052 and 7062.

RCM and the pairing window can be enabled later, without the installer: in HQ Monitor, the Clusters page, or with commands as the fbagent user:

Command
cd <fbagent folder>                                    # /opt/hqbird/fbagent-fb50
sudo -u firebird ./fbagent --cluster-init <name>      # RCM host; open ports 7443 and 7444 yourself
sudo -u firebird ./fbagent --cluster-pairing on      # window for 24 hours
sudo -u firebird ./fbagent --cluster-pairing off
sudo -u firebird ./fbagent --cluster-pairing status

Old installers (fbagent before 3.1.0) do not ask about the cluster: they install RCM on every host and open the pairing window everywhere. You need the RCM of one host; the RCMs of other hosts stop themselves after linking.

On Windows. The same two choices exist in the Windows installer: the parameters /RCM and /DISCOVERY.

Step 2. Log in to RCM

Open the address from the RCM: line of the RCM host (https://<host>:7444). Log in with the login rcmadmin and the password from the file in the Password: line. RCM immediately asks for a new password (1.5).

In RCM open Settings → Link hosts.

  1. In the Hosts field enter the addresses or names of the hosts, one per line. The first line is the RCM host, it is already in the cluster.
  2. Click Check hosts. RCM probes the pairing ports 7450–7459 of each host and shows the platform, Firebird version, fingerprint and state. The RCM host does not need a pairing window. Compare the fingerprints with the installer output (the Pairing: line).
  3. Select the Master (default first host) and click Link.
Link hosts: check result
Link hosts: check result

RCM asks for confirmation and shows progress in the Progress section. When all nodes are running, it says “Done” and shows the cluster revision number:

Link hosts: all nodes running
Link hosts: all nodes running

The same from the command line:

Command
cd /opt/hqbirdrcm
export RCM_USER=rcmadmin RCM_PASSWORD='<password>'
./hqbirdrcm cluster link -hosts db1,db2,db3 -master db1

What happens next:

  1. RCM sends an invitation to each free host.
  2. The host’s fbagent joins the cluster and closes the pairing port. If the host has its own RCM without nodes, fbagent stops and disables it, and moves its empty cluster to cluster/left/ in the fbagent folder.
  3. RCM saves the cluster: node <host>-master on the master, <host>-replica on the other hosts, port 7051, master → replica links.
  4. Each host’s fbagent installs and starts the node. RCM waits until all nodes are running and shows Running for each host.

Host states after Check hosts:

State What to do
Ready to link nothing. If below it says “This host’s own RCM will stop after linking” — that is normal
In cluster the host is already in this cluster
Error unreachable port 7450 is closed: the pairing window is off (as by default) or the 24 hours have passed. Right after installation wait a minute and click Check hosts again. Otherwise open the window (--cluster-pairing on) and check the firewall
Error in_other_cluster the host is in another cluster
Error claimed another RCM is linking this host right now
Error seats_exceeded more than 3 seats

Releases. fbagent installs the node only from RCM. RCM takes signed releases from its import/ folder (next to rcm.json) at startup and every 5 minutes. The installer puts the RCM host’s releases there, and fbagents of other hosts download the release for their platform themselves. The list of releases is at the bottom of the Link hosts page. A release can also be imported manually (step 5 below).

Manually from the command line

Order: steps 1–3 on the RCM host, step 4 on each other host, steps 5–6 in RCM.

Step 1. Install HQbird with fbagent

Install HQbird on each database server and on the RCM host with the full installer. On the RCM host you can specify --rcm on right away (then step 3 is not needed). On other hosts do not specify cluster parameters: by default there is no RCM and no pairing window. Do not register with HQCenter Admin (--enroll).

On the RCM host the installer leaves the RCM program in the fbagent folder (update/hqbirdrcm/staged/). Without it step 3 fails with the error no hqbirdrcm to install.

Step 2. Install the bound licence

As at the beginning of the section, “Licence”.

Step 3. Create the cluster on the RCM host

If the installer has already created the cluster (--rcm on: an RCM: line with an address in its output), skip this step.

Run the command as the fbagent user (on Linux — firebird), not as root: otherwise the CA files and rcm.json will be owned by root.

Command
cd <fbagent folder>
sudo -u firebird ./fbagent --cluster-init shop-prod --address 10.0.0.7

On Windows (PowerShell as administrator): .\fbagent.exe --cluster-init shop-prod --address 10.0.0.7

  • shop-prod — cluster name: lowercase Latin letters, digits and -.
  • --address — the address of the RCM host by which other hosts connect to it. Can be repeated several times. Without the parameter, the host’s addresses and name are used.

The command:

  1. creates the cluster CA and the agent CA (in the fbagent folder, clusterca/<name>/);
  2. writes the RCM certificate, rcm.json and cluster-init.json to the RCM folder;
  3. installs RCM and registers the hqbirdrcm service. On Linux the service runs as a separate user hqbirdrcm;
  4. switches fbagent to RCM (cluster.source = rcm) and restarts it;
  5. prints the RCM address and CA fingerprint:
Code
Cluster:    shop-prod
RCM URL:    https://10.0.0.7:7443
CA hash:    sha256:9f86d0...

Write down RCM URL and CA hash: they are needed on the other hosts. Re-running for the same cluster is safe: the CA and keys stay the same.

Log in to RCM immediately and change the rcmadmin password (1.5).

On Linux. The user hqbirdrcm is in the fbagent group. RCM reads rcm.json through this group, but cannot write it. Therefore Settings → RCM does not save changes (error rcm_config_read_only). Edit rcm.json on the host as the fbagent user and restart RCM.

Step 4. Connect fbagents of other hosts

  1. On the RCM host create a join token for each fbagent: in RCM Settings → Cluster, the Create join token button. Or from the command line:

    Command
    cd /opt/hqbirdrcm
    export RCM_USER=rcmadmin RCM_PASSWORD='<password>'
    ./hqbirdrcm cluster join-token -note db2

    The token is valid for 24 hours, for one fbagent only and only once. RCM shows it once, together with a ready-made command.

  2. On the node host run this command as the fbagent user:

    Command
    cd <fbagent folder>
    sudo -u firebird ./fbagent --cluster-join https://10.0.0.7:7443 \
      --ca-hash sha256:9f86d0... --token-file token.txt

    Instead of --token-file you can use --token <token>. On Windows — the same with .\fbagent.exe in PowerShell as administrator.

fbagent verifies the RCM CA by fingerprint. If the fingerprint does not match, it stops and writes nothing. Then fbagent receives its certificate (90 days, renewed automatically) and restarts. A connected fbagent is visible in RCM in the Agents table. While it has no members, nothing is installed on the host.

Step 5. Import releases into RCM

fbagent takes the node and RCM only from RCM. If the required release is not in the Releases in mirror list (Settings → Link hosts), import each product (hqclusternode, hqbirdrcm) for each platform of your hosts (linux-amd64, windows-amd64). The release file and the .sig signature are provided by the vendor.

Command
cd /opt/hqbirdrcm
sudo -u hqbirdrcm ./hqbirdrcm release import -config rcm.json \
  -file hqclusternode-linux-amd64 -product hqclusternode \
  -platform linux-amd64 -version 2027.5.12
sudo -u hqbirdrcm ./hqbirdrcm release list -config rcm.json
  • RCM looks for the signature in <file>.sig, or specify it with -sig. Without a valid vendor signature RCM will not accept the file.
  • A new release replaces the previous one. No RCM restart is needed.
  • fbagent checks for new versions every 3 minutes. The update order and locking are as via HQCenter Admin (1.2, “Updates”).

Step 6. Describe the cluster

Set the members and links in RCM, Settings → Cluster (chapter 2.3). After saving, fbagent installs the nodes in the same way as via HQCenter Admin (1.2, step 3): it writes certificates, node.json and rcm.json, installs and starts the services.

Ports — at the beginning of the section.

1.4. Manual installation

Step 1. Install HQbird with fbagent

Install HQbird on every database server (with the same script, but without --enroll, or with the full HQbird installer). Verify that fbagent is running.

Note down three values from fbagent’s agent_config.json. They are needed for node.json:

fbagent field node.json field Example
local_api.listen firebird.fbagent_url http://127.0.0.1:13050
local_api.token firebird.fbagent_token —
local_api.instance_id firebird.instance_id —

Checking fbagent (the response is the Firebird instance’s JSON):

Command
curl -s -H "Authorization: Bearer <token>" http://127.0.0.1:13050/v1/instances/<instance_id>

If the host has several HQbird instances, leave only your instance’s service in firebird.update.services of the fbagent configuration (see 3.5).

Step 2. Lay out the files

Linux:

Code
/opt/hqclusternode/
  hqclusternode        # program
  node.json            # node configuration (permissions 0600)
  certs/               # ca.crt, <node_id>.crt, <node_id>.key
  work/                # working folder
/opt/hqbirdrcm/
  hqbirdrcm            # program
  rcm.json             # cluster configuration
  certs/               # ca.crt, rcm.crt, rcm.key
  rcm-data/            # state, users, audit log
Command
sudo mkdir -p /opt/hqclusternode/certs /opt/hqclusternode/work
sudo install -m 0755 hqclusternode-linux-amd64 /opt/hqclusternode/hqclusternode
# on the RCM host:
sudo mkdir -p /opt/hqbirdrcm/certs /opt/hqbirdrcm/rcm-data
sudo install -m 0755 hqbirdrcm-linux-amd64 /opt/hqbirdrcm/hqbirdrcm

On Windows. The default folder is C:\hqclusternode with the same layout. A path containing a space is allowed.

If one host has two nodes (master and replica), give each its own folder, for example /opt/hqclusternode/master and /opt/hqclusternode/replica.

Step 3. Create the certificates

Nodes and RCM communicate only over mTLS. You need one CA and a certificate for each participant. Run this once on any machine:

Command
./hqclusternode gencerts -out certs \
  -nodes 'node-a:master,node-b:replica' \
  -hosts 'localhost,127.0.0.1,<ip-node-a>,<ip-node-b>,<ip-rcm>'

The command creates ca.crt and ca.key, node-a.crt/.key, node-b.crt/.key and rcm.crt/.key. In -hosts, list all names and addresses by which the participants will be connected to.

gencerts and the files it created
gencerts and the files it created

ca.key is the CA key. Do not copy it to the hosts; store it separately.

A new node is added later with -add: it signs the pair with the same CA, and the other files are left unchanged:

Command
./hqclusternode gencerts -out certs -add -nodes 'node-c:replica' -hosts '<ip-node-c>'

Certificates are valid for 90 days. gencerts -out certs -list shows the expiry dates, and gencerts -out certs -renew renews them with the same key. Copy the new .crt to its host: the node will pick it up without a restart.

Copy ca.crt and only its own certificate and key to each host.

Step 4. Write node.json

One file per node. Example for the master (node-a):

json
{
  "node_id": "node-a",
  "role": "master",
  "listen_addr": ":7051",
  "rcm": { "url": "https://<ip-rcm>:7443", "ca_cert": "certs/ca.crt",
           "client_cert": "certs/node-a.crt", "client_key": "certs/node-a.key" },
  "firebird": {
    "root": "/opt/firebird",
    "user": "SYSDBA",
    "password": "<SYSDBA password>",
    "port": 3050,
    "replication_conf": "/opt/firebird/replication.conf",
    "replication_log": "/opt/firebird/replication.log",
    "firebird_log": "/opt/firebird/firebird.log",
    "fbagent_url": "http://127.0.0.1:13050",
    "fbagent_token": "<local_api.token>",
    "instance_id": "<local_api.instance_id>",
    "service_name": "<Firebird service name>"
  },
  "windows": { "restart_window": "22:00-05:00", "transfer_window": "23:00-05:00" },
  "peers": [ { "node_id": "node-b", "url": "https://<ip-node-b>:7051", "compress": "zstd" } ]
}

The replica node (node-b) has the same file, but with "role": "replica", its own certificates, and the master node-a in peers.

Main fields:

Field Value
node_id node name; the same in the certificate, rcm.json and the peers of other nodes
role master or replica
listen_addr the node API address; a second node on the same host uses a different port (:7052)
rcm.url the address for receiving RCM statistics; none — no RCM
firebird.* the Firebird and fbagent of this instance; the fbagent values must match exactly
firebird.auto_restart whether the replica node may restart Firebird itself: auto (default), never, always
windows.restart_window the Firebird restart window; for the master the default is 22:00-05:00, for the replica it is always
windows.transfer_window the window in which initialisation runs at full speed
peers for the master — its replicas, for the replica — its master
databases the databases folder; better not to write it by hand but to set it via RCM (chapter 3)

The remaining fields have default values. The node will not start if the fbagent fields are not set.

Protect the file: it contains the password and the token.

Command
chmod 600 /opt/hqclusternode/node.json

On Windows. Write paths in JSON format: "C:\\HQbird\\replication.conf". service_name is the Windows service name for Firebird.

Step 5. Write rcm.json

json
{
  "ingest_addr": "0.0.0.0:7443",
  "operator_addr": "127.0.0.1:7444",
  "users_file": "rcm-data/users.json",
  "data_dir": "./rcm-data",
  "tls": { "ca_cert": "certs/ca.crt", "cert": "certs/rcm.crt", "key": "certs/rcm.key" },
  "nodes": [
    { "node_id": "node-a", "url": "https://<ip-node-a>:7051", "role": "master",  "host": "A" },
    { "node_id": "node-b", "url": "https://<ip-node-b>:7051", "role": "replica", "host": "B" }
  ]
}
  • url — the node address as seen by the RCM host. https only.
  • host — the Firebird instance label. A master node and a replica node with the same label run on the same Firebird (companion, needed to promote a replica to master, chapter 8). One label — no more than one master and one replica.
  • Relative paths are resolved from the rcm.json folder.

Step 6. Register the services

Start order: RCM first, then the replicas, then the master.

RCM (Linux):

Command
cd /opt/hqbirdrcm
sudo ./hqbirdrcm svc install -config /opt/hqbirdrcm/rcm.json
sudo ./hqbirdrcm svc start

Node (Linux):

Command
cd /opt/hqclusternode
sudo ./hqclusternode svc install -config /opt/hqclusternode/node.json -certs /opt/hqclusternode/certs
sudo ./hqclusternode svc start -config /opt/hqclusternode/node.json
systemctl status hqclusternode-master-p3050

svc install creates the service hqclusternode-<role>-p<port> running as the firebird user. If the Firebird folder is owned by root, the command grants the firebird group write access to it (with the sticky bit) and prints every change: the node needs to replace replication.conf.

For svc start, svc stop and svc uninstall, specify the same node.json: the service name is derived from the role and the Firebird port.

On Windows (PowerShell as administrator):

Command
cd C:\hqclusternode
.\hqclusternode.exe svc install -config C:\hqclusternode\node.json -certs C:\hqclusternode\certs

The service HQClusterNode-<role>-p<port> runs as LocalSystem, starts automatically and restarts after a failure. The node service does not depend on the Firebird service. If the node was installed with version 2027.1.2–2027.1.9, remove the dependency once: .\hqclusternode.exe svc fix-deps -config C:\hqclusternode\node.json.

svc install and svc start on Windows
svc install and svc start on Windows

Step 7. Open the ports

On the node host, allow inbound connections on 7051 (the listen_addr port) from the other nodes and from RCM. On the RCM host, allow inbound connections on 7443 from all nodes. Do not expose ports 7444 and 13050 externally.

Example ufw on the node host:

Command
sudo ufw allow from <ip-rcm> to any port 7051 proto tcp
sudo ufw allow from <ip-partner-node> to any port 7051 proto tcp

Manual update

Stop the service, replace the program, start the service:

Command
sudo systemctl stop hqclusternode-master-p3050
sudo install -m 0755 hqclusternode-linux-amd64 /opt/hqclusternode/hqclusternode
sudo systemctl start hqclusternode-master-p3050

The configuration and certificates are not changed by this.

1.5. First login to RCM

The address and password depend on how RCM was installed.

How RCM was installed Address Login and password
By the HQbird installer or fbagent --cluster-init (1.3) https://<RCM host>:7444 from any computer rcmadmin and the password printed by the installer (or your own from --rcm-password-file)
Via HQCenter Admin (1.2) or manually (1.4) http://127.0.0.1:7444, only on the RCM host rcmadmin, password strong password (with a space)

HTTPS (1.3). The RCM certificate is signed by the cluster CA, which the browser does not know, so it warns you. On the login page RCM shows the CA fingerprint. Compare it with the CA hash printed by the installer, and import the CA into the browser or into the system as a trusted root: the Download CA certificate link on the login page (https://<RCM host>:7444/ca.crt, file hqcluster-ca.crt). To make RCM listen only on localhost, set "operator_addr": "127.0.0.1:7444" in rcm.json and restart RCM.

HTTP (1.2, 1.4). From another computer, connect via an SSH tunnel:

Command
ssh -L 7444:127.0.0.1:7444 <rcm-host>

Then open http://127.0.0.1:7444 in your browser.

RCM login page
RCM login page

Generated password. RCM asks you to change it on first login: until you change it, the other pages will not open. Change the strong password immediately (the Password link in the header).

Changing the password on first login
Changing the password on first login

Language. Select Russian in the language list in the page header (it is also available on the login page). If you do not select one, the language is taken from the browser.

Changing passwords and new users

Each user changes their own password: the Password link in the page header. You need the current password and a new one of at least 12 characters. After the change, that user’s other sessions are closed.

If the installer generated a password, it is written to the file clusterca/<name>/rcm-initial-password in the fbagent folder. After the password is changed, fbagent deletes this file itself. Do not delete the initial-admin.json file in the RCM folder: it contains only a hash. If the users file is lost and this file is absent, RCM will create a user with the default password.

New users. There is no users page in the interface. Users are stored in the file rcm-data/users.json. The password is stored as an HA1 hash: MD5("<name>:hqbirdrcm:<password>").

  1. Compute the hash:

    Command
    printf '%s' 'rcmadmin:hqbirdrcm:<new password>' | md5sum
  2. Write it into the user’s ha1 field in users.json. To add a user, add another element to users:

    json
    { "username": "ivanov", "ha1": "<hash>", "algo": "MD5", "created_at": "2026-11-03T00:00:00Z" }
  3. Restart RCM immediately: it reads the file only at startup. When a password is changed in the interface, RCM rewrites the file from memory, and any edits it has not yet read will be lost.

Command
chmod 600 /opt/hqbirdrcm/rcm-data/users.json
sudo systemctl restart hqbirdrcm

Without a role field, a user has all permissions. Since HQbird 2027.2.31 there is a "role": "viewer" role: view only, any modification request gets 403.

1.6. How to verify the installation

  • The node and RCM services are running and are not restarting in a loop: systemctl status hqclusternode-master-p3050, systemctl status hqbirdrcm.

    systemctl status of the node
    systemctl status of the node
systemctl status of RCM
systemctl status of RCM
  • RCM responds (without a password): curl http://127.0.0.1:7444/healthz. When installed via RCM (1.3), the console works over HTTPS: curl -k https://127.0.0.1:7444/healthz.

  • In RCM, the Overview page shows all nodes, with no “not responding” marker.

  • From the command line on the RCM host:

    Command
    export RCM_USER=rcmadmin RCM_PASSWORD='<password>'
    ./hqbirdrcm nodes
    ./hqbirdrcm status
  • The cluster via RCM: Settings → Cluster, the State section — each member has applied the latest revision and reports no error. From the command line: ./hqbirdrcm cluster status.

1.7. Logs

What Where
Node log <node.json folder>/logs/node-YYYYMMDD.jsonl
Node service output (Linux) journalctl -u hqclusternode-<role>-p<port>
RCM service output (Linux) journalctl -u hqbirdrcm
Firebird replication log firebird.replication_log from node.json
Logs from any node without SSH RCM, Logs page

1.8. Common errors

What you see Cause What to do
The node does not start: an error about fbagent_url, fbagent_token or instance_id fbagent is not running or the values do not match agent_config.json check fbagent with the curl command from 1.4, step 1
The installation script exited with code 2 the fbagent request was not approved in time approve the request and run the script again
Nodes do not appear via HQCenter Admin fbagent is not in the cluster, or the cluster has not been obtained yet check the cluster (chapter 2), wait up to 10 minutes
The node in RCM is “not responding” port 7051 is closed, the url in rcm.json is incorrect, the address is not in the certificate check the ports and the certificates’ -hosts
No sign-in to RCM: “Too many failed sign-in attempts” 10 failed sign-ins within 5 minutes wait a few minutes
The new version will not install the version is in blocked-versions.json remove the version line from the file
--cluster-init: no hqbirdrcm to install the installer did not leave the RCM program install HQbird with the full installer (1.3, “Manually”, step 1)
--cluster-join: token_used, token_expired, token_revoked the token has already been used, is older than 24 hours, or has been revoked create a new token (1.3, “Manually”, step 4)
--cluster-join: hostname_conflict the cluster has an fbagent with the same hostname and different addresses give the hosts different names
--cluster-join: this agent is in cluster … from goafts fbagent is already in the HQCenter Admin cluster remove it from the HQCenter Admin cluster
The node will not install via RCM RCM has no release for this platform hqbirdrcm release list, import the release (1.3, “Manually”, step 5)

2. Cluster configuration

In this chapter you define which nodes are in the cluster and which master sends segments to which replica. Then you configure the RCM nodes and users.

Relationships are defined in one of three ways:

Nodes installed Where relationships are defined Section
via HQCenter Admin in HQCenter Admin: the Clusters page of HQ Monitor or the HQCenter Admin API 2.2
via RCM, without HQCenter Admin in RCM: Settings → Cluster, hqbirdrcm cluster or the Clusters page of HQ Monitor 2.3
manually in RCM: Settings → RCM and Settings → Nodes 2.3

On nodes installed via HQCenter Admin or via RCM, change relationships only in the cluster description. fbagent overwrites node.json and rcm.json according to the cluster description. Editing relationships in Settings → RCM or Settings → Nodes of RCM is rejected (409 cluster_managed). Earlier versions of RCM accepted it, but it was lost.

2.1. Roles and schemes

Each node has one role: master or replica. Two nodes — a master and a replica — can run on a single Firebird instance.

Supported schemes:

Code
1 → 1        node-a (master) ──▶ node-b (replica)

1 → N        node-a ──▶ node-b
                    └─▶ node-c

N → 1        node-a ──┐
                      ├─▶ node-b    (one replica for several masters)
             node-x ──┘

Both nodes   host-1: node-1m (master) + node-1r (replica)  ← one Firebird
on a host    host-2: node-2m (master) + node-2r (replica)  ← one Firebird
             node-1m ──▶ node-2r,  node-2m ──▶ node-1r

The link always goes from the master to the replica. RCM links the master’s database to its replica by database GUID, not by node name.

Companion. A master node and a replica node on the same Firebird instance are companions. Promotion of a replica to master (chapter 8) works only for such a pair. In rcm.json the pair is marked by an identical host value.

Where to describe a cluster

  • HQ Monitor, Clusters page (administrators only; configuration delivery must be enabled). The usual way.
  • HQCenter Admin API: POST /v1/admin/clusters and PUT /v1/admin/clusters/{id}. The HQCenter Admin panel has no clusters page.

``

Steps on the Clusters page

  1. Enter a New cluster id (lowercase Latin letters, digits and -) and click Create.
  2. Add members (Members). For each one specify:
    • the host’s fbagent;
    • the role: master, replica or rcm;
    • the host addresses;
    • the port: for a node — 7051 (a second node on the same host — e.g. 7052), for RCM — the statistics intake port 7443.
  3. Add links (Replication links) from the master to the replica.
  4. Click Validate, then Save.

If another administrator changed the cluster at the same time, the page will ask you to reload the cluster and repeat the change.

Rules

  • An fbagent can belong to only one cluster.
  • On a single fbagent — no more than one master, one replica, one RCM and one router. The router (role router, from HQbird 2027.2.30) has no links and does not take up a licence seat (chapter 8.7).
  • A cluster has exactly one RCM. The fbagent on the RCM host runs the cluster’s certificate authority (CA).
  • The member name (member_id) — Latin letters, digits, -. The name, fbagent and role of a member cannot be changed. To move a member to another host, create a new member with a new name. Do not use the old name again.
  • Every replica must have a link to the master.
  • HQCenter Admin builds the node address itself: https://<first address>:<port>.
  • HQbird 2.5/3.0: set the cluster’s replconf_valid_till date (YYYY-MM-DD) — the replconf plugin configuration is valid until that date.

Example via the API

json
{"contract":1,"cluster_id":"shop-prod","name":"Shop","pki":"managed",
 "cert_validity_days":90,"renew_before_days":30,
 "members":[
  {"member_id":"db1-m","agent_id":"<fbagent of host 1>","role":"master","addresses":["db1.lan"],"listen_port":7051},
  {"member_id":"db1-rcm","agent_id":"<fbagent of host 1>","role":"rcm","addresses":["db1.lan"],"listen_port":7443},
  {"member_id":"db2-r","agent_id":"<fbagent of host 2>","role":"replica","addresses":["db2.lan"],"listen_port":7052}],
 "links":[{"from":"db1-m","to":"db2-r"}]}
  • Create: POST /v1/admin/clusters (the document can be sent together with members). Replace the whole document: PUT /v1/admin/clusters/{id}.
  • PUT requires the header If-Match: <revision> (the revision is provided by GET and ETag). Without the header — 428, with an old revision — 409, with an error in the document — 422 with a list of errors.
  • POST /v1/admin/clusters/{id}/validate validates the document without saving.
  • GET /v1/admin/clusters/{id}/status shows the state of members and certificate expiry dates.

What happens after saving

  1. Each member’s fbagent receives the cluster. Within the cluster — a check every 60 seconds (the fbagent on the RCM host — every 30 seconds). Outside a cluster, an fbagent learns about the cluster on its next configuration poll. If that poll does not reach the server, the check runs every 10 minutes.
  2. Each fbagent creates a key and sends a certificate request. The fbagent on the RCM host signs it. RCM signs its own certificate without a request.
  3. fbagent writes to node.json: name, role, address, peers (from the links), the RCM address, certificate paths, Firebird and fbagent data. To rcm.json: the list of nodes (nodes), the list of routers (routers), the statistics intake address and the certificate paths. fbagent does not touch the other fields of the files.
  4. fbagent installs and starts the services (chapter 1).

Changing a cluster:

What changed What fbagent does
only the peers of a running node passes them to the node without a restart
other fields of a node stops the node, writes the file, starts the node — within the node’s windows.transfer_window (not set — immediately)
rcm.json restarts RCM immediately
certificates or CA restarts the service
a member is removed stops and removes the service; cluster-managed.json → cluster-removed.json, the program → *.removed; the configuration, certificates and data remain

After a node is removed, its section # BEGIN managed-by:<name> … # END … remains in the shared replication.conf. Delete it manually and restart Firebird via RCM.

Approving cluster certificates

By default the fbagent on the RCM host signs requests itself (auto). Manual approval can be enabled. Commands on the RCM host:

Command
fbagent --cluster-ca-approval manual     # or auto
fbagent --cluster-status                 # this fbagent's membership in the cluster (JSON)
fbagent --cluster-ca-approve <request_id>

In manual mode, a new member or a new key waits for approval. Renewal of a certificate with a known key is signed automatically.

RCM cluster (without HQCenter Admin)

If the cluster was created by the HQbird installer or by the fbagent --cluster-init command (chapter 1.3), the cluster description is stored by RCM. The easiest way to add hosts is on the Settings → Link hosts page (chapter 1.3, “Link hosts in RCM”): RCM creates the members and links itself. The rules are the same as in HQCenter Admin (2.2, “Rules”), plus:

  • in RCM there is one cluster; the rcm member always exists and cannot be deleted;
  • no more than 3 seats (hosts with master or replica nodes), otherwise the error seats_exceeded occurs;
  • a member can only be added on an fbagent that has connected to RCM (chapter 1.3: “Link hosts” or “Manually”, step 4).

Steps on the Settings → Cluster page:

  1. In the Agents table, click Add member in the fbagent row.
  2. Specify the role (master, replica or router), member_id, addresses and port. The page suggests the name <host>-<role> and the port: 7051 for the first fbagent node, 7061 for the second. A port that is already taken by another member of the host is rejected by RCM. The fbagent addresses are buttons: click to add an address.
  3. In the Links section, click Add link from the master to the replica.
  4. Click Validate: RCM will show errors and warnings.
  5. Click Save. The page will show the list of changes and ask for confirmation. If members are being deleted, the window is red.

Before saving, the changes exist only on the page (the “Not saved” label). If someone saved the cluster at the same time, the page will offer to reload it. If only RCM fields changed (revision, CA, commissions), the page will save your change itself.

Other sections of the page:

Section What is there
header cluster name, revision, cluster CA and agent CA; the Export button for the old CA
Connection tokens create a token for a new fbagent, revoke an unused one
JSON document the whole document in a text field; Validate and Save use it
Keys and CAs commissions: new member key (rekey), new cluster CA (rotate_ca), new agent CA (rotate_agent_ca); unavailable when there are unsaved changes
State the revision each member has applied, certificate expiry, service, last error

From the command line on the RCM host:

Command
export RCM_USER=rcmadmin RCM_PASSWORD='<password>'
./hqbirdrcm cluster show > c.json        # document with revision
# edit c.json
./hqbirdrcm cluster validate -f c.json   # validate without saving anything
./hqbirdrcm cluster apply -f c.json      # save (If-Match = revision from the file)
./hqbirdrcm cluster agents               # connected fbagents and their members
./hqbirdrcm cluster status               # what the fbagents reported

The document format is as in 2.2, “Example via API”. apply with an old revision gets 409: run show again. RCM keeps the last 20 revisions in rcm-data/cluster/history/.

HQ Monitor. The Clusters page also works with RCM. On the RCM host, HQ Monitor accesses RCM through that host’s fbagent, with the fbagent certificate: the RCM user and password are not needed. In the RCM audit log such actions are recorded as agent:<id> (<user>). Another way, for earlier RCM versions: in the HQ Monitor configuration set rcm_admin_url (http://127.0.0.1:7444), rcm_admin_user (the RCM user) and rcm_admin_password_file (a file with its password).

After saving, everything proceeds as via HQCenter Admin (2.2, “What happens after saving”). RCM re-reads nodes in rcm.json itself, without a restart.

Remove an fbagent from the cluster: delete its members and save, then click Revoke in the Agents table (or ./hqbirdrcm cluster revoke -agent <id>). To bring it back, a new token is needed. After revocation the fbagent (since version 3.1.0) leaves the cluster itself: it stops its nodes and deletes the cluster data. The same is done on a node host by the fbagent --cluster-leave command. On the RCM host it is rejected.

Manual installation: add a node to RCM

  1. Settings → RCM, the Connected nodes table.
  2. Click Add node. Fill in node_id, url (https://<address>:7051), role, host (Firebird instance label), server (for on-screen grouping only).
  3. Click Validate in the node row: RCM will check the connection and permissions.
  4. Click Save. RCM will write rcm.json and save the old file as rcm.json.<UTC time>.bak (the last 20).
  5. Restart RCM. Until the restart the old configuration is in effect; the page shows which fields are waiting for a restart.
Settings → RCM
Settings → RCM
Command
sudo systemctl restart hqbirdrcm

You can also do it without the interface: edit rcm.json and restart RCM.

The link must be recorded on both sides:

  1. Settings → Nodes, select the master node.
  2. In the Peers section, click Add peer: the replica’s node_id, its url, compression (zstd).
  3. Click Save. RCM will show the list of changes and ask for confirmation. The peers change takes effect immediately.
  4. Repeat for the replica node: its peer is the master.
The master node's Peers section
The master node's Peers section

If the peer is not recorded on one of the sides, RCM will not offer that replica node when initialising the database.

2.4. Node settings

Settings → Nodes shows the node.json settings that the node allows to be changed from RCM. Only the changed fields are saved, after a window listing the changes.

Settings → Nodes: selecting a node
Settings → Nodes: selecting a node
Node database settings with "after scan" markers
Node database settings with "after scan" markers

Each row is marked with when the change will take effect:

When Which settings
Immediately channel_a_mbps, channel_a_burst_mbps, windows.restart_window, peers; from HQbird 2027.2.24 — data check settings (limits.data_check_*); from HQbird 2027.2.31 — advertise_addr
After node restart the remaining limits, windows.transfer_window; the page shows them on a separate line until the node is restarted
After scan databases.*, exclude_filter; if replication.conf has changed, a Firebird restart is required (chapter 3.5)

Read-only (changed in node.json on the node host): firebird (including the password and auto_restart), rcm, tls, segments, the listen_addr address.

Main settings:

Setting Default Meaning
windows.restart_window master 22:00-05:00, replica — always when the node may restart Firebird; the Firebird master never restarts itself
windows.transfer_window 23:00-05:00 the window in which initialisation runs at the reinit_window_mbps rate
limits.channel_a_mbps 50 segment transfer rate, MB/s (channel_a_burst_mbps — short-term peak, 200)
limits.reinit_window_mbps 100 initialisation rate within the transfer window, MB/s
limits.reinit_day_disk_fraction 0.5 fraction of the measured disk speed for initialisation outside the window
limits.retention_days 2 how many days to keep sent segments in the master’s archive
limits.free_space_floor_gb 1 minimum free space, GB; below this the replica does not accept segments
limits.max_concurrent_reinits 1 how many initialisations the node runs concurrently

For details on speed and archive retention, see chapters 5 and 7.

firebird.auto_restart (in node.json only) determines whether a replica node may restart Firebird itself:

Value Behaviour
auto (default) may, if this Firebird does not serve master databases
never never
always always, even if this Firebird hosts master databases

Via HQCenter Admin. The fbagent node’s windows.transfer_window is also used as the window for restarting the node service itself when the cluster changes. If the window is not set in node.json, fbagent restarts the service immediately.

2.5. Multiple HQbird instances on one host

fbagent restarts the services listed in firebird.update.services of its configuration. An empty list means all HQbird services on the host. Leave only the service of your own instance in the list. The node does not see this setting and writes a warning fbagent_shared to the log if fbagent knows about more than one instance.

2.6. SYSDBA Password

The node connects to Firebird with the user and password from firebird.user and firebird.password in node.json. fbagent stores its own copy of the password. If you have changed the SYSDBA password:

  1. Correct firebird.password in node.json of every node of this instance (master and replica).
  2. Correct the password in the fbagent configuration and reload it (see the fbagent documentation).
  3. Restart the node services.

Otherwise the publication and verification pages in RCM will show a login error (SQLSTATE 28000).

2.7. RCM users and audit

  • Users are in the rcm-data/users.json file (chapter 1.5). Without a role, a user has all rights; the viewer role (from HQbird 2027.2.31) — view only.
  • A session lasts 12 hours after the last action.
  • After 10 failed logins within 5 minutes, login for that user and address is temporarily blocked. A successful login resets the counter.
  • The audit log is the rcm-data/audit.jsonl file: one JSON line for each change, login and logout. There is no audit log in the interface.
  • The Commands page shows the author of each command.

2.8. Licence

Once a minute, the master node reads the fbagent licence of its host. If there is no licence (the trial period has ended, the licence has expired or is invalid), the node stops shipping segments for all databases on the master. Firebird and the databases keep running, but replicas stop receiving changes.

What you see in RCM:

  • the master is red (NEEDS_ATTENTION), with a “no license: shipping stopped” badge;
  • there is no Resume button: only the licence clears the pause;
  • on the replica: “not receiving since …: the master has no license”;
  • in the Nodes table, a “no license” badge; in Settings → Nodes, the fbagent licence block;
  • the alerts license_paused and license_pause_archive_released;
  • initialisation is prohibited (license_paused).
fbagent licence block in Settings → Nodes
fbagent licence block in Settings → Nodes

Install the licence in fbagent. The node clears the pause on its own within a minute. The master retains the archive for replicas for limits.retention_days (2 days by default) from the start of the pause. If you install the licence later, the replicas will have to be initialised again.

Cluster via RCM, without HQCenter Admin. A floating licence without HQCenter Admin stops working after online_check_days plus the grace days. Each fbagent in the cluster needs a bound licence (fbagent --license-request, then fbagent --license-install, chapter 1.3, “Licence”). After the trial period, fbagent will not add a new node or RCM without the cluster feature in the licence; it never prohibits updates. For HQbird 2.5/3.0, the replconf_valid_till date in RCM cannot be later than the end of the licence plus the grace days on the fbagent nodes (error replconf_after_license).

2.9. How to check

  • Overview: all nodes are present and grouped by server.

    Cluster overview
    Cluster overview
  • Settings → Nodes: the master has a replica in Peers, and the replica has the master.

  • Settings → RCM: the Check button in each node’s row passes.

  • Cluster report as a single file:

    Command
    ./hqbirdrcm report -o report.json

2.10 Common errors

What you see Cause What to do
Editing peers in RCM rejected (409 cluster_managed) or lost after node restart the node was installed via HQCenter Admin or via RCM change the links in the cluster description (2.2, 2.3)
seats_exceeded when saving the cluster in RCM nodes on a fourth host no more than 3 hosts with nodes
Settings → RCM: rcm_config_read_only cluster via RCM on Linux: rcm.json is owned by fbagent edit rcm.json as the fbagent user and restart RCM
Revoke agent does not work (agent_has_members) fbagent has members first delete the members and save
Replica node is not offered during initialisation the peer is not recorded on one of the sides add the peer on both sides (2.3)
A change in Settings → RCM does not take effect RCM has not been restarted restart RCM
422 when saving the cluster in HQCenter Admin an error in the document fix it according to the errors list, check via validate
409 when saving the cluster the cluster was changed by someone else re-read the cluster and retry
A new HQCenter Admin cluster member does not receive a certificate manual approval on the RCM host fbagent --cluster-ca-approve <id>
SQLSTATE 28000 error on publication pages the SYSDBA password was changed 2.6

3. Adding a database to the master

In this chapter the master database comes under the control of the node. The node records it in replication.conf and creates folders for the segments. After restarting Firebird the database is ready for replication.

Creating the replica is the next steps (publication, initialisation). They are described in chapters 4–6.

Button names are given according to the Russian RCM interface, with the English ones in brackets.

3.1. How a node discovers databases

The master node looks for databases in a single folder — the database root (databases.root). What comes under management is decided by rules:

Rule Field Default
Database folder root not set
Search subfolders recursive yes
Inclusion mask template *.fdb
Exclusion mask exclude_mask none
File size, MB min_size_mb, max_size_mb no limits (0)
Tables for data comparison sync_tables with HQbird 2027.2.24 — tables from the database publication, if none — RDB$FIELDS; in 2027.5.7 — RDB$FIELDS
Inclusion list include_paths empty (template mode)
Exclusion list exclude_paths empty

The root path is a path on the node host, not on the RCM host. The node applies the inclusion mask first, then the exclusion mask, then the size.

The rules are the same for the whole node. The sync_tables list is also shared: the check (chapter 5.10) looks for each table from the list in every database of the node that does not have its own list.

Tables for comparison, with HQbird 2027.2.24. Each database can have its own list (Synchronization tracking tables). The node stores it in databases.sync_tables_by_db. The check takes the first list that is set:

  1. the database’s own list;
  2. the node’s shared sync_tables list;
  3. tables from the database publication;
  4. RDB$FIELDS.

A shared list saved earlier takes precedence over the publication: a database without its own list uses it. The master’s database page shows which list is currently in effect (3.3).

A table that is absent from both the primary database and the replica gets the status “not in database” (absent): it is not compared, and the check will not show “Not OK” because of it. A table that is missing on only one side is a discrepancy.

RCM and node 2027.5.7. There is a single list for the whole node (sync_tables); without a list the check uses RDB$FIELDS. A missing table is an error, and the database check will show “Not OK”.

All rules are stored in the node’s node.json. The node applies them on every scan.

Template mode and list mode:

  • Template mode (inclusion list empty): the node takes all files matching the mask, except excluded ones. A new file in the folder will be picked up by the node automatically on the next scan.
  • List mode (inclusion list not empty): the node takes only the files from the list. A new file will not come under management automatically — it must be added.

Choose list mode if the folder contains database copies, test databases, or other files that do not need to be replicated.

3.2. Before you begin: check permissions

The user under which Firebird and the node run (firebird on Linux, LocalSystem on Windows) must have read and write permissions on:

  • the database root folder and database files;
  • replication.conf and its folder (the node replaces the file via a temporary file);
  • the segment folders <database>.ReplLog and <database>.LogArch (the node creates them itself).

RCM checks permissions before registration (preflight). If the check fails, the command terminates with the error preflight failed on <node> and a list of paths.

Check on the node host (Linux):

Command
sudo -u firebird /opt/hqclusternode/hqclusternode checkpermissions -config /opt/hqclusternode/node.json

3.3. Database enrollment page

Database enrollment: form and preview
Database enrollment: form and preview
  1. Open the master node enrollment page. The link is available in two places:

    • Databases → Nodes table → actions column → Enroll;
    • Overview → master node card → Enroll databases.

    A replica node does not have this page.

  2. The form shows the node’s current rules (the node sent them itself). If the node is not responding, the preview and enroll buttons are disabled: without the current rules they would reset the include and exclude lists.

  3. Fill in or correct the fields: Path to the folder on this server, recursive, masks, Size filter, Synchronization tracking tables.

    The Synchronization tracking tables field:

    • with HQbird 2027.2.24 and later — the list is per database that this action will enroll. Empty field: the database uses the tables of its own publication. Enrollment does not change the node’s shared list (sync_tables). If it exists, the form says so: a database without its own list uses it rather than the publication. The list of an already managed database is set on its page: Databases → master database → Synchronization tracking tables block. The block shows which list is currently in effect. An empty field removes the database’s own list;
    • in RCM and node 2027.5.7 — a single list for all databases of the node (sync_tables).
  4. Click Preview. The node will show all files matching the include mask. It changes nothing.

  5. Check the list. Each file has a checkbox:

    • checked — the file will be managed;
    • checked, locked, marked already managed — the database is already enrolled;
    • locked with a reason — the file is excluded by mask, size or the exclude list. To take it, change the rule.
  6. Check the required databases and click Enroll selected. RCM will ask for confirmation and show the number of files.

Enrollment confirmation
Enrollment confirmation

What this button does:

  • in template mode — files without a checkbox are added to the exclude list, files with a checkbox are removed from it;
  • in list mode — files with a checkbox are added to the include list.

The button adds databases to those already under management. It does not change locked rows. 7. The Enroll button applies the form without changing the lists. 8. Wait for the completion message and the number of added databases. If there is no response for a long time, check the result on the Commands page.

With HQbird 2027.2.24 and later the message is “Enrolled: N”. If the master’s Firebird needs to be restarted, RCM says which button to press (usually Restart master Firebird in the database’s row) and provides a link to the master databases. RCM 2027.5.7 may say “No new suitable databases”, even though databases have been added: check the Databases page.

Enrollment result
Enrollment result

If no databases were found after enrollment, but the node already had databases, the node refuses (409) and returns the old rules. This way a mistake in the path does not disable replication of all databases.

The same from the command line

Command
hqbirdrcm enroll-databases -node node-a -root /databases/master

Note: the command sends only the fields from its parameters. The remaining rules (masks, sizes, include and exclude lists, sync_tables) will be reset by the node. The command has no parameters for these rules. Before running the command, check the current rules (hqbirdrcm getconfig -node node-a). If they are set, use the enrollment page.

3.4. What happens after registration

  1. The node writes the rules to node.json.
  2. The node scans the folder (scansync).
  3. For each database found, the node writes a block into its own area of replication.conf (between the lines # BEGIN managed-by:node-a role:master and # END …) and creates the folders <database>.ReplLog and <database>.LogArch. The node does not modify text outside its own area.
  4. Before making a change, the node saves a copy of the file: replication_YYYYMMDD_hhmmss_NNN.conf.
  5. If replication.conf has changed, the databases enter the PENDING_RESTART state: Firebird will only read the new file after a restart. In the Nodes table this is shown in the Pending restart column.
The “Pending restart” column in the “Nodes” table
The “Pending restart” column in the “Nodes” table

States of a new database:

State Group in RCM Meaning
UNCONFIGURED Pending database found, configuration not yet written
PENDING_RESTART Pending configuration written, Firebird restart required
CONFIGURED Pending Firebird restarted, database ready for publication and initialisation

3.5. Restarting the master Firebird

The master node never restarts Firebird by itself. Neither does RCM. Restarting the master is the operator’s decision: a restart drops all connections to all databases of that Firebird instance.

Choose a low-load time and click one of the buttons:

  • Restart master Firebird — on the Databases page, in the row of the master of the database that is awaiting a restart;
  • the same button in the Initialize window, chapter 5;
  • Restart Firebird — on the database page.

RCM will ask for confirmation. It shows the node name and a warning: Firebird will restart immediately, without a restart window, and all connections will be dropped.

Confirmation of master Firebird restart
Confirmation of master Firebird restart

The same from the command line

Command
hqbirdrcm restartfirebird -node node-a

This command also restarts immediately, without a restart window.

How the node restarts Firebird:

  1. Checks that fbagent is responding and manages the same instance (identifier, port, service name from node.json). If any value does not match, the node refuses.
  2. Asks fbagent to restart Firebird.
  3. Waits for Firebird to respond on its port.
  4. Moves the databases from PENDING_RESTART to CONFIGURED.

Several HQbird instances on one host. fbagent restarts the services listed in firebird.update.services of its configuration. An empty list means all HQbird services on the host. If there are two HQbird instances on the host, leave only your instance’s service in the list. Otherwise, restarting one node will also stop the other instance.

On Windows. If replication.log has grown to log_portion_mb or more, the restart proceeds as follows: stop Firebird → archive the log → start.

3.6. A database that is already replicated manually

If replication is already configured manually for a database (its own block in replication.conf, publication enabled, segments flowing), it can be brought under management without re-initialising the replica. For this there is a detect_existing mode (master only).

  1. Run a preview:

    Command
    hqbirdrcm enroll-databases -node node-a -root /databases/master -detect-existing -dry-run

    In the response, taken_over lists the databases that will be migrated. Add -verbose to see the skipped databases with the reason.

  2. If the list is correct, run the command without -dry-run.

The node migrates a block only if:

  • the segment folders are named exactly <database>.ReplLog and <database>.LogArch;
  • segments already exist in the archive;
  • publication is enabled in the database.

The node does not migrate a block with different paths and reports it as “found but unsupported”.

After migration the databases get PENDING_RESTART, even though the replication settings are not changed: only the position of the block in the file changes. Restart Firebird in a maintenance window.

3.7. How to verify

  • Databases: the database is visible in the master node’s row, state CONFIGURED (after restart).

  • Next to the database there may be a publication OFF icon: publication is not yet enabled. This is normal, publication is the next step (chapter 6).

    New database in the master row
    New database in the master row
  • On the node host: the database block is present in replication.conf inside the managed-by:node-a area, and next to the database there are the .ReplLog and .LogArch folders.

3.8. Remove a database from management

The usual way is the button. On the Databases page, tick the databases, select the bulk action Remove from management and click Apply to selected. RCM:

  1. adds the files to the node’s exclusion list;
  2. scans the folder once: the database blocks are removed from replication.conf;
  3. deletes the database records in the node.

The database files and segment folders remain on disk. Firebird is not restarted. Firebird continues to write the database segments until its next restart; RCM will say if a restart is needed.

RCM will refuse (has_replicas) for a master database that has a live replica: otherwise the replica would silently stop. First turn the replica into an ordinary database or remove it (chapter 8).

Do not delete the database block from replication.conf manually: on the next scan the node will write it again.

If the database file has disappeared or has been excluded by a rule. Then the database gets the state ORPHANED (the Orphaned group). The node keeps its record and the segment archive: a missing database is more often a mistake than a decision. If the file comes back, the database works again and the alert is cleared. To delete such a database permanently, perform two steps in this order:

Command
# 1. delete the segment archive (optional)
hqbirdrcm cleanuparchive -node node-a -db <db_id> -orphaned
# 2. delete the database record
hqbirdrcm forgetdatabase -node node-a -db <db_id>

After forgetdatabase the cleanuparchive command will no longer find the database. Both commands work only for a database in the ORPHANED state and do not touch the database file.

Caution. After promoting a replica to master (chapter 8), do not run cleanuparchive -orphaned for the old record: the new master uses the same .LogArch folder, and the command will delete its working archive.

3.9. Changing the database root

Registration with a different path is permitted. The node:

  • adds databases from the new folder;
  • moves databases from the old folder that are not present in the new one to ORPHANED (notification orphaned_database) and keeps their archives.

If you chose the wrong folder, register the correct folder again. A database with the same full path will continue to operate under the same db_id. A database at a different path is a new database (new db_id); its replica must be initialised again.

3.10. Common errors

What you see Cause What to do
preflight failed on node-a no permissions on the folders fix the permissions, retry (3.2)
409 scan_would_empty_region the new folder contains no databases, but the node had some check the path and the mask
Preview is empty the mask does not match or the folder is wrong check the path, the mask, “recursive”
Buttons are disabled, “Failed to read node configuration” the node is not responding check the node service and the network (chapter 1)
The database stays in PENDING_RESTART for a long time the master does not restart Firebird by itself restart Firebird (3.5)
Restart rejected because of fbagent fbagent is not running or its data does not match node.json chapter 1, checking fbagent
The new file did not appear the node is in list mode add the file via “Register selected”

4. How to make a database a replica: steps and disk location

This chapter covers the whole journey of a database from primary to a working replica, plus the disk space calculation. The initialisation itself is described in detail in chapter 5.

4.1. The main rule

A database appears on a replica only through initialisation from the master (reinit): the master node itself transfers a copy via nbackup.

Do not make a copy manually:

  • gbak creates a new database GUID and resets the segment number. The master — replica link breaks.
  • A file copy of a running database is not consistent, and RCM will see two databases with the same GUID (duplicate_guid) and will prohibit initialisation.

An already running replica configured manually can be taken under management without a new copy — see 4.6.

4.2. Stages

No. Stage Where Master database state after the stage
1 Registering the database on the master chapter 3.3 PENDING_RESTART
2 Restarting the master’s Firebird chapter 3.5 CONFIGURED
3 Synchronising the publication chapter 6 CONFIGURED, publication ON
4 Preparing the replica: folder, permissions, Firebird version 4.3 —
5 Checking free space on the master and on the replica 4.4 —
6 Initialisation chapter 5 SEEDING → IN_SYNC
7 Restarting the replica’s Firebird (if needed) chapter 5.6 the replica starts applying segments
8 Verifying the data chapter 5.10 —

Stage 3 is mandatory for Firebird 4/5. Without a publication, the master’s Firebird does not write table changes to segments, and the replica silently falls behind the master. On the Databases page such a database is marked with the publication OFF icon. Click Synchronise publication (Sync publication) before initialisation.

As of HQbird 2027.2.24, the Initialise window warns about this and offers a Synchronise publication and initialise button, while the master node refuses initialisation with the publication_off error (chapter 5.1). RCM and node 2027.5.7 do not check the publication.

HQbird 2.5/3.0. There are no publications, so stage 3 is skipped. The set of tables is defined by the replconf plugin filters (chapter 6).

Database states:

State Group in RCM Meaning
CONFIGURED Pending Firebird has read the settings, there is no replica yet
PUBLISHING Pending publication synchronisation is in progress
SEEDING Pending initialisation is in progress
IN_SYNC Ok the replica has received all segments
LAGGING Pending the replica is lagging, segments are queued
NEEDS_REINIT Error the replica must be initialised again
FAILED Error the last initialisation failed

4.3. Preparing the replica

Replica database folder

The replica node places each database in its own databases.root folder at the path:

Code
<databases.root>/<master node name>/<database path relative to the master root>

Example: master node-a with root /databases/master and database /databases/master/sales/employee.fdb. On replica node-b with root /databases/replica, the database will be here:

Code
/databases/replica/node-a/sales/employee.fdb

This way, databases with the same names from different masters do not clash.

Nobody creates the root folder automatically — neither the installer, nor HQCenter Admin, nor the node at startup. Create it in one of the following ways:

  • in the Initialize window: the Replica folder field, the Create folder button. The button is visible only when the folder does not exist. With HQbird 2027.2.24, if the replica node does not yet have a databases.root, the field already contains /databases/replica/<replica node id>. For a replica on Windows the field is empty: enter a folder with a drive letter. Each replica node on the same host needs its own folder. The folder is created by the replica node: owner — firebird (or the node service user), permissions 2770. If the node cannot create it, the window will show the command for the replica host. If you specify a different folder, RCM will write it to the replica node’s databases.root before initialisation;

  • manually on the replica host (Linux):

    Command
    sudo mkdir -p /databases/replica
    sudo chown firebird:firebird /databases/replica
    sudo chmod 2770 /databases/replica

    The setgid bit (2770) is needed so that all files inside get the Firebird group.

You can change the replica root only while it has no databases: otherwise the old databases will end up outside the scan. To put a single database in a different location, use Move on the database page.

Permissions

The replica node and Firebird must be able to write to the database root. The node checks that the placed files will get the folder’s group. If it cannot — it refuses with ownership_unfixable and text on how to fix it (usually the node service runs with Group=root).

With HQbird 2027.2.24, the Initialize window also shows a folder that exists but that the node service cannot write to: the node’s text with the fix command and a Check again button. RCM 2027.5.7 in this case says “Folder exists”: check the permissions yourself (hqbirdrcm checkpermissions -node <replica> -root <folder>).

Firebird version

The major Firebird version on the replica must match the master (5.0 → 5.0, 4.0 → 4.0). Otherwise the replica will refuse before any data transfer: incompatible_engine.

Connectivity

The replica node must be a peer of the master, and the master must be a peer of the replica (chapter 2). Otherwise RCM will not offer this node in the Initialize window.

4.4. How much space is needed

In brief

Where What How much
Master, database volume free before initialisation ≥ 105 % of the database size
Replica, database folder volume free before initialisation ≥ 105 % of the database size, with HQbird 2027.2.24 — plus free_space_floor_gb (plus the old file, if the replica already existed; with the Remove the old replica checkbox — without it)
Master, <database>.ReplLog Firebird working segments up to 64 segments × segment size (16 MB by default) ≈ 1 GB per database
Master, <database>.LogArch archive of segments for replicas changes over retention_days (default 2 days) + whatever the replicas have not yet confirmed
Replica, <database>.Incoming queue of segments to apply up to 256 segments × 16 MB ≈ 4 GB per database in the worst case
Replica, whole volume headroom at least free_space_floor_gb (default 1 GB)

Checking space before initialisation

Initialisation checks space twice and only before locking the database:

  1. RCM — from the latest node statistics, before sending the command. If space is clearly insufficient, RCM immediately returns error 507.
  2. Master node — a live measurement on its own volume, then a request to the replica so that it measures its own. If there is not enough on at least one side, initialisation ends at the SPACE CHECK step with the error insufficient disk space on master or … on replica.

The requirement on both sides is 105 % of the master database size.

With HQbird 2027.2.24 the replica node calculates as follows:

  • it needs 105 % of the database size plus the replica’s free_space_floor_gb (7.3): after initialisation a reserve for segments remains on the volume;
  • leftovers from previous initialisations (<database>.hqcluster-part, <database>.new) count as free space: the node deletes them before the new copy;
  • the error text names all the numbers. It also names the files <database>.pre-reinit-*.fdb and <database>.failed-seed-*.fdb with their size. The node never deletes them: delete them yourself if they are not needed.

force (forced) skips only the check in RCM. The node check cannot be bypassed: it protects the node from running out of disk in the middle of copying. If space is insufficient only because of the old replica copy, RCM 2027.2.24 in error 507 names the Remove the old replica checkbox rather than force. For this, both nodes must be HQbird 2027.2.24 or newer. Otherwise the error names the node that needs updating, and force.

Why so much

Master. During copying the database is locked by nbackup -L. All changes are written to the <database>.delta file next to the database. The size of .delta depends on the write load during copying. Under heavy load and slow transfer it can grow considerably — watch the space during the first initialisation.

Replica. The copy is written straight to the <database>.hqcluster-part file next to the database location. The node reserves the full size in advance. The file is then checked, repaired (nbackup -FIXUP -SEQUENCE) and renamed into place. A second copy is not needed — the database size is needed once, on the database volume.

If the replica already existed (repeated initialisation), the old file sits on disk until the last step. This means the old file and the new copy are needed on the volume at the same time. Free space is calculated without the old file — therefore the requirement is the same: 105 % of the database size free.

Remove the old replica (with HQbird 2027.2.24). If there is no space on the replica volume for the new copy next to the old one, tick this checkbox in the initialisation window (chapter 5.1). The replica node then deletes the old copy and its incoming segments (.Incoming) before transfer, and the volume has enough for 105 % of the database size plus free_space_floor_gb. The node deletes the copy only if all conditions are true:

  • without deletion there is not enough space, and with deletion there is enough;
  • it is a copy installed by this node’s initialisation: the header is readable, the mode is read-only replica;
  • the database has not been made a normal one (no <database>.hqcluster-normal.json file, chapter 8);
  • overwrite_non_replica is not set (5.9).

Otherwise the node deletes nothing and refuses with the reason in the text. To delete, the node stops the replica’s Firebird for the duration of the deletion (5.6).

The checkbox is off by default. Risk: while the transfer is in progress, there is no database copy on the replica. If the database has only one replica, the master is unprotected the whole time. While there is no new copy, the database on the replica is in the NEEDS_REINIT state (not FAILED and not ORPHANED), and the replica issues the alert replica_removed_for_reinit (warn). Once the new copy is in place, the alert is cleared. If initialisation fails after the deletion, the pair remains NEEDS_REINIT: start the initialisation again (or automatic reinitialisation will start it, 5.8). A new initialisation transfers the database from the beginning.

Master archive (.LogArch). The node keeps sent segments for retention_days days so that the replica can request them again. Segments that the replica has not yet applied are kept longer. If space is low, the node cleans the archive in stages:

  1. segments applied by all replicas older than retention_days;
  2. applied segments that are newer (when space is short);
  3. unapplied segments — only as a last resort. Then the replica loses the segments it needs and gets NEEDS_REINIT.

If a replica is unavailable for a long time, the archive grows. Its volume = the database changes over the downtime period.

Calculation example

Database 100 GB. Load — 5 GB of segments per day. Initialisation runs at night, copying takes 2 hours, during which about 2 GB ends up in .delta.

Where Calculation Total free
Master, database volume 105 GB for checks (includes .delta growth) + .ReplLog 1 GB + .LogArch 2 days × 5 GB = 10 GB ≥ 116 GB
Replica, new database 105 GB + .Incoming up to 4 GB + 1 GB reserve ≥ 110 GB
Replica, repeated initialisation old file 100 GB already on disk + 110 GB free volume ≥ 210 GB
Replica, repeated initialisation with the Remove the old replica checkbox (with HQbird 2027.2.24) old file deleted before transfer: 105 GB + .Incoming up to 4 GB + 1 GB reserve volume ≥ 110 GB

Add a reserve for database growth and for replica downtime (the master archive grows by 5 GB per day of downtime).

Segments on a separate volume

The .ReplLog, .LogArch and .Incoming folders are by default located next to the database. They can be moved to another volume: the segments.storage parameter in node.json (with mirror_hierarchy: true the node replicates the structure of the database folders). Then the archive does not take space away from the databases. The 105 % check still applies to the database volume.

Where to look at space

  • Databases → Nodes and Overview: the free space icon for the databases.root volume. Yellow — less than 20 %, red — less than 10 %.

    Databases page: the Disk column in the Nodes table
    Databases page: the Disk column in the Nodes table
  • Cluster report: hqbirdrcm report.

4.5. Multiple replicas of one database

The second and subsequent replicas are created in the same way: Initialise in the master row, selecting a free replica node.

  • While a new replica is being initialised, the master does not transmit this database to the already running replicas either. They will catch up with the master after initialisation finishes. The confirmation window says so.
  • One database — one initialisation at a time, even for different replicas.
  • Space on the master is checked separately for each initialisation.
Master database with two replicas
Master database with two replicas

4.6. Manually configured replica

If a replica is already running from a hand-written replication.conf, you can take it under management and not copy the database again:

  1. Master: register the databases with detect_existing (chapter 3.6).
  2. Replica: Databases → replica node → Take over, or hqbirdrcm takeover-replica. RCM will find the replica blocks, match the files to the master databases by GUID and show for each one: in_sync, resumable or needs_reinit.
  3. Confirm the rows. The node will move the blocks into its own replication.conf area.
  4. Restart the replica’s Firebird so that it reads the moved blocks.
  5. Initialise only the needs_reinit rows: the master archive no longer contains the segments required by these replicas.

4.7. Common errors

What you see Cause What to do
insufficient disk space on master / on replica not enough space (4.4). On re-initialisation the old copy takes up space until the end (4.4) free up space, retry. The second way out on the replica (from HQbird 2027.2.24) is the Remove the old replica checkbox (4.4, 5.1)
507 from RCM immediately according to the space statistics there is clearly not enough same. If there is space without the old copy, RCM 2027.2.24 names the checkbox Remove the old replica
incompatible_engine different major Firebird versions upgrade the replica’s Firebird
Folder missing on the replica node root not created Create folder in the initialisation window (4.3)
Folder exists, but the node service cannot write to it (from HQbird 2027.2.24) no write permissions run the command from the window, click Check again (4.3)
publication_off (from HQbird 2027.2.24) the master’s publication is disabled or has no tables Synchronise publication and initialise (5.1)
ownership_unfixable the node service has a foreign group fix the service group as described in the error text
duplicate_guid two databases with the same GUID (a file copy) remove the copy or give it a new GUID (nbackup -F)
Replica node not in the Initialise list no connection in the peers chapter 2
Replica IN_SYNC, but the data diverges the publication was disabled synchronise the publication, then repeat the initialisation

5. Initialization (reinit)

Initialization is the transfer of a copy of the master database to the replica. The first transfer is called initialization (the Initialize button), a repeat transfer is reinitialization (the Reinitialize button). The mechanism is the same.

Before starting, complete steps 1–5 from chapter 4: the database is registered, the master’s Firebird has been restarted, publication is enabled, the replica folder exists, and there is enough space. How much space is needed on the master and on the replica is described in chapter 4.4.

5.1. How to start

Publication first. Registering a database does not include publication. Without publication, the replica receives no changes. Before initialisation, run Synchronise publication (chapter 6) or hqbirdrcm setpublication -node <master>.

  • With HQbird 2027.2.24, the initialisation window checks the publication. If it is off or has no tables with a key, the window shows a warning and a Synchronise publication and initialise button. RCM synchronises the publication, waits for the command to finish and starts initialisation with the selection from the window.
  • A 2027.2.24 master node itself refuses such initialisation (and the first one too) with the error publication_off and issues the notification master_publication_off (chapter 6.1). After the refusal, the database cell again has the Synchronise publication and initialise button.
  • If the operator turned the publication off (6.4) or removed all tables with a key from it (6.3), synchronisation preserves that. RCM does not start initialisation and provides a link to the Publication tables page.
  • RCM and node 2027.5.7 do not check the publication: initialisation succeeds, the database shows “OK” in RCM, but the replica receives no changes. If the publication was turned on after initialisation — reinitialise the database.
The "Reinitialise database?" window
The "Reinitialise database?" window
  1. Open Databases.
  2. In the master database row, in the replica node column, click Initialise (Initialize). The replica node is set by the column: it is not selected in the window. For an existing pair — Reinit in the expanded master card.
  3. In the window (with HQbird 2027.2.24 the title for a new pair is “Initialise database?”; in RCM 2027.5.7 — “Reinitialise database?” and also on first initialisation):
    • check the Replica folder. The window shows whether the folder exists. If it does not, click Create folder: the replica node will create it (4.3). If the node cannot create it, the window shows the command for the replica host. With HQbird 2027.2.24 the window also shows a folder that the node service cannot write to: the fix command and a Check again button;
    • select the Reinit method: Smooth (recommended) or Standard (5.4). With HQbird 2027.2.24, if the master’s archive has no segments yet (usually before the first initialisation), the window shows a hint: on a busy database Standard is faster;
    • if the master database is still waiting for a Firebird restart, the window will say so and show a Restart master Firebird button. Without the restart the database is not replicated;
    • Remove the old replica (with HQbird 2027.2.24, off by default) is needed only for reinitialisation when the replica volume has no space for a new copy next to the old one (4.4). The window shows the numbers in advance: how much is needed, how much is free, how much the old copy occupies. If there is space without the old copy, the window asks you to tick the checkbox. The checkbox is inactive if the master or replica node is an old version: the window states which node to update. Until the new copy is ready, the replica has no copy of this database (4.4).
  4. Confirm. RCM will state that the database will be locked on the master and copied to the replica.

The progress of initialisation is visible in the expanded master card on the Databases page: steps and percentages. With HQbird 2027.2.24 the progress is also visible after refreshing the page: on first initialisation — in the replica node cell, and also in the Reinitialisation progress row on the master database page. The Initialise button is inactive while the master node is copying the database. If RCM is not tracking this initialisation (for example, it was started from the node’s command line), the steps are taken from the node state, without percentages. In RCM 2027.5.7, after refreshing the page the progress of the first initialisation is not visible: wait until the master database leaves SEEDING.

Initialisation steps in the master row
Initialisation steps in the master row

Every RCM command (initialisation, Firebird restart, publication synchronisation) is recorded on the Commands page with its result.

The "Commands" page
The "Commands" page

The same from the command line

Command
# one database
hqbirdrcm reinit -node node-a -db <db_id> -to node-b
# all master databases
hqbirdrcm reinit -node node-a -all -to node-b

Parameters: -mode smooth|standard (default smooth), -hold=false (do not hold the archive during long transactions), -allow-restart (5.6), -force (5.9).

-remove-old-replica (with HQbird 2027.2.24) — the same as the Delete old replica checkbox. For a single database only: -db and -to are required; with -all the command refuses. If the master or replica node is an old version, RCM refuses with node_too_old and sends nothing.

Command
hqbirdrcm reinit -node node-a -db <db_id> -to node-b -remove-old-replica

-all tries all databases and collects errors rather than stopping at the first one.

5.2. Steps

No. Step (in RCM) What happens
1 SPACE CHECK checks space on the master and on the replica (chapter 4.4); the replica checks in advance that it can accept the file
2 LOCKING nbackup -L: the master’s database file is frozen, changes go to <database>.delta
3 COPYING TO NODE the file is transferred in parts: read → checksum → compression → send
4 UNLOCKING nbackup -N immediately after the transfer: changes from .delta are returned to the database
5 SEEDING on the replica: verification of the file checksum, nbackup -FIXUP -SEQUENCE, installation of the file, gfix -replica, writing the block to replication.conf
6 RESTARTING REPLICA restart of the replica’s Firebird, if it is needed (5.6)

After step 6 the database gets the IN_SYNC state. The master continues sending segments from the point where the copy ends.

What is important to know:

  • The master runs all the time. Users write to the database, changes go to .delta. The lock is released immediately after the transfer.
  • Transactions that coincided with the lock are not lost. The node finds the transactions open at the moment of nbackup -L, and the replica replays them from the segments.
  • The transfer continues after an interruption. If the connection or the replica node restarted, the master will continue from the last received part. An unfinished transfer is kept on the replica for 24 hours (reinit_partial_keep_hours).
  • On an error the node releases the lock, deletes the unfinished file on the replica and sets the database to FAILED. The next attempt starts from a known state.
  • With the “Delete old replica” checkbox (since HQbird 2027.2.24) the replica deletes the old copy after the LOCKING step, before the transfer. The COPYING TO NODE step at this time shows “removing the old replica copy to make space”. An error after the deletion gives the pair NEEDS_REINIT rather than FAILED (chapter 4.4).
  • nbackup only. gbak is not used: it changes the database GUID.

5.3. Impact on operation

What Impact
Users of the master keep working; during copying .delta grows, after -N the changes are transferred into the database
Other replicas of this database do not receive segments of this database while initialisation is in progress; they catch up afterwards
Other databases of the master not affected
Firebird of the replica, new database not stopped; restart per 5.6
Firebird of the replica, repeated initialisation is stopped while the file is being replaced: all databases of this Firebird are unavailable for seconds
Firebird of the replica, the node deletes the old copy (“Delete old replica” checkbox, since HQbird 2027.2.24) is stopped earlier, while the deletion is in progress, before the transfer. The node installs the new copy without a second stop if the database block in replication.conf has not changed. If there is enough space and no deletion is needed, the node deletes nothing, and everything proceeds as with a normal repeated initialisation
Network and disk load according to the speed limits (5.5)

On Windows. A file opened by Firebird cannot be replaced. During repeated initialisation the node stops the replica’s Firebird via fbagent, installs the file and starts Firebird again. On Linux Firebird is also stopped before replacement: otherwise it would continue working with the old deleted file.

5.4. Smooth and standard methods

Smooth (default). Before locking, the node looks for long transactions: user write transactions that started before the oldest segment in the master’s archive. After initialisation the replica will not be able to replay them and will stop at their first change (“Transaction … is not found”). Transactions without changes are not counted.

If such transactions exist, initialisation does not start: RCM displays Smooth reinit is waiting for long transactions and provides an Open list link.

Long transactions page: no long transactions
Long transactions page: no long transactions

On the Long transactions page:

  • summary: how many transactions and connections, the oldest segment;
  • Top 5 connections with the oldest transactions: user, client, start, age, changes, current query;
  • Hold segments / Release hold — while the hold is enabled, the master’s archive is not purged and history is not truncated;
  • Disconnect selected and retry reinit — disconnects the connections (their transactions are rolled back) and immediately starts a smooth reinit;
  • Retry reinit, Check again.

The hold is enabled automatically when a reinit is refused because of long transactions. It is released after a successful initialisation, manually, or when the archive grows to reinit_hold_max_segments (1000 segments by default, notification retention_hold_released).

Standard locks the database immediately, without checking. Use it only if you know there are no long write transactions.

First initialisation under load. While the master’s archive is empty, there is no segment to compare against.

With an HQbird 2027.2.24 master node: if the node enabled publication itself and the archive has not changed since then, history begins at the moment publication was enabled. A transaction started later enters the journal from its first change and does not interfere with the smooth method. A transaction started earlier did not enter the journal: the reinit waits for it before locking (6 attempts every 10 s), and the locking step shows its number, connection and user. If it has not finished, the reinit ends with the error reinit_pre_publication_transactions: finish it and retry. The Long transactions page shows such transactions in the Transactions started before publication was enabled table.

If publication was enabled manually, not via the node, or the archive has already changed, the start of history is unknown. In that case the smooth method refuses if any write transaction is open, and the message says that the archive does not yet contain a segment. In this case RCM 2027.2.24 does not open the list but writes what to do in the database’s cell. Choose Standard or retry once Firebird archives the first segment.

With a 2027.5.7 master node, the smooth method refuses immediately on the first initialisation of a database that is currently being written to: long_transactions … segment 0 (0001-01-01T00:00:00Z). For the first initialisation of such a database, choose Standard.

For subsequent reinitialisations the smooth method works as usual (the archive already exists).

The same from the command line

Command
hqbirdrcm longtx -node node-a -db <db_id>                       # list
hqbirdrcm longtx -node node-a -db <db_id> -disconnect 12,34 -retry-to node-b
hqbirdrcm longtx -node node-a -db <db_id> -hold false           # release hold

Transactions older than publication. Firebird does not write to segments a transaction that started before publication was enabled. If the node enabled publication, initialisation waits (6 attempts every 10 seconds) until such transactions finish; otherwise it ends with the error reinit_pre_publication_transactions listing them. After synchronising a publication that added tables, reconnect the applications, then start the initialisation.

5.5. Speed

When Speed Setting
Within the transfer window windows.transfer_window (default 23:00-05:00) constant limits.reinit_window_mbps, default 100 MB/s
Outside the window fraction of the measured disk speed limits.reinit_day_disk_fraction, default 0.5

The node recalculates the mode every 30 seconds: an initialisation started during the day will speed up on its own at night.

The node measures the disk speed with a test. To repeat the test on the master host:

Command
hqclusternode benchmark -config /opt/hqclusternode/master/node.json

Initialisation time ≈ database size ÷ speed. Example: 100 GB at 100 MB/s — about 17 minutes of pure transfer; in practice it is limited by the network or the disk.

5.6. Restarting the Firebird replica

A new database on the replica. The file is placed without stopping Firebird. Firebird will read the new replication.conf block only after a restart. Until then the replica applies nothing. The RESTARTING REPLICA step shows the wait:

Label What it means
waiting for a Firebird restart on the replica, it is scheduled for … the node will restart Firebird itself after the last initialisation (reinit_restart_delay_sec), once the other initialisations have finished
… the restart window is closed, the next one will open at … waiting for the replica’s restart_window
… the last attempt failed, the next one at … will retry later
the replica cannot restart Firebird itself: restart it manually auto_restart: never, or this Firebird also serves the master’s databases

The node restarts Firebird once after a series of initialisations, not after each database.

To restart manually, click Restart replica Firebird. The wait ends after any successful Firebird restart or when the replica has applied a segment newer than the copy.

Re-initialisation on a Firebird that also serves the master’s databases. The node does not stop such a Firebird itself and refuses with restart_not_allowed. Start the initialisation again and tick Allow to stop Firebird for this reinit (-allow-restart on the command line) — at a time when this Firebird’s connections can be dropped, including to its master databases. Run such initialisations one at a time per host.

Deleting the old copy (the Remove the old replica checkbox, with HQbird 2027.2.24) also stops the replica’s Firebird, by the same rules. If the node cannot stop this Firebird itself, it deletes nothing and refuses with restart_not_allowed: also tick Allow to stop Firebird for this reinit. If at that time another initialisation is finishing on the replica or Firebird is restarting, the node deletes nothing and refuses with engine_busy: the master releases the lock, retry later.

5.7. Queue and concurrent initializations

  • One database — one initialization at a time (even for different replicas). The Initialize button is disabled while an initialization of this database is in progress.
  • The master node enforces limits.max_concurrent_reinits initializations of different databases simultaneously (default 1). The rest wait in the queue; the response contains the queue position.
  • Do not run your own nbackup -L (or ALTER DATABASE BEGIN BACKUP) on a database during its initialization, and do not start an initialization during such a backup: a database can have only one lock. Initialization checks this and refuses with nbackup backup lock already active.

The queue and current transfers are visible on the Transfer page.

Transfer / queues page
Transfer / queues page
- If a `.delta` file remains after a failure, make sure no backup is running and run `nbackup -N `. Do not delete the `.delta` manually.

5.8. Automatic reinitialization

RCM can reinitialize a database in the NEEDS_REINIT state by itself, only within a configured window. Disabled by default.

When a database enters NEEDS_REINIT:

Cause Where it is visible
recurring unrecoverable apply error (PK, UNIQUE, FK violation) — 3 times in a row replica, replication_unrecoverable_error alert
master database GUID has changed (restore, file replacement) replica, replication_guid_mismatch alert
the master’s archive no longer contains the segments the replica needs master
the replica has deleted the old copy for initialization and there is no new copy yet (since HQbird 2027.2.24) replica, replica_removed_for_reinit alert; master

To enable — Settings → RCM (or rcm.json, the auto_reinit section):

json
"auto_reinit": {
  "enabled": true,
  "default_window": "22:30-04:00",
  "max_concurrent": 1,
  "nodes": {
    "node-a": { "window": "23:00-05:00",
                "databases": { "sales/employee.fdb": "01:00-03:00" } }
  }
}
  • Window: per database → per node → global. "always" — at any time.
  • Windows are calculated by the RCM host’s clock.
  • max_concurrent — how many initializations (manual and automatic) may be running in the cluster for RCM to start another automatic one.
  • Automatic initialization goes through the same space and GUID checks as manual initialization. The command author is system:autoreinit.
  • remove_old_replica_when_short (since HQbird 2027.2.24, the Delete old replica when short of space row) — disabled by default. If enabled, automatic initialization sends the Delete old replica flag (4.4). RCM sends it only if the replica is itself in NEEDS_REINIT or the master’s NEEDS_REINIT cause names it. For a replica that is IN_SYNC or LAGGING the flag is never sent. If the node is an older version, RCM starts the same initialization without the flag.
  • After changing, restart RCM.

Snooze in the master’s card disables automatic initialization of a single database for 1 hour, 4 hours, 24 hours or until cancelled. Manual initialization remains available.

5.9. Errors

Error Cause What to do
insufficient disk space on master / on replica, 507 not enough space. During re-initialisation the old copy occupies space until the end chapter 4.4; on the replica (from HQbird 2027.2.24) — the Remove the old replica checkbox (5.1)
node_too_old (from HQbird 2027.2.24) the Remove the old replica checkbox is set, but the master or replica node is an old version upgrade the node as stated in the error text or run without the checkbox
engine_busy, reclaim_in_progress (from HQbird 2027.2.24) the replica cannot delete the old copy right now: Firebird or this database is busy retry later (5.6)
replica_removed_for_reinit, NEEDS_REINIT pair (from HQbird 2027.2.24) the replica deleted the old copy, but initialisation did not finish start initialisation again (4.4)
publication_off (from HQbird 2027.2.24) the master’s publication is disabled or contains no tables with a key Synchronise publication and initialise (5.1)
long_transactions, “Smooth reinit is waiting for long transactions” long write transactions are open 5.4
long_transactions, “the master’s archive does not yet contain a single segment” first initialisation under load Normal (5.4)
reinit_pre_publication_transactions transactions started before the publication was enabled reconnect the applications, retry
duplicate_guid two master databases with the same GUID remove the copy; if the NEW GUID step was skipped when promoting to master — -force (chapter 8.4)
license_paused the master’s fbagent has no licence chapter 2.8
409 orphaned the database is outside the master’s scan chapter 3.8
incompatible_engine different major Firebird versions chapter 4.3
target_not_replica a working database sits in the replica’s place (it was made normal) see below
target_other_replica (from HQbird 2027.2.24) a copy of another database of this node sits in the replica’s place see below
restart_not_allowed the replica’s Firebird also serves master databases 5.6, the “Allow … to stop Firebird” checkbox
fbagent_required fbagent is not installed on the replica chapter 1
nbackup backup lock already active an nbackup backup is in progress 5.7
… -L did not finish in 2m0s and was stopped nbackup -L on the master did not finish within 2 minutes, so the node stopped it retry; if the error recurs, check the master’s Firebird (firebird.log)
reinit already in progress for this database initialisation of this database is already running wait for it to finish

target_not_replica. A working database sits in the copy’s place: it was made normal (the file <database>.hqcluster-normal.json exists) or it is not in replica mode. The node does not overwrite a working database. If it needs to be replaced, a request to the node’s API with overwrite_non_replica: true (there is no button for this in RCM) will replace the file and keep the old one as <database>.pre-reinit-<UTC time>.fdb.

target_other_replica (from HQbird 2027.2.24). The replica places the copy in <databases.root>/<master id>/<database path>. The master id is unique only within its own cluster, so two databases can end up with the same path. The replica node does not overwrite a copy belonging to another of its databases: the error names that database and its master. The node also recognises a copy whose path is recorded differently (a symbolic link or junction in the path, a short Windows folder name). overwrite_non_replica does not clear this error. Move one of the copies to another folder or remove that database from the replica, then start initialisation again. The node sees only its own databases: two replica nodes on the same host must have different replica folders (databases.root).

5.10. How to check the result

  1. Databases: the master — replica pair in the group is Ok, state IN_SYNC.

  2. Verify in the master’s card: the nodes count the rows of the tables from the database’s table list (chapter 3.1) on the master and on the replica and compare them. Check during quiet time: with active writes the numbers may differ.

    With HQbird 2027.2.24 the result page names each side’s list (for example, “this database’s own list” or “publication tables (list not set)”). RCM counts the replica using the master’s list. A table that is absent from both databases is shown as “not in database” and is not counted as an error. If no tables were compared, the result is “No tables to compare”, not a mismatch. If the comparison did not finish within the allotted time, the result is “Not compared (time limit)”, which is also not a mismatch.

Verification result
Verification result
Command
hqbirdrcm verifydata -master node-a -master-db <db_id> -replica node-b -replica-db <db_id>
  1. Periodic check (optional): the Data check page, Turn on for all nodes. With HQbird 2027.2.24 the node applies the settings immediately, no node service restart is needed: after Save RCM writes “Saved and already in effect”. Node 2027.5.7 needs the hqclusternode service restarted, and RCM 2027.2.24 says so. The nodes compare the row counts themselves when the master and replica are on the same segment. A mismatch raises the data_mismatch alert; then run Verify.

A table that the operator has removed from the publication is shown as “not published”, not as a mismatch.

6. Configuring publication

The publication tells the Firebird master which tables’ changes to write to the segments. Without it, the replica receives nothing.

6.1. What the publication is

In Firebird 4 and 5, the tables to be replicated are defined by the RDB$DEFAULT publication:

  • Publication enabled — in RDB$PUBLICATIONS it has RDB$ACTIVE_FLAG = 1 (the ALTER DATABASE ENABLE PUBLICATION command).
  • Published tables — the list in RDB$PUBLICATION_TABLES.

Firebird writes changes to the segments only for published tables and only when the publication is enabled. Registering the database (chapter 3) does not enable the publication. It is enabled by Synchronise publication (6.2).

The master node checks the publication every hour (limits.publication_check_interval_sec, 3600 by default). If it is disabled, the node checks it every 2 minutes.

On the Databases page, the master database has an icon:

Icon Meaning
publication ON publication enabled
publication OFF Firebird writes nothing to the segments, replicas receive no changes
publication unknown the node could not read the state

The Publication off filter shows databases with a disabled or unknown publication.

If a database with a disabled publication already has a replica, the master raises the master_publication_off alert.

With HQbird 2027.2.24, the master raises this alert also when it refused to initialise a database with the publication_off error (chapter 5.1): the publication is disabled or contains no table with a key. The alert level:

  • warn — the database has no replica yet: initialisation is waiting for the publication;
  • serious — the database already has a replica;
  • info — the operator disabled the publication.

The alert is cleared when the publication is fixed or another initialisation has started.

6.2. Synchronise publication

Master row: publication disabled
Master row: publication disabled

Where to click:

  • in the master database row on the Databases page. When the publication is disabled, the button is next to the icon;
  • on the database page;
  • for several databases: tick the rows, in the Action list select Synchronise publication. RCM will send a separate command for each database.
Confirming publication synchronisation
Confirming publication synchronisation

What the node does (a single transaction):

  1. Enables the publication. If the operator disabled it themselves (6.4), it does not enable it.
  2. Adds to the publication all tables with a key that are not there yet.
  3. Removes from the publication tables without a key and tables that the operator excluded (6.3).
  4. Disables automatic addition of new tables (EXCLUDE ALL FROM PUBLICATION, then adds the required tables). Only the node defines the composition of the publication.
Master row after synchronisation: publication enabled
Master row after synchronisation: publication enabled

A table with a key — a user permanent table that has a primary key or an active unique index. System and temporary tables are not published.

Synchronisation also removes from the publication tables without a key that the DBA added manually.

If the composition is already correct, the node changes nothing.

When the node refuses:

Code Reason
no_publications HQbird 2.5/3.0 (6.6)
not_master the node is not the master
reinit_in_progress initialisation of this database is in progress
orphaned, database_failed, file_missing the database is outside scanning, in error, or has no file

All node databases at once — from the command line only. On the Commands page the result is shown per database: done, skipped or error. Databases in the FAILED or SEEDING state and databases without a file are skipped.

The same from the command line

Command
hqbirdrcm setpublication -node node-a -db <db_id>   # one database
hqbirdrcm setpublication -node node-a               # all node databases

The command waits up to 5 minutes and prints a line for each database. On error the exit code is 1. -nowait — do not wait for the result.

6.3. Publication tables

The “Publication tables” page
The “Publication tables” page

Open Publication tables in the master database row.

The page shows:

  • the publication state (on/off, disabled by operator) and the counters;
  • each table: name, key and state.
State Meaning
published changes go to the segments
not yet published a table with a key, will be added on synchronisation
excluded the operator removed it from the publication
no key cannot be published

In the key column — the primary key, or, if there is none, the unique index with the smallest number of fields.

Why a table without a key cannot be published. The replica looks up the row for UPDATE and DELETE by key. Without a key it will not find the row. Add a primary key or a unique index, then synchronise the publication.

Exclude from publication / Return to publication:

  1. Tick the tables.
  2. Click Exclude from publication or Return to publication.
  3. The node saves the list and immediately synchronises the publication.
Confirming “Exclude from publication”
Confirming “Exclude from publication”

The list of excluded tables is stored on the master node. Synchronisation does not change it. When a replica is promoted to master (chapter 8), the list moves to the new master.

Check (chapter 5.10) shows an excluded table as “not published”, not as a discrepancy.

6.4. Enabling and disabling the publication

On the Publication tables page: Disable publication / Enable publication.

  • Disabling is the operator’s choice. Synchronisation does not re-enable the publication.
  • Changes made while the publication is disabled (or while a table is excluded) are not sent to the replica, even after it is enabled.
  • After enabling, click Check (chapter 5.10). If the data diverges — reinitialise the replica (chapter 5).

6.5. A new table

A new table does not get into the publication by itself: the node disables automatic addition.

After DDL:

  1. Create a primary key or a unique index on the table.
  2. Click Synchronise publication.
  3. Ask the applications to reconnect if reinitialisation is needed. Firebird does not write to the segments transactions that started before the publication was enabled; COMMIT RETAINING keeps such a transaction open. Initialisation waits for such transactions and then fails with reinit_pre_publication_transactions (chapter 5.4).
A new table in the “not yet published” state
A new table in the “not yet published” state

Rows written to the new table before synchronisation will not reach the replica. If there are any, check the data and, if necessary, reinitialise the replica.

6.6. HQbird 2.5 and 3.0

HQbird 2.5 and 3.0 have no publications. Firebird writes segments if the replconf plugin file contains a master block. The publication buttons on such nodes are hidden, and the API responds with no_publications.

The set of tables is defined by the engine parameters:

Parameter Meaning
include_filter which tables to replicate: names or patterns separated by commas; empty — all
exclude_filter which tables not to replicate

The parameters are changed in Settings → Replication parameters (chapter 7.1). The Publication tables page shows them read-only, with a link Change in engine settings. Firebird reads the new values after a restart.

Publication tables on HQbird 3.0: engine filters
Publication tables on HQbird 3.0: engine filters

The replconf plugin. The node writes its own replconf.hqcluster.hqbird file in the Firebird folder. Firebird reads it only when replconf.properties points to it. The plugin must be activated once:

  1. Settings → Nodes, the Replication plugin (HQbird 2.5 / 3.0) block. The block shows where replconf.properties points and the problems that the node found.

Plugin not activated: replconf.properties points to the HQbird DataGuard file
Plugin not activated: replconf.properties points to the HQbird DataGuard file
  1. Activate… The window lists what the node will do: install the plugin, point replconf.properties to the node’s file, migrate the entries of the old file. Choose when: Only within the node restart window or Now, ignoring the restart window.

Plugin activation window
Plugin activation window
  1. After the Firebird restart, the Activated field shows “yes: the engine reads the node’s file”.

Plugin activated
Plugin activated

Activation restarts Firebird and affects all databases of this Firebird. Perform it during a quiet period.

7. Other replication options

7.1. Replication parameters

Settings → Replication parameters, wizard form
Settings → Replication parameters, wizard form

The Settings → Replication parameters page changes the keys in replication.conf for the selected databases or for the node as a whole.

  1. Add master and replicas — select the databases. RCM groups them by Firebird version and role: a separate form for masters and for replicas.

  2. For each parameter, choose an action:

    Action What it does
    no changes the value is not changed
    set value a new value
    reset (to the next level) remove its own value, take the value from the level above
    do not write (Firebird value) the key is not written, the Firebird value applies
  3. Choose where to apply it: To the selected databases or As the default for their nodes.

  4. Preview — the nodes check the values without changing anything.

Preview of changes
Preview of changes
  1. Apply. RCM sends the command to each node.

Where the value comes from (the Current column):

Source Value
this database set for the database
node value set as the default for the node
built-in node value the node value from the code
Firebird value the key is not written
from file (not yet set here) the value from replication.conf, RCM has not changed it yet
set by the node read-only (paths, prefixes)

A node that is not responding shows its databases as unavailable for selection. Changes require fbagent (otherwise fbagent_required).

Who restarts Firebird (Firebird reads replication.conf only at startup):

Node Who
master you restart it: the master never restarts Firebird itself
replica with firebird.auto_restart: never you
replica whose Firebird also serves the master’s databases you
other replicas the node restarts it in its own restart window

For a manual restart, the page has a Restart Firebird button.

7.2. Main parameters

Master, Firebird 4/5:

Parameter Label in RCM Node value Firebird Tip
journal_archive_timeout Archive timeout 10 s 60 s how many seconds Firebird waits before closing an incomplete segment. Lower — the replica is closer to the master. Must be less than limits.data_check_quiet_sec (90)
journal_segment_count Number of segments 64 8 how many segments can reside in .ReplLog. Must be greater than limits.backlog_critical (48)
journal_segment_size Segment size — 16 MB from 1 MB to 512 MB; the receiver will not accept more than 512 MB
buffer_size — 1 MB

The node also writes log_errors = true (cannot be changed) and disable_on_error = true.

Replica, Firebird 4/5:

Parameter Label in RCM Firebird Value
apply_idle_timeout Idle pause 10 s pause when there are no new segments
apply_error_timeout Pause after apply error 60 s pause before retrying after an error
verbose_logging node: true verbose replication.log; not recommended to disable
skip_upddel_without_pk node: true skip UPDATE/DELETE on tables without a key

Firebird 5 only: include_filter, exclude_filter, cascade_replication.

HQbird 2.5/3.0 — log_* keys with the same meaning: log_archive_timeout, log_segment_count, log_segment_size (not less than 2 × buffer_size). The alert_command key is set only in node.json.

A hint for each parameter is available in the RCM form.

7.3. Archive storage

The <database>.LogArch archive on the master is cleaned up in stages:

  1. Segments that have been applied by all replicas and that are older than retention_days.
  2. If free space is less than free_space_floor_gb — applied segments younger than the retention period. Alert disk_serious_cleanup.
  3. If space is still low — unapplied segments, starting with the oldest. The replica receives NEEDS_REINIT, alert replication_paused_reinit_scheduled.
Parameter (Settings → Nodes, Storage and files) Default
limits.retention_days — Archive retention, days 2
limits.retention_min_kept_segments — Always keep the last segments 0 (disabled)
limits.free_space_floor_gb — Minimum free space 1 GB
Settings → Nodes, “Storage and files”
Settings → Nodes, “Storage and files”

Clean up the archive manually:

Command
hqbirdrcm cleanuparchive -node node-a              # stages for all databases on the node
hqbirdrcm cleanuparchive -node node-a -db <db_id>  # a single database

-orphaned (only together with -db) deletes all segments of the database in the ORPHANED state (chapter 3.8). For any other database the node will respond not_orphaned. Never run -orphaned on the promotion donor (chapter 8.5).

7.4. Transfer: compression, speed, windows

Compression is set per peer: Settings → Nodes → Peers, column compress.

Value Meaning
empty or none no compression (default)
zstd zstd compression

The change takes effect immediately. Compression is useful on a slow network.

Segment transfer speed (Settings → Nodes, Transfer and reinitialization):

Parameter Default When it takes effect
limits.channel_a_mbps — Segment transfer speed 50 MB/s immediately
limits.channel_a_burst_mbps — Segment transfer burst 200 MB immediately

The unit is mebibytes (MiB), even though the name says “mbps”. The burst cannot be less than the speed.

Settings → Nodes, “Transfer and reinitialization”
Settings → Nodes, “Transfer and reinitialization”

Time windows (Time windows):

Parameter Default When it takes effect What it affects
windows.restart_window — Firebird restart window master 22:00-05:00, replica always immediately when the replica node may restart Firebird
windows.transfer_window — Transfer window 23:00-05:00 after node restart full initialization speed (chapter 5.5)

The label next to each field shows when the change takes effect: immediately or node restart.

Settings → Nodes, “Time windows”
Settings → Nodes, “Time windows”

7.5. Data verification

  • Verify — a one-off comparison of the row count on the master and the replica (chapter 5.10).
  • Data verification — a periodic comparison (chapter 5.10). The node performs the comparison when the master and the replica are on the same segment.
  • With HQbird 2027.2.24, verification settings take effect immediately after Save; restarting the node service is not required. RCM displays “Saved and already in effect”. After enabling, the first verification runs after one interval. After disabling, a verification in progress is interrupted and writes nothing. The new interval is counted from the end of the last verification. Keep, days, Query limit, Quiet and Progress generator take effect from the next verification.
  • Node 2027.5.7 reads these settings only at startup: after Save, restart the hqclusternode service on that node. RCM 2027.2.24 displays “Saved, but not yet in effect” and states the reason.
  • Which tables to compare is defined by the database’s table list (chapter 3.1). With HQbird 2027.2.24, each database can have its own list: the field at registration or the Synchronization tracking tables block on the master’s database page (chapter 3.3). The Data verification page shows which list was used, and the number of tables “not in the database”. In 2027.5.7 the list is per node (sync_tables).
  • Tables excluded from publication are shown as “not published” (chapter 6.3).
Data verification page
Data verification page

7.6. Alerts

Alerts page
Alerts page

The Alerts page. Actions:

  • Acknowledge — the alert remains visible but is marked acknowledged;
  • Mute for 60 min — hide for an hour;
  • Delete / Delete all with this code — RCM passes the deletion to the node that raised the alert. If the cause remains, the alert will appear again.
Command
hqbirdrcm clearalerts -code backlog_warn   # also -id, -acked, -all

As of HQbird 2027.2.33, RCM sends alerts to Telegram, to a webhook (Slack, Teams, JSON) and by e-mail. The channels are configured in the notify.channels section of the rcm.json file. The Notification channels block on the Alerts page shows the channels and sends a test message. RCM reports problems of level warn and above: a problem has appeared, has become more serious, or has been cleared. Without channels, RCM sends nothing. Earlier versions have no sending: watch the Alerts page or poll the RCM API.

Main codes:

Code Meaning
backlog_warn / backlog_critical 32 / 48 unarchived segments in .ReplLog
disk_serious_cleanup the archive is being cleaned up at stage 2 (7.3)
replication_paused_reinit_scheduled unapplied segments deleted, initialisation required
shipping_paused shipping to the peer is suspended (7.7)
apply_error, replication_stopped, replication_unrecoverable_error the replica cannot apply a segment
replication_guid_mismatch the master database GUID has changed
segment_missing, segment_rejected, sequence_gap problems with the segment sequence
firebird_down, firebird_log_error Firebird is not running or is logging errors
firebird_not_answering as of HQbird 2027.2.30: Firebird has not answered a database connection for a minute (hang). This also includes a connection that failed to establish due to a timeout; firebird_down is a connection refusal only
master_publication_off publication is off (chapter 6); as of HQbird 2027.2.24 and after a publication_off failure (chapter 6.1)
replica_removed_for_reinit as of HQbird 2027.2.24: the replica has deleted the old copy for initialisation, and there is no new copy yet (chapter 4.4)
orphaned_database database outside scanning (chapter 3.8)
restart_required_after_reinit the replica’s Firebird needs a restart (chapter 5.6)
data_mismatch the periodic check found a discrepancy
license_paused fbagent has no licence (chapter 2.8)

7.7. Pausing shipping

On the Databases page, in the master row: Pause shipping / Resume shipping. For multiple databases — via the Action list.

Command
hqbirdrcm pauseshipping  -node node-a -db <db_id>
hqbirdrcm resumeshipping -node node-a -db <db_id>
Pause shipping confirmation
Pause shipping confirmation
Master row during a pause
Master row during a pause
  • The pause remains in effect until Resume shipping or until initialisation.
  • The master Firebird keeps writing segments. The .LogArch archive grows: while the replica is stopped, stage 1 (7.3) does not delete its segments.
  • A long pause may reach stage 3 and NEEDS_REINIT. Keep an eye on space.
  • The button is absent if the database has no replica or publication is disabled. The Pause: no licence icon means a licence-related pause (chapter 2.8); it is cleared only by a licence.

7.8. Logs

Log Where
Node <installation folder>/logs/node-YYYYMMDD.jsonl (/opt/hqclusternode or C:\hqclusternode), 32 MB each, 20 files
Firebird replication.log, firebird.log paths in node.json: firebird.replication_log, firebird.firebird_log
RCM Linux: journalctl -u hqbirdrcm
RCM audit rcm-data/audit.jsonl

Remote node log — the Logs page: select a node and a source (replication.log, firebird.log, node agent).

Logs page
Logs page
Command
hqbirdrcm getlogs -node node-b -source replication -tail 200

8. How to transfer a replica to a regular database

There are two paths:

Path When What you get
Turn to normal, 8.1 you need a separate working copy: reports, tests, a failure without a new master a regular database outside replication
Promote to master, 8.3 the replica should become the new master a database managed by a companion node with the master role

Neither path can be undone through the interface.

8.1. Make Normal

Replica “Action” list: the “Make Normal” item
Replica “Action” list: the “Make Normal” item
  1. Databases, replica side: tick the replica rows.

  2. In the Action list, select Make Normal and confirm. There is no separate button in the row. There is no command-line command either.

“Make Normal” confirmation
“Make Normal” confirmation
3. The replica node: - runs `gfix -replica none` (HQbird 2.5/3.0: `gfix -replica {}`). If the database is busy applying segments, the node puts it into `single`-shutdown for a few seconds (`gfix -shut single -force 0`), clears the replica mode and returns it with `gfix -online normal`. **All connections to this database are dropped**; - writes the marker file `.hqcluster-normal.json` (time, author, db_id, GUID); - starts clearing the `.Incoming` queue (8.2). 4. RCM shows the database in the **Not a replica (normal)** group.
The “Databases” page after “Make Normal”
The “Databases” page after “Make Normal”

After this, the database is open for writing.

Remove the replica block from replication.conf. The node removes it on the next scan, and Firebird reads the file at startup:

Command
hqbirdrcm scansync -node node-b
hqbirdrcm restartfirebird -node node-b

Until then, “database is not in replica mode” errors for this database are not raised as alerts.

The marker file protects the database. Initialization will not overwrite such a database: it is refused with target_not_replica (chapter 5.9). Deleting only the marker is not enough: the node also checks the replica mode in the database header. The Delete old replica checkbox (chapter 4.4) does not delete such a database either.

With HQbird 2027.2.24, while the replica is deleting the old copy of this database for initialization, Make Normal is refused with reclaim_in_progress: retry later.

If the database needs to become a replica again — only a new initialization with overwrite will do. This is done via the node API (overwrite_non_replica: true); there is no such button in RCM. The old file is saved as <database>.pre-reinit-<UTC time>.fdb, and the marker is deleted.

8.2. What Happens to Segments

  • Until the node has scanned the databases, the master keeps sending segments. Every 30 seconds the replica node deletes old segments and keeps only the last 100 (keep_segments).
  • After scanning, the database record on the replica gets the state ORPHANED. The replica node replies to the master with unknown_database. The master stops sending segments to this replica, stops keeping an archive for it, and re-checks it every 10 minutes.
  • Other replicas of this database are not affected.
  • The ORPHANED record can be deleted (chapter 3.8): hqbirdrcm forgetdatabase. The database file itself is not deleted.

8.3. Promote to master

The replica becomes the new master. Its file stays in place, and control passes to the companion node.

Conditions:

Condition Rejection
the database is a replica bad_request
a node with the master role — the companion — is running on the same host (same host label in rcm.json, chapter 2.1) no_companion
promotion of this database is not already in progress already_running
there is no initialisation into this database or from its master reinit_running

RCM does not create the companion node: install it in advance (chapter 1).

How to start:

  • in the replica row, Promote to master, or in the Action list;

  • from the command line:

    Command
    hqbirdrcm promotetomaster -node node-b -db <db_id>   # -timeout 10m

RCM will ask for confirmation and name the companion node.

Confirmation “Promote to master?”
Confirmation “Promote to master?”

Steps (visible in the replica row, names in English):

No. Step What it does
1 TURN TO NORMAL as in 8.1; skipped if the database is already normal
2 NEW GUID a new database GUID in place (8.4)
3 EXCLUDE ON DONOR the replica node (donor) excludes the file from its scan (exclude_paths)
4 ENROLL ON the companion node registers the file as a master database
5 PUBLICATION SYNC publication synchronisation; the list of excluded tables is carried over from the old master (chapter 6.3)
6 RESTART DONOR FIREBIRD restart of the donor’s Firebird, without waiting for a window
7 RESTART FIREBIRD restart of the companion’s Firebird, without waiting for a window
8 FINALIZE the donor row becomes Ghost replica; the old master may become Master degraded (8.5)
Promotion steps in the replica row
Promotion steps in the replica row

On HQbird 2.5 and 3.0 there are no publications: the PUBLICATION SYNC step completes with the text “this engine has no publication”.

Promotion steps on HQbird 3.0 (Windows)
Promotion steps on HQbird 3.0 (Windows)

If the donor and the companion use the same Firebird, that Firebird is restarted: connections to all of its databases are dropped.

If a step fails, the job gets failed with the error text. Fix the cause and click Promote to master again: completed steps are not repeated. If RCM restarts during promotion, the job will also become failed — check the state of the nodes and retry.

Licence. If fbagent on the master’s host has no licence, the master has paused sending. Promotion is still possible. RCM will warn you: changes made after the pause began will be lost.

8.4. New GUID

After promotion, the database must get its own GUID. Otherwise the new master writes segments under the old master’s GUID, and its replicas are rejected with duplicate_guid.

The NEW GUID step on the node:

  1. gfix -shut single -force 0 — all connections to the database are terminated;
  2. nbackup -L, then nbackup -F without -SEQUENCE;
  3. deletes <database>.delta;
  4. checks the header: new GUID, REPLICATION_SEQUENCE = 0;
  5. gfix -online normal.

Firebird is not restarted. The method works on all versions (2.5, 3.0, 4.0, 5.0).

If the step was interrupted, the node issues the alert newguid_interrupted. The database may remain in single-shutdown. Repeat Promote to master: the step will continue.

The step is skipped (the text begins with skipped:) if the donor node is an old version or the file already has its own GUID.

After a new GUID, the old nbackup chain is no longer suitable. Make a new level-0 nbackup copy.

If the database was created with ADD DIFFERENCE FILE, its .delta is located in a different place: delete it manually.

8.5. After promotion

Donor record. On the replica node the database remains ORPHANED, the path stays in exclude_paths, and the .Incoming folder and the label remain. In RCM this is the row Ghost replica with the buttons Open on new master and Clear.

Donor row: «Ghost replica»
Donor row: «Ghost replica»
  • To remove the row, click Clear. This changes RCM only.
  • To remove the donor record from the replica node, run forgetdatabase. The file and the .Incoming folder remain. HQbird 2027.2.30 and newer does not reject the command (mailbox_active), because the donor path is in exclude_paths. Nodes of earlier versions reject it while the file and the .Incoming folder are present.
  • Never run cleanuparchive -orphaned on the donor. The donor and the companion share the same <database>.LogArch folder. The command will delete the live archive of the new master.

Old master.

  • If the donor was its last working replica, RCM sets the old master to Master degraded and pauses sending to the donor (“sending paused: … (promotion to master)”). Do not resume sending to the donor: it already holds a working database. To clear this state, initialise the database onto another replica.

    Old master: «Master degraded», sending paused
    Old master: «Master degraded», sending paused
  • Initialisation from the old master to the donor node will be rejected (target_not_replica): it already holds a working database.
  • If the NEW GUID step was skipped, the old and new masters have the same GUID. In that case initialisation from the old master requires -force (chapter 5.9).
  • After a manual promotion RCM does not turn the old master into a replica of the new master. Stop the applications on the old master and remove its database from management (chapter 3.8). For a database described as a service, this is done by Rejoin (8.7).

New master. Set up a replica for it: replica node, peers (chapter 2), publication (chapter 6), initialisation (chapter 5).

New master: database with a new GUID, no replica yet
New master: database with a new GUID, no replica yet

8.6. Planned switchover: checklist

This is the procedure for a manual switchover. For a database described as a service, RCM switches the master itself (8.7). A manual promotion does not check whether applications are writing to the old master. Procedure:

  1. In advance. On the replica host, a companion node with the master role is running, and both nodes have the same host label (chapter 2.1). A node for the future replica is ready for the new master.
  2. Stop application writes to the old master’s database.
  3. Wait until the replica has received and applied everything: Databases → IN_SYNC, no queue.
  4. Check (chapter 5.10): the row counts match.
  5. Promote to master (8.3). Wait for the FINALIZE step.
  6. Check: the database is in the companion’s master list, publication on.
  7. Switch the applications to the new master.
  8. Make an nbackup level 0 copy of the new database (8.4).
  9. Clear the ghost record for the donor row (8.5).
  10. Connect the replica to the new master and initialise it (chapter 5).
  11. Old master: remove the database from management or leave it as an archive (8.5).

8.7. Services: switchover and failover in RCM

With HQbird 2027.2.30, RCM moves the master to another host by itself. To do this, the database is described as a service: one logical database on several hosts. On each host of the service there is a master node and a replica node, and the path to the database file is the same. The service is created on the Services page (from HQbird 2027.2.31) or with the command hqbirdrcm services put -id <id> -file <file>.json.

Action What it does Commands
Switchover planned transfer of the master without data loss. Afterwards the old master automatically becomes a replica of the new master hqbirdrcm switchover -service <id> -check, then hqbirdrcm switchover -service <id> -to <host>
Failover the master host is lost: RCM promotes the replica. The master’s most recent changes may be lost hqbirdrcm failover -service <id> -check, then hqbirdrcm failover -service <id> -to <host> -accept-data-loss
Rejoin after a failover, returns the old host as a replica of the new master hqbirdrcm rejoin -service <id>
Resume takes the service out of the failed state hqbirdrcm resume -service <id> -check, then hqbirdrcm resume -service <id>

The same actions are available on the service page: the Switchover…, Failover…, Rejoin… and Resume… buttons.

  • Automatic failover is off by default. It is enabled for each service separately: hqbirdrcm policy -service <id> -auto-failover on -max-data-loss 30. It requires routers. RCM starts a failover only when the master node is silent, the majority of routers cannot see the master’s Firebird, and the replica lag is no more than max_data_loss_sec.
  • A router (hqclusternode router, Linux) keeps HAProxy on the current master of the service. Applications connect to <router host>:<router_port> and do not change the connection string when a switch occurs.
  • After a failover, RCM fences the old master (role fenced) once it responds again. Bring it back with the rejoin command.