HQCenter: server fleet
Many servers in one console
HQCenter is one Linux server at the customer site. All the customer's agents connect to it. The agents send their data there and fetch settings, commands, licenses and new versions from it. The administrator sees the whole fleet in HQ Monitor and manages it from a browser.
- 0 inbound ports to talk to the center
- mTLS certificate for each agent
- 3 min until data reaches the center
- ~1 min until settings reach the agent
- with rollback agent and center updates
Agent · nginx · goafts · fblogspacker · HQ Monitor
Connection without inbound ports
The agent opens every connection. HQCenter never connects to a database server. So you do not open ports on Firebird servers to talk to the center, and a server in a branch office behind NAT works the same as a server in the next rack.
Three programs run behind nginx on HQCenter. goafts receives archives, issues certificates and stores settings, commands, licenses and new versions. fblogspacker sorts archives into agent folders. HQ Monitor shows these folders in a browser.
| HQCenter port | Who connects | Check |
|---|---|---|
:443 | Agents: archives, settings, commands, license, new versions | TLS 1.3 and the agent certificate; a revoked certificate is refused |
:9443 | New agent on first connection, and installers | No agent certificate: certificate request with the key from attach.json, center key fingerprint, installer downloads |
:8443 | Administrator and console: management | Administrator certificate and list of allowed addresses |
:8085 | Browser: HQ Monitor console | List of allowed addresses |
- Connect with one commandThe center gives a file
attach.json: address, key fingerprint and expiry date. On the server:fbagent --attach --file attach.json. HQbird 2027 installers do this for you. A request with a regular key is approved by the administrator in the center panel. A license key approves the request itself while the license has free seats. - Agent name is the certificate nameThe agent's data folder, settings and console row use this name. The Windows installer asks for the server name; the default is the computer name. A certificate is valid for one year; a revoked one stops working at once.
- Nothing is lostAn archive is up to 120 MiB. On a network error, 3 retries. A sent archive is deleted; an unsent one is never deleted. Each agent sends with its own offset of up to 150 s, so the agents of one center do not arrive in the same second.
- Licenses from the centerThe agent fetches a new license in about a minute: the reply to the settings poll tells it about the license. The agent also asks for the license at start and every 6 hours. It checks the license itself, without the network. The Licenses page shows the state of each server.
- The console does not touch databasesHQ Monitor builds its pages from agent files. It does not connect to Firebird and puts no load on databases.
- One Linux serverUbuntu or AlmaLinux. One script installs the center; a second run updates what is there and keeps passwords and keys.
IBSurgeon receives summaries: server health, the server list, the list of backups and restores (without the files), events and weekly reports, cluster descriptions, licenses. Logs, traces, backup files, passwords and the audit log stay with the customer.
Settings editor · Run now · delivery status
Settings from the center
The center keeps a copy of the settings of each agent. The administrator changes jobs and settings in the same editor as next to the agent, and clicks Save and Send to agent. The agent fetches the change in about a minute.
The agent does not write new settings over the working ones. It checks the candidate first. A bad candidate is set aside, the working settings stay, and the console shows the reason for the refusal.
Status on the servers page
For each server you see how far the last change got: a status icon, the send time (push) and the agent reply time (ack). The hint on the icon shows why the agent refused and the exact time.
In a minute you see which servers took the new backup job, which did not, and why.
The server level in the list is the verdict of the agent on the server as a whole. Under the icons there is one line with the main finding. A separate “Agent” icon appears only when the state of the agent itself is worse than the state of the server.
A server that is silent for more than 30 minutes gets a gray “no data since …” icon and moves to the end of the list. The summary line above the list counts the servers by level.
The agent protects the server
The agent watches its own memory and CPU. If an agent task uses too much, the agent pauses it and stops the run that holds the memory. It does not touch backups and data upload. If the agent memory still grows, the agent restarts itself. A task that the agent turned off is shown in the health report and in the editor; one button turns it on again.
Run now from the center
You can start a backup, restore, sweep, gfix check or index recalculation now, without waiting for the schedule. The request takes the same path: into goafts, then the agent fetches it when it polls and puts it in its queue, with the usual checks.
- DraftsSave a change in the console and send it later. You see the state: not sent, sent, applied, superseded.
- Passwords are hidden from the centerThe console encrypts a new password with the agent public key. Only the agent on its own server can decrypt it.
- Settings copiesBefore each replace, the agent saves a copy: the last 10, for at least 14 days. Rejected candidates are kept for 90 days.
- Server onlyRelease signature check, the license, remote commands, program paths and console login cannot be changed from the center.
- Database replace needs two sidesA remote restore that replaces the working database runs only if the server itself allows it and the user has a separate right. Before the start, you must type the database file name.
- databases.conf from the centerYou can send the file to one agent. goafts only passes it on; the agent checks and writes the file and keeps the old one next to it.
Signature · rollback point · check
Updates with rollback
Agents get new versions from HQCenter, not from the internet. The agent updates itself, HQ Monitor, the cluster programs and Firebird. Every update follows one scheme: check the signature, save a rollback point, install, check that it works. If it does not work, go back to the old version.
- 1Signaturerelease signed by IBSurgeon
- 2Rollback pointcopy of the current version
- 3Installreplace the files
- 4Checkthe new version reports "I am working" and runs for 45 s
- 5Donethe new version works
- ↺Rollbackrestore the copy from the rollback point
- ✓Old version worksafter 2 rollbacks the version is blocked
| What is updated | When | Protection |
|---|---|---|
| Agent | at once when the reply to the settings poll reports a new version, and once an hour; without such a reply, every 180 s | Signature, 2 rollback points. The new version must report "I am working" and run for 45 s. After 2 rollbacks the version is blocked. |
| HQ Monitor | once an hour while the settings poll works; otherwise every 180 s | Console health check for 30 s, copy of the old version. |
| Cluster programs | check every 180 s | Replica first, then master. |
| Firebird | only in the window, default 03:30–04:00 | Waits for backups to finish. Check after install, rollback on failure. A rolled-back build is not installed again. |
| HQCenter | every 15 min, if IBSurgeon turned on updates for this center; in a window that IBSurgeon sets, for example 02:00–05:00 | Off by default. The center owner can forbid the update: STACK_UPDATE=off. The stack signature is checked, then services are checked for up to 5 minutes. If they fail, the previous stack from the cache is installed. |
- Test first, then everyoneIBSurgeon publishes a release to the test zone first, and after checks to the production zones. HQCenter copies the releases of its zone once an hour.
- Signature checked twiceThe center checks the signature before download and again on the written bytes. The agent checks it again before install.
- No downgradeYou cannot install a version lower than the current one, unless the server itself allows it.
- On Linux, root changes the filesThe agent service does not run as root and cannot replace trusted install files.
Status watch · email · webhook
Problem emails and a weekly report
HQCenter reads the health snapshot of each agent. A new problem, or a problem that got worse, is an event. The center itself sends the email or webhook. Once a week the center builds a report for the whole fleet.
The center changes nothing by itself. If a problem stays for weeks and that is normal for the server, the center suggests a new threshold. Only a person applies it.
- 1Read statusevery 2 minutes, if the agent file changed
- 2Eventnew, got worse, gone, came back
- 3Emailor webhook; no repeat within 24 h
- 4Weekly reportMonday at 07:00 by default
- 5✓ Suggestionnew threshold; the administrator applies it
The whole fleet over 14 days
For each server: share of time without problems, episodes per week, and which findings are "stuck".
Not noise but normal
Active at least half of the time, and for at least one week in a row. For example, server CPU is always near the threshold.
Threshold from real data
A bit past what the server showed: a quarter above the maximum. For a lower threshold, such as free space, a fifth below the minimum. You can change it before approval; it is valid for 30 days.
- One host, one emailIf a host runs several agents, a host CPU, memory or disk problem counts once.
- Gone or hiddenThe event says why the problem went away: fixed, acknowledged by the administrator, or became info.
- Approval is a new settings versionThe threshold goes to the agent by normal delivery, on behalf of the administrator. Each approval and refusal is in the audit log.
- Link in the emailThe email names the host, gives a hint and a link to the server in HQ Monitor.
The center administrator turns on email and webhook. Until they are set up, no emails go out. Events and reports are visible in the center panel and in its API.
Users · rights · audit · languages
Users and rights
Each user sees only their own servers. Rights to make changes are granted separately, one by one.
The administrator sees all servers and manages users. A regular user sees the servers from their list. A new right or a new list takes effect on the next page load; a new password signs the user out of all browsers.
| Right | What it allows |
|---|---|
| Send settings | Send changed settings to the agent through the center. |
| Run jobs | Start and cancel a backup, restore, sweep or index recalculation (Run now). |
| Jobs that replace a database | Start a restore that replaces the working database. Only together with the run right, and only if the agent also allows it. |
- Audit logEach save, send, password change and job start is written with the user login. In goafts, a hash chain protects the log.
- Agent request logThe agent itself writes a line for each request that changes the server and for each access refusal: Local API, commands from the center, Run now, settings changes, updates. Passwords and request bodies do not go into the log. The file goes to the center together with the logs.
- HQ Monitor loginSession lasts 7 days. No more than 8 login attempts from one address in 15 minutes.
- Center admin panelA separate login, only with an administrator certificate. Session lasts 8 hours. After 5 failed attempts in 15 minutes, login is closed for 15 minutes.
- Up to 2000 usersEach has their own password and server list; search by server when you assign servers.
- 5 interface languagesEnglish, Russian, German, Portuguese (Brazil), Spanish. Statistics reports also use the chosen language.
← Back to the HQbird 2027 overview · Server health in a minute →