Many servers in one console

HQCenter is one Linux server at the customer site. All the customer's agents connect to it. The agents send their data there and fetch settings, commands, licenses and new versions from it. The administrator sees the whole fleet in HQ Monitor and manages it from a browser.

  • 0 inbound ports to talk to the center
  • mTLS certificate for each agent
  • 3 min until data reaches the center
  • ~1 min until settings reach the agent
  • with rollback agent and center updates

Settings editor · Run now · delivery status

Settings from the center

The center keeps a copy of the settings of each agent. The administrator changes jobs and settings in the same editor as next to the agent, and clicks Save and Send to agent. The agent fetches the change in about a minute.

The agent does not write new settings over the working ones. It checks the candidate first. A bad candidate is set aside, the working settings stay, and the console shows the reason for the refusal.

Delivery states: SENT (in the center), TAKEN, then APPLIED, ALREADY SET, RESTART NEEDED or REJECTED; SUPERSEDED and DRIFT
    Log times are examples. SENT means: the change waits in the center, and the agent has not taken it yet. DRIFT appears if the agent says nothing about the change for 10 minutes (default). The English interface uses these labels: SENT, TAKEN, APPLIED, ALREADY SET, RESTART NEEDED, REJECTED, SUPERSEDED, DRIFT.

    Status on the servers page

    For each server you see how far the last change got: a status icon, the send time (push) and the agent reply time (ack). The hint on the icon shows why the agent refused and the exact time.

    In a minute you see which servers took the new backup job, which did not, and why.

    The server level in the list is the verdict of the agent on the server as a whole. Under the icons there is one line with the main finding. A separate “Agent” icon appears only when the state of the agent itself is worse than the state of the server.

    A server that is silent for more than 30 minutes gets a gray “no data since …” icon and moves to the end of the list. The summary line above the list counts the servers by level.

    Delivery status. English interface. APPLIED: the agent applied the change; REJECTED: the agent refused, the reason is in the hint.
    Editor in the center. A copy of the agent settings. Save a draft, or send it to the agent at once.
    Agent work from afar. Activity tab: how many times each task ran, and how much time, CPU and memory it used.

    The agent protects the server

    The agent watches its own memory and CPU. If an agent task uses too much, the agent pauses it and stops the run that holds the memory. It does not touch backups and data upload. If the agent memory still grows, the agent restarts itself. A task that the agent turned off is shown in the health report and in the editor; one button turns it on again.

    Run now from the center

    You can start a backup, restore, sweep, gfix check or index recalculation now, without waiting for the schedule. The request takes the same path: into goafts, then the agent fetches it when it polls and puts it in its queue, with the usual checks.

    pendingdeliveredqueuedwaitingrunningdonefailedrefusedexpiredcancelled
    • DraftsSave a change in the console and send it later. You see the state: not sent, sent, applied, superseded.
    • Passwords are hidden from the centerThe console encrypts a new password with the agent public key. Only the agent on its own server can decrypt it.
    • Settings copiesBefore each replace, the agent saves a copy: the last 10, for at least 14 days. Rejected candidates are kept for 90 days.
    • Server onlyRelease signature check, the license, remote commands, program paths and console login cannot be changed from the center.
    • Database replace needs two sidesA remote restore that replaces the working database runs only if the server itself allows it and the user has a separate right. Before the start, you must type the database file name.
    • databases.conf from the centerYou can send the file to one agent. goafts only passes it on; the agent checks and writes the file and keeps the old one next to it.

    Signature · rollback point · check

    Updates with rollback

    Agents get new versions from HQCenter, not from the internet. The agent updates itself, HQ Monitor, the cluster programs and Firebird. Every update follows one scheme: check the signature, save a rollback point, install, check that it works. If it does not work, go back to the old version.

    1. 1Signaturerelease signed by IBSurgeon
    2. 2Rollback pointcopy of the current version
    3. 3Installreplace the files
    4. 4Checkthe new version reports "I am working" and runs for 45 s
    5. 5Donethe new version works
    1. ↺Rollbackrestore the copy from the rollback point
    2. ✓Old version worksafter 2 rollbacks the version is blocked
    rollbacks of this version: 0 of 2
    Click "Start update".
    What is updatedWhenProtection
    Agentat once when the reply to the settings poll reports a new version, and once an hour; without such a reply, every 180 sSignature, 2 rollback points. The new version must report "I am working" and run for 45 s. After 2 rollbacks the version is blocked.
    HQ Monitoronce an hour while the settings poll works; otherwise every 180 sConsole health check for 30 s, copy of the old version.
    Cluster programscheck every 180 sReplica first, then master.
    Firebirdonly in the window, default 03:30–04:00Waits for backups to finish. Check after install, rollback on failure. A rolled-back build is not installed again.
    HQCenterevery 15 min, if IBSurgeon turned on updates for this center; in a window that IBSurgeon sets, for example 02:00–05:00Off by default. The center owner can forbid the update: STACK_UPDATE=off. The stack signature is checked, then services are checked for up to 5 minutes. If they fail, the previous stack from the cache is installed.
    • Test first, then everyoneIBSurgeon publishes a release to the test zone first, and after checks to the production zones. HQCenter copies the releases of its zone once an hour.
    • Signature checked twiceThe center checks the signature before download and again on the written bytes. The agent checks it again before install.
    • No downgradeYou cannot install a version lower than the current one, unless the server itself allows it.
    • On Linux, root changes the filesThe agent service does not run as root and cannot replace trusted install files.

    Status watch · email · webhook

    Problem emails and a weekly report

    HQCenter reads the health snapshot of each agent. A new problem, or a problem that got worse, is an event. The center itself sends the email or webhook. Once a week the center builds a report for the whole fleet.

    The center changes nothing by itself. If a problem stays for weeks and that is normal for the server, the center suggests a new threshold. Only a person applies it.

    1. 1Read statusevery 2 minutes, if the agent file changed
    2. 2Eventnew, got worse, gone, came back
    3. 3Emailor webhook; no repeat within 24 h
    4. 4Weekly reportMonday at 07:00 by default
    5. 5✓ Suggestionnew threshold; the administrator applies it
    Report

    The whole fleet over 14 days

    For each server: share of time without problems, episodes per week, and which findings are "stuck".

    Stuck finding

    Not noise but normal

    Active at least half of the time, and for at least one week in a row. For example, server CPU is always near the threshold.

    Suggestion

    Threshold from real data

    A bit past what the server showed: a quarter above the maximum. For a lower threshold, such as free space, a fifth below the minimum. You can change it before approval; it is valid for 30 days.

    • One host, one emailIf a host runs several agents, a host CPU, memory or disk problem counts once.
    • Gone or hiddenThe event says why the problem went away: fixed, acknowledged by the administrator, or became info.
    • Approval is a new settings versionThe threshold goes to the agent by normal delivery, on behalf of the administrator. Each approval and refusal is in the audit log.
    • Link in the emailThe email names the host, gives a hint and a link to the server in HQ Monitor.

    The center administrator turns on email and webhook. Until they are set up, no emails go out. Events and reports are visible in the center panel and in its API.

    Users · rights · audit · languages

    Users and rights

    Each user sees only their own servers. Rights to make changes are granted separately, one by one.

    The administrator sees all servers and manages users. A regular user sees the servers from their list. A new right or a new list takes effect on the next page load; a new password signs the user out of all browsers.

    New user. Type, three rights and a server list.
    RightWhat it allows
    Send settingsSend changed settings to the agent through the center.
    Run jobsStart and cancel a backup, restore, sweep or index recalculation (Run now).
    Jobs that replace a databaseStart a restore that replaces the working database. Only together with the run right, and only if the agent also allows it.
    • Audit logEach save, send, password change and job start is written with the user login. In goafts, a hash chain protects the log.
    • Agent request logThe agent itself writes a line for each request that changes the server and for each access refusal: Local API, commands from the center, Run now, settings changes, updates. Passwords and request bodies do not go into the log. The file goes to the center together with the logs.
    • HQ Monitor loginSession lasts 7 days. No more than 8 login attempts from one address in 15 minutes.
    • Center admin panelA separate login, only with an administrator certificate. Session lasts 8 hours. After 5 failed attempts in 15 minutes, login is closed for 15 minutes.
    • Up to 2000 usersEach has their own password and server list; search by server when you assign servers.
    • 5 interface languagesEnglish, Russian, German, Portuguese (Brazil), Spanish. Statistics reports also use the chosen language.

    ← Back to the HQbird 2027 overview · Server health in a minute →