Library

How to collect dumps for Firebird on Linux and generate stack traces

Introduction

In case of serious problem with Firebird, like hanging or abnormal termination, it is necessary to configure dump collection and then create stack traces. With dumps and stack traces, Firebird core developers of IBSurgeon can analyze the detailed picture of the hanging/failure and give the solution.

Dump and stack traces analysis is available as a part of advanced Firebird technical support service from IBSurgeon. Our plans start from USD199/server/month, and include support through chat (WhatsApp, Telegram, etc), so in case of a serious problem you will have professional help almost immediately.

For details, visit: https://ib-aid.com/en/firebird-support-service/
All actions below require root or sudoer.

1. Prerequisites

1.1 Delete or Stop Interfering Packages

Ubuntu/Debian

  • systemd-coredump

    apt-get purge --auto-remove systemd-coredump

To verify deletion:

cat /proc/sys/kernel/core_pattern

There should be no output like:

/lib/systemd/systemd-coredump%P %u %g %s %t %e

  • apport

    # Search
service --status-all | grep -m1 -i apport

# Mock deletion
apt-get -s remove apport

# Delete
sudo apt-get remove apport

# Clean remaining files
sudo apt-get purge apport

CentOS/RedHat

  • abrtd

To check:

cat /proc/sys/kernel/core_pattern

If you see output like:

/usr/libexec/abrt-hook-ccpp %s %c %p %u %g%t e %P %I

Stop and delete abrtd:

For CentOS 6:

service abrtd stop

For CentOS 7:

systemctl disable abrtd.service

Remove:

yum -y remove abrt*

# Clean remaining files
rm -r /var/cache/abrt/

# Verify cleanup
cat /var/log/yum.log | grep abrt

1.2 Install GDB

GDB is required to create stack traces from collected dumps. GDB installation does not affect performance or put Firebird in "debug" mode.

For CentOS:

yum install gdb

For Ubuntu/Debian:

apt-get install gdb

2. Remove Dump Size Limitations

Check current dump size limit:

ulimit -c

If you see 0, dump creation is prohibited.

For systems with systemd, modify the service file:

CentOS/RedHat
/usr/lib/systemd/system/firebird-superserver.service
Ubuntu/Debian
/lib/systemd/system/firebird-superserver.service

Add to [Service] section:

LimitCORE=infinity

Restart services:

systemctl daemon-reload
systemctl restart firebird-superserver.service

Verify unlimited dump size:

strings /proc/$(pgrep firebird)/limits

3. Configure Dump File Location and Naming

This step is optional but highly recommended to specify meaningful dump file names and ensure adequate storage space.

Temporary Configuration

sysctl -w kernel.core_pattern=/path/to/dumps/core.%e.%t.%p

Permanent Configuration

Add to /etc/sysctl.conf:

kernel.core_pattern=/path/to/dumps/core.%e.%t.%p

Set Ownership

chown firebird /path/to/dumps -R

4. Test Dump Collection

Prerequisites

  1. Restart SSH

  2. Verify ulimit -c returns "unlimited"

Test Crash Commands

Firebird 3+ (SuperServer or SuperClassic)
echo "shell kill -6 $(pgrep firebird);" | /opt/firebird/bin/isql -q localhost:employee -user sysdba -pas masterkey
Firebird 2.5 (SuperClassic)
echo "shell kill -6 $(pgrep fb_smp_server);" | /opt/firebird/bin/isql -q localhost:employee -user sysdba -pas masterkey
Classic Mode
select mon$server_pid from mon$attachments where mon$attachment_id = current_connection;
shell kill -6 <found_PID>;

5. Install Debug Files

  • Debug files must match your Firebird build version

  • Installing debug files does not affect performance

  • Update debug files when updating Firebird

Expected debug file locations:

/opt/firebird/bin/.debug/firebird.debug
/opt/firebird/bin/.debug/gbak.debug
/opt/firebird/bin/.debug/isql.debug
/opt/firebird/lib/.debug/libfbclient.so.3.0.11.debug
/opt/firebird/lib/.debug/libib_util.so.debug
/opt/firebird/plugins/.debug/libfbtrace.so.debug
/opt/firebird/plugins/.debug/libEngine12.so.debug
HQbird installations include correct debug files and handle updates automatically.

6. Create Stack Traces from Running Processes

  1. Create gdb-commands.txt:

thread apply all bt
quit
yes

  1. Generate stack trace:

gdb -q -x ./gdb-commands.txt /opt/firebird/bin/firebird $(pgrep firebird) 1>./stacktrace.txt 2>&1

7. Create Dump from Running Process

gcore -o /path/to/dumps/firebirdprefix $(pgrep firebird)
The tool will append the process ID to the prefix.

8. Create Stack Trace from Existing Dump

gdb -q -x ./gdb-commands.txt /opt/firebird/bin/gfix /tmp/core.gfix.23041.1559703551 1>./gfix.trace1.txt 2>&1

9. Deep Dump Analysis

To analyze shared libraries:

echo -e "info sharedlibrary\nquit\n" | gdb