Library

How Firebird Database Encryption Works

(c) Alexey Kovyazin, IBSurgeon, Alex Peshkoff, Firebird Project, 2021

The article is based on the materials of the workshop "Database Encryption" at Firebird Conference 2019 in Berlin, Germany. It describes how Firebird database encryption works, on the server-level, on the client-side, how to configure database encryption, how to use it from the various types of applications (Delphi, Java, .NET). The article's examples are based on IBSurgeon Firebird Encryption Plugin Framework (FEPF) but can be adapted to the majority of currently available implementations of encryption plugins.

Contents:

1. Why we need database encryption (and when we don't)?
2. How Firebird database encryption works on the server-side
   1. What part of the database is being encrypted?
   2. When data pages are being encrypted?
   3. How to protect the key transfer?
3. How Firebird encryption works on the client-side
   1. Native applications
   2. Java applications
   3. .NET applications
4. Installation and Configuration
5. How to track encryption progress
6. Summary

1. Why we need database encryption (and when we don't)?

In this article, we consider the internals of database encryption on the basic level, to give developers of Firebird applications a better understanding of how database encryption works.

1. To protect databases with sensitive/valuable data from "physical" stealing. If an intruder steals the disk with a copy of the encrypted database or somehow obtains a copy of a database file, it will be not possible to read data from it without an appropriate key, as well as it will be not possible to use recovery software like FirstAID to extract the data. Of course, it depends on the encryption algorithm and computing power, but cracking AES256 will require too long time or too expensive computing resources.
2. To protect the database from access from non-authorized applications without encryption keys. Examples are
   • direct access with a developer tool by a non-authorized person, to change the sensitive information (e.g., money transactions),
   • changing or stealing business logic (texts of stored procedures and triggers).
3. Protect databases with pre-filled data from exporting to or accessing by non-authorized applications.
4. Governments recently introduced data protection laws (GDPR/DSVGO in Europe, LGPD in Brazil, etc) which require, among other things, a higher level of protection for personal and other sensitive data, and encryption is mentioned as one of the adequate protection measures.

In some cases it is better to use the security and configuration capabilities of Firebird instead of database encryption:

• To protect the database from physical access over the network, it is necessary to configure network access: i.e., close network shared folders, because Firebird does not require network shared access to database files, and tighten security permissions (for Linux, for example, database files must have read-write access only for user “firebird”).
• To restrict access to the specific database for the specific subset of users, the easier solution will be to configure a separate security database.
• To restrict access to the database objects (tables, stored procedures) it is necessary to use Firebird security mechanisms: users, roles, etc

2. How Firebird database encryption works on the server-side

2.1. What part of the database is being encrypted?

Figure 1. Database pages types

Some pages are designed to store users' data, and some others are needed to store system information, like transactions and page inventory pages (more details about database pages are available here).

When Firebird database is being encrypted, only pages with user data are being encrypted: data pages, indices, generators, and BLOBs:

Figure 2. Only database pages with users’ data are encrypted

Please note, that database metadata (stored procedures, tables, views, triggers, generators names, etc) do not differ from “users’ data” inside the part of the engine responsible for the encryption, and they are encrypted.

Why system pages are not being encrypted? For performance reason, mostly, and due to the fact that they don’t contain sensitive data which require protection.

The database header page is not encrypted, because it contains information needed for encryption (for example, key name).

2.2. When data pages are being encrypted?

Let’s consider the details of this process:
​

Usually, the process starts with series of reads of database pages from a database file, and they are being cached in the Operating System file cache.

Firebird also can be configured to bypass file cache and use only the own cache, but by default, file cache is used.

After that, Firebird reads pages and puts them into the Firebird Page cache (it is defined by DefaultDBCachePages parameter in firebird.conf and/or databases.conf or in the database header page).

Then, pages from the cache are being selected to the result set of the particular SQL statement (SELECT in our example).

The figure below shows details:

So, database pages are encrypted in OS file cache but arrive at Firebird page cache non-encrypted, and vise versa.

The part of Firebird software responsible for encryption/decryption is called "encryption plugin". Since the majority of plugin implementations (known to authors) are called DbCrypt, we will refer to it as DbCrypt.

On the figure below you can see variants for Windows (DbCrypt.dll) and Linux (libDbCrypt.so):

Figure 5. Encryption plugin (DbCrypt) is doing encryption/decryption

If you look at this picture long enough, the next question will appear pretty quickly: how DbCrypt gets the proper key to encrypt/decrypt database pages?

The answer – there is another plugin for key management.

The typical name for key management is KeyHolder, which serves as a key storage/management facility for the encryption plugin (DbCrypt). KeyHolder implements the interface for key management which is used by DbCrypt.


Figure 6. DbCrypt and KeyHolder
 

What does “key management” means?

In the simplest case, DbCrypt can read keys from the file on the server. The file can be a simple plain text file, which can be hidden in the “secret” place or on a USB stick, or can be an encrypted file (using Windows Crypto API, for example, or with a built-in internal key).

The key file can contain several keys, stored, for convenience, as named list, and can look like this (the example below is taken from the IBSurgeon encryption plugin framework):

Key=Red 0xec,0xa1,0x52,0xf6,0x4d,0x27,0xda,0x93,0x53,0xe5,0x48,0x86,0xb9,0x7d,0xe2,0x8f,0x3b,0xfa,0xb7,0x91,0x22,0x5b,0x59,0x15,0x82,0x35,0xf5,0x30,0x1f,0x04,0xdc,0x75,
Key=Green 0xab,0xd7,0x34,0x63,0xae,0x19,0x52,0x00,0xb8,0x84,0xa3,0x44,0xbd,0x11,0x9f,0x72,0xe0,0x04,0x68,0x4f,0xc4,0x89,0x3b,0x20,0x8d,0x2a,0xa7,0x07,0x32,0x3b,0x5e,0x74,

Database header page information:
....
Creation date    Jan 11, 2017 15:12:20
Attributes       force write, encrypted, plugin DBCRYPT

Variable header data:
 Crypt checksum: MUB2NTJqchh9RshmP6xFAiIc2iI=
 Key hash:       ask88tfWbinvC6b1JvS9Mfuh47c=
 Encryption key name:    RED
 Sweep interval:         0
*END*

Figure 7. Multiple keys for multiple databases on the same server (i.e., Firebird instance)

Choosing the correct key
Often developers ask the question "How does the plugin recognize which key is for what database?" The answer is pretty simple: by the key name, which is stored on the header page of the database.
Less often, but still important question - what if the key name will be as it is recorded in the header, but the key value is different? To prevent page read errors due to the wrong key, DbCrypt plugin stores the encrypted test sequence (digits 0...F) on the header, and then the key is activated, the plugin tries to encrypt the sample data with a key and compare its hash with the stored result, to ensure that passed key value is actually correct for this particular database.

The approach when encryption plugin reads keys directly, is simple and beneficial for debugging, performance tests, etc, because it implements encryption in a transparent way: i.e., client applications and developers tools do not know that database is encrypted.

However, in reality, we need to restrict access of applications to the database: only client application which has a key should be able to connect to the encrypted database.
For this, we need the KeyHolder plugin, to get the key from the client application (which usually is on another computer) through the Firebird network protocol.

2.3. How to protect the key transfer?

It is possible that we want to protect the database in the situation when the customer decides to get direct access to the encrypted database, bypassing the authorized applications (many vendors want to restrict data from the direct access, either read-only or read-write).

It is equivalent to the situation when an intruder has the access to the server but doesn’t have keys.

Let’s consider the following attack scenarios to intercept keys on the server-side:

1) When the intruder creates the fake encryption plugin (DBCrypt.dll) and puts it on the server, and when KeyHolder passes the key, fake DbCrypt makes the dump of the key:

Figure 8. Attack with fake DbCrypt.dll

2) When the intruder creates the fake firebird.exe file and makes the dump with it:

Figure 9. Attack with fake firebird.exe

In order to protect from such attacks, the good implementation of encryption and key management plugins need to protect key exchange.

Key exchange can be protected with asymmetric encryption with the pair of public/private keys.

These keys are generated during the build process and built-in for the specific pair of encryption and key management plugins. For the best protection, it is necessary to use specially built pairs of DbCrypt/KeyHolder.

When DbCrypt and KeyHolder exchange the keys, they use the following protocol (it is simplified, but the idea is clear, I guess):

DbCrypt → KeyHolder: 
	Give Me The Database Key With this Salt
KeyHolder:
	Encrypts DbKey With Public key using salt from DbCrypt
	Transfers Encrypted DbKey to DbCrypt
DbCrypt:
	Decrypt DbKey With Private Key
	Validate salt correctness
	Ready To Work

Please note that encryption and correctness of transferred keys depend on the plugin implementation, the Firebird engine provides only basic low-level transfer service “send N bytes from this plugin instance to that plugin instance”.

Summary for the server-side part of the encryption

• Encryption/decryption is done by database encryption plugin (DbCrypt), page by page, during data exchange between Operating System file cache and Firebird page cache
• Key management can be implemented in a simple way when DbCrypt reads keys directly, but usually, it is done with key management plugin (KeyHolder)

3. How Firebird encryption works on the client-side

3.1. Native applications

Please note: from here and below, “native” means that such application establishes a network connection with the server using fbclient.dll, usually, such app is built in Delphi, C++, PHP. Unlike native applications, Java and .NET implement their own version of the protocol, they will be considered below.

Connection process:

1. Client application loads client library
   1. fbclient.dll – native Windows apps
   2. libfbclient.so - native Linux apps
2. Client application initiates a connection, sending
   1. Username, e.g. SYSDBA
   2. Password, e.g., masterkey
   3. Path/alias of the database

It is important to say that passing key should be done before the regular connection, due to the fact that data pages with metadata, including database owner name, charset, etc, are encrypted.

So, it leads us to the following:

1. Additional network roundtrip is necessary to pass the key before the regular connection
2. Key transfer from the client application to Firebird requires coding with the usage of asymmetric encryption and implementation of callback interface, it can be pretty complex. To simplify this task, plugin vendors provide an example of the code to connect, or, like in the IBSurgeon plugin framework, create the additional library fbcrypt.dll/libfbcrypt.so, which implements the easy to use suitable interface to transfer keys from the client application.

In BeforeConnect event handler:

fbcrypt_init(PAnsiChar(‘C:\Firebird30\fbclient.dll’));
fbcrypt_key(‘RED’, ‘0xec,0xa1,0x52,0xf6,...’));
fbcrypt_callback(nil);

 // Then connect as usual
Database1.Active:=True;

Figure 10. Connection process to the encrypted database for native applications

What about thread safety in the case of multi-thread client applications?

Let me remind you of some general moments of implementation of multi-thread client applications.
Since Firebird 2.5, several threads inside the application can safely use the single attachment to the database,  because all necessary synchronization is done inside the fbclient.dll. 
However, in this case, threads will be able to work with the attachment only one by one. 

This is Ok for the applications which do not require high-performance data exchange with the database — if it is not a problem to wait to perform SQL query from the one thread when another thread is executing another SQL, it is easier to use a simple model when 1 attachment is shared between several threads. 

If the application requires to execute SQL queries in parallel (i.e., it is a full-scale client application), it is better to use a separate thread per each connection.

The situation with the key exchange for the attachments to the encrypted databases is a bit more complex. 

On every attachment, the client library transfers the keys from the client, but it is not directly related to the threads in the client application, the situation depends on the API used.
As you know, the Firebird client library since 3.0 offers 2 types of API: new Object-Oriented API, based on concept of providers, and legacy isc_ API, implemented as a workaround, to maintain compatibility with old Firebird drivers.
If the new object-oriented client API is used, it is enough to create the provider, supply it with the necessary keys, and then use it for the new attachments.
If isc_ client API is used, for every attachment client library will create its own temporary provider, which is not directly visible or accessible by the end-user.
In this case, the key is transferred exactly from the thread where isc_attach_database is invoked, and thread local storage is used to store that key.

In practice, since almost all client libraries use isc_ API (at the moment, among popular drivers, only Python driver uses OO API),  it is necessary to invoke fb_database_crypt_callback() in each thread that connects to the encrypted database.

Key transfer calls (fbcrypt.dll calls in FEPF example) must be done before the connection, in the same thread where the connection will be established.

When we work with many databases (for example, SaaS web server with many clients databases), it is important to remember, that every invocation of fbcrypt_key() adds a key to KeyHolder storage, associated with the current connection. 

Keys values must be set before the connection, after the attachment, the key value cannot be changed.

In case of detaching, keys are not unloaded, they will be kept in memory till the unload of fbcrypt.dll.

3.2. Java applications

From Jaybird 4 Readme:

“The current implementation is simple and only supports replying with a static value from a connection property. Be aware that a static value response for database encryption is not very secure as it can easily lead to replay attacks or unintended key exposure.

Future versions of Jaybird (likely 5) will introduce plugin support for database encryption plugins that require a more complex callback.”

Practically, it means that we need to set value for encryption callback (normally, it is a key name and key-value pair) in the connection property dbCryptConfig.

For example:

 edConnectionString.setText("jdbc:firebirdsql://localhost/g:/Databases/ODS12/crypt.fdb?lc_ctype=utf8&dbCryptConfig=MyKey:0xec,0xa1,0x52,0xf6,0x4d,0x27,0xda,0x93,0x53,0xe5,0x48,0x86,0xb9,0x7d,0xe2,0x8f,0x3b,0xfa,0xb7,0x91,0x22,0x5b,0x59,0x15,0x82,0x35,0xf5,0x30,0x1f,0x04,0xdc,0x75,");

In IBSurgeon implementation of key management plugin such transmission of the key is considered as more or less unsafe: if the network protocol encryption is not enabled (btw, to enable it, set in firebird.conf WireCrypt=Required and don’t use legacy authentication), the key can be easily spotted with network traffic analyzer like WireShark, so, in order to enable transmission of keys in this way, it is necessary to set UnsafeClient=true in KeyHolder.conf of IBSurgeon key management plugin.

3.3 .NET applications

.NET example of the connection string for encrypted databases:

string connectionString =
                "User=SYSDBA;" +
                "Password=masterkey;" +
                "Database=G:\\Databases\\ODS12\\CRYPT.FDB;" +
                "DataSource=localhost;" +
                "Port=3053;" +
                "Dialect=3;" +
                "Charset=NONE;" +
                "Role=;" +
                "Connection lifetime=15;" +
                "Pooling=true;" +
                "MinPoolSize=0;" +
                "MaxPoolSize=50;" +
                "Packet Size=8192;" +
                "ServerType=0;" +
                 "cryptkey = TXlLZXk6MHhlYywweG…...;";

"MyKey:0xec,0xa1,0x52,0xf6,0x4d,0x27,0xda,0x93,0x53,0xe5,0x48,0x86,0xb9,0x7d,0xe2,0x8f,0x3b,0xfa,0xb7,0x91,0x22,0x5b,0x59,0x15,0x82,0x35,0xf5,0x30,0x1f,0x04,0xdc,0x75,"

and use it as param for "cryptkey=xxx;" with ";" at the end of the connection string.

4. Installation and Configuration

 KeyHolderPlugin = KeyHolder 

myencrypted = C:\Temp\EMPLOYEE30\EMPLOYEE30.FDB { KeyHolderPlugin = KeyHolder } 

    • DbCrypt.dll
    • DbCrypt.conf
    • KeyHolder.dll
    • KeyHolder.conf – for debug mode only!

    • fbcrypt.dll
    • libcrypto-1_1-x64.dll
    • libssl-1_1-x64.dll
    • firebird.msg

isql.exe localhost:C:\Temp\EMPLOYEE30\EMPLOYEE30.FDB -user SYSDBA -pass masterkey
SQL>alter database encrypt with dbcrypt key red;
SQL> show database;
Database: localhost:C:\Temp\EMPLOYEE30\EMPLOYEE30.FDB
….
ODS = 12.0
Database encrypted
Default Character set: NONE

alter database encrypt with "DbCrypt" key Red;

For this, we must put in the folder with client app the following files:

• Demo app from FEPF – CryptTest.exe (32bit)
• Mandatory files:
  • fbclient.dll
  • fbcrypt.dll
  • libcrypto-1_1.dll
  • libssl-1_1-x64.dll
• Optional files:
  • firebird.conf
  • in plugins
  • ◦ KeyHolder.conf
  • ◦ keyhodler.dll

It enables the transparent work of Firebird developers tools (like Firebird SQL Studio, DatabaseWorkbench, IBExpert, FlameRobin, RedExpert, etc), and transparent usage of Firebird command-line tools (gfix.exe, nbackup.exe, etc). 

5. How to track encryption progress

In order to track the encryption process, run either SQL query from MON$:

select mon$crypt_page * 100.0 / mon$pages as Percent from mon$database; 
commit;

gstat -e dbname

Database "D:\ENCDB\TESTENCRYPT.FDB"
Gstat execution time Tue Mar 16 11:45:11 2021

Database header page information:
        Flags                   0
        Generation              10697
        System Change Number    3
        Page size               8192
        ODS version             12.0
        Oldest transaction      7053
        Oldest active           7054
        Oldest snapshot         7054
        Next transaction        7054
        Sequence number         0
        Next attachment ID      17834
        Implementation          HW=Intel/i386 little-endian OS=Windows CC=MSVC
        Shadow count            0
        Page buffers            0
        Next header page        0
        Database dialect        3
        Creation date           Oct 9, 2019 6:42:31
        Attributes              encrypted, plugin DBCRYPT

    Variable header data:
        Database backup GUID:   {866B4967-ED58-427E-A481-DB9206CEA2ED}
        Crypt checksum: MUB2NTJqchh9RshmP6xFAiIc2iI=
        Key hash:       ask88tfWbinvC6b1JvS9Mfuh47c=
        Encryption key name:    RED
        Database GUID:  {323FE494-1771-4608-E99D-C1B69C84578B}
        *END*

Data pages: total 105, encrypted 105, non-crypted 0
Index pages: total 95, encrypted 95, non-crypted 0
Blob pages: total 0, encrypted 0, non-crypted 0
Generator pages: total 1, encrypted 1, non-crypted 0
Gstat completion time Tue Mar 16 11:45:11 2021

6. Summary

1. Firebird database encryption is a powerful feature to protect the information in the databases from non-authorized access.
2. The encryption process requires a server-side dynamic library - the encryption plugin (usually called DbCrypt), and, in the vast majority of situations, the key management plugin (usually it is called KeyHolder).
3. The secure and reliable implementation of DbCrypt and KeyHolder plugins should be done with the most popular attack types in mind.
4. In order to work with the encrypted database, client applications must transfer the encryption key.
5. Installation and configuration of encryption plugin on server-side is trivial, it requires 1 parameter in the firebird.conf/databases.conf, and several files.
6. The encryption process can be lengthy, it is done in the separate background thread, the progress can be progress with MON$ call or gstat.

What next?

Contact us: