Library

InterBase and Firebird Recovery Guide

NOTICE: This document is the chapter from the book "The InterBase World" which was written by Alexey Kovyazin and Serg Vostrikov.

The chapter from book "The InterBase World" devoted to the database repairing.

1. The history of this guide

The Russian book "The InterBase World" was published in September, 2002.
Its pressrun was 3000 copies. After 3 months it was sold out and second, improved edition was published in April, 2003 with pressrun 5000 copies.
Now it is on top in the largest Russian online-bookstores and we intend that it will sold out very soon.

The authors of the book are Alexey Kovyazin, developer of IBSurgeon and
well-known Russian InterBase specialist, and Serg Vostrikov, CEO of Devrace
company www.devrace.com
It’s a funny thing, not a single book devoted to InterBase was published in English!

Thousands and thousands of developers use InterBase and Firebird, discuss the
topic in various conferences (take a look here: Links).
The community of Interbase developers averages out to tens of thousands of
people. The strong demand on the InterBase books in various countries proves the InterBase community is really big.

We can stake the case of beer on the fact that the edition of 10000 copies will be swept away from Amazon.com in a month. But people in publising companies
"knows everything" and they are sure that nobody buy book about InterBase. Its areal pity.

Here we'd like to offer you the draft of one chapter of this book devoted to recovery of InterBase/Firebird databases.
 

2. How to recover InterBase/Firebird database

2.1. Review of main causes of database corruption
Unfortunately there is always a probability that any information store will be
corrupted and some information from it will be lost. Database is not an exception to this rule. In this chapter we will consider the principal causes that lead to InterBase database corruption, some methods of repairing databases and extracting information from them. Also we will get to know the recommendations and precautions that will minimize a probability of information loss from database.
First of all, if we speak about database repairing we should clarify a notion of
“database corruption”. Database is usually called damaged if trying to extract or modify some information errors appear and /or the extracting information turns out to be lost, incomplete or not right at all. There are cases when database corruptions are hidden and are found only by testing with special facilities, but there are also real database corruptions when it is impossible to be connected to the database, when adjusted programs-clients show strange errors (when no manipulations were executed with database), or when it is impossible to restore the database from backup copy.

2.2. Principal causes of database corruption are:
  • Abnormal termination of server computer, especially electric power interruption.For IT- industry it is a real lash and that is why we hope there is no need to remind you once again about the necessity of having a source of uninterrupted power supply on server.
  • Defects and faults of server computer, especially HDD (hard disk drive), disk controllers, main memory of the computer and cache memory of Raid controllers.
  • Not a proper connection string with a multi-client database of one or more users (in versions prior to 6.x ) When connecting via TCP/IP, the path to database must be pointed server name: drive:/path/databasename /for servers on UNIX platform servername: /path/databasename /, according to NETBEUI protocol \\servername\drive:\path\databasename. Even when connecting database from the computer, on which database is located and server is running one should use the same line renaming servername for localhost. One cannot use mapped drive in the connection line. If you break one of these rules, server considers that it works with different databases and database corruption is guaranteed.
  • File copy or other file access to database when server is running. The execution of the command “shut-down” or disconnecting the users in usual way is not a guarantee that server is doing nothing with the database, if sweep interval is not set to “0”, garbage collection can be executed. Generally the garbage collection is executed immediatelly after the last user disconnects from database. Usually it takes several seconds, but if before it many DELETE or UPDATE operations were committed, a process may be  longer.
  • Using unstable InterBase server versions 5.1-.5.5.Borland Company officially admitted that there were several errors in these servers and a steady upgrate 5.6 removed only after an output of certified InterBase 6 was in free-running mode for all clients of servers 5.1-5.5 on its site.
  • Exceeding size restriction of database file (not a database!). For pre-InterRase 6 versions and some InterBase 6 beta database file limit is 4Gb, for InterBase 6.5 and all releases of Firebird (1.0, 1.5, 2.0, 2.1) - 32Tb. When approaching database size to a limit value, an additional file must be created.
  • Exhaustion of free disk space when working with database.
  • For Borland InterBase servers versions under 6.0.1.6 - exceeding of restriction to a number of generators according to Borland InterBase R & D defined in thefollowing way (look table 1).
    Version Page size=1024 Page size=2048 Page size=4096 Page size=8192
    Pre 6 248 504 1016 2040
    6.0.x 124 257 508 102
Table 1: Critical number of generators in early InterBase versions

• For all Borland InterBase servers – exceeding of permissible number of
transactions without executing backup/restore. One can know the number of
transactions happened in database from the time of last creation by invoking the utility gstat with a key – h- parameter NEXT TRANSACTION ID will be desired
quantity of transactions. According to Ann W.Harrison the critical number of
transactions depends on page size, and has the following values (look table 2):   
Database page size Critical number of transactions
1024 byte 131 596 287
2048 byte 265 814 016
4096 byte 534 249 472
8192 byte 1 071 120 384

Table 2: Critical number of transactions in Borland InterBase servers

Constraints of Borland InterBase servers enumerated above are not applied to
Firebird servers except for the earliest versions 0.x., existence of which has already become a history. If you use the final version Firebird 1.0 or InterBase 6.5-7.x, you should not be worried about points 5, 6, 8 and 9 and you should concentate your efforts on other causes. Now we will consider the most frequent of them in detail.

2.3. Power supply failure
When shutting-off the power on server, all the activities of data processing are
interrupted in the most unexpected and (according to Murphy’s law) dangerous
places. As a result of it the information in database may be distorted or lost. The simplest case is when all uncommitted data from client’s applications were lost as a result of emergency server shutdown. After power-fail restart server analyzes the data, notices incomplete transactions related to none of the clients and cancel all the modifications made within the bounds of these «dead» transactions. Actually such behavior is normal and supposing from the beginning by InterBase developers.

However power supply interruption in not always followed by such insignificant
losses only. If server was executing database extension at the moment of power supply interruption, there is a big probability of having orphan pages in database file (pages that are physically allocated and registered on page inventory page (PIP) , data writing on which is impossible). If you want to know more about orphan pages look chapter «The structure of InterBase database».
Only the tool of repairing and modification gfix (we will consider it below) is able to fight with orphan pages in database file. Actually orphan pages lead to unnecessary expense of disk space and as such are not the cause of data loss or corruption.

Power loss leads to more serious damages. For example, after shutting off the
power and restarting a great amount of data, including committed, may be lost (after adding or modification of which a command «commit transaction» was executed). It  happens because confirmed data are not written right to database file on disk. And file cache of operating system (OS) is used for this aim. Server process gave data writing command to OS. Then OS assured server that all the data were saved on disk and in reality data were stored in file cache. OS doesn’t hurry to flush these data to disk, because it considers that there is much main memory left and puts off slow operations of writing to disk until main memory is filled.

2.4. Forced writes – cuts both ways
In order to affect the situation tuning of data write mode is provided in InterBase 6. This parameter is called forced writes (FW) and has 2 modes – ON (synchronous) and OFF (asynchronous). FW modes define how InterBase communicates with disk. If FW is turned on, the setting of synchronous writes to disk is switched on, when confirmed data are being written to disk just after command commit, server is waiting for writing completion and only then continues processing. If FW is turned off InterBase doesn’t hurry to write data to disk after the command of transaction commit and delegates this task to parallel thread while main thread continues data processing not waiting until writes are done to disk. Synchronous writes mode is one of the most careful and it minimizes any possible data loss, however it may cause some loss of performance. Asynchronous writes mode increases a probability of loss of great number of data. In order to achieve maximum performance FW Off mode is usually set. But as a result of power interruption much more number of data is lost during the asynchronous writes than synchronous. When setting the write mode you should decide whether a few percents of performance are more significant than a few hours of work if power interruption happens unexpectedly.

Very often users are careless to InterBase. Small organizations save on any trifle, often on computer-server where DBMS server and different server programs (and not only server) are set as well. If they hang-up people thinking for not a long time press RESET (it happens several times a day). Although InterBase is very steady to such activities comparing with other DBMS and allows to start working with database just after emergency reboot, but such use isn’t desired. The number of orphan pages increases and data lose connections among themselves as a result of fault reboots.
It may continue for a long time, but sooner or later it will come to an end. When damaged pages appear among PIP or generators pages or if database header page is corrupted, database may never open again and become a big piece of separate data from which one can’t extract a single byte of useful information.

2.5. Corruption of hard disk
Hard disk corruptions lead to missing of database important system pages and/or corruption of links among the remained pages. Such corruptions are one of the most difficult cases, because almost always they require low-level interference to restore the database.

2.6. Mistakes of database design
It’s necessary that you know about some mistakes made by database developers
that can lead to impossibility of database recovery from a backup copy (*.gbk files created by gbak program). First of all this is a careless use of constraints on database level. Typical example is constraints NOT NULL. Let’s suppose that we have a table filled with the number of records. Now we’ll add to this table using ALTER TABLE command one more column and point that it mustn’t contain non-defined values NULL. Something like this:

ALTER TABLE sometable Field/INTEGER NOT NULL

And in this case there will be no server’s error as it could be expected. This
metadata modification will be committed and we won’t receive any error or warning message that creates an illusion of normality of this situation.
However, if we backup database and try to restore it from a backup copy, we’ll
receive an error message at the phase of restoring (because Nulls are inserted into the column that has NOT NULL constraint, and the process of restoring will be interrupted. (The important note provided by Craig Stuntz - with version InterBase 7.1 constraints are ignored by default during restore (this can be controlled by a command-line switch) and nearly any non-corrupt backup can be restored. It's always a good idea to do a test restore after making a backup, but this problem should pretty much go away in version 7.1. ) This backup copy can’t be restored. If restoring was directed to file having the same name as the existing database (duringrestoring existing database working file was being rewritten) we’ll lose the whole information.

It is connected with the fact that constraints NOT NULL are implemented by system triggers that check only arriving data. During restoring the data from backup copy are inserted into the empty just created tables - here we can find inadmissible NULLs in the column with constraint NOT NULL.

Some developers consider that such InterBase behavior to be incorrect, but other one will be unable to add a field with NOT NULL restriction to the database table.

 A question about required value by default and filling with it at the moment of
creation was widely discussed by Firebird architects, but wasn’t accepted because of the fact that programmer is obviously going to fill it according to the algorithm, rather complicated and maybe iterative. But there is no guarantee, whether he’ll be able to distinguish the records ignored by previous iteration from unfilled records or not.

The similar problem can be caused by garbage collection fault because of setting not a correct path to database (the cause of corruption 3) at the time of connection and file access to database files when server is working with it (the cause of corruption 4) and records whole filled with Null can appear in some tables. It’s very difficult to detect these records, because they don’t correspond to integrity control restrictions, and operator Select just doesn’t see them, although they get into backup copy. If it is impossible to restore for this reason, one should run gfix program (look below), find and delete these records using non-indexed fields as search conditions, after it retry to make a backup copy and restore database from it. In conclusion we can say that there is great number of causes of database corruption and you should always be ready for worst - that your database will be damaged for that or other reason. You also must be ready to restore and save valuable information. And now we’ll consider precautions that guarantee InterBase database security, as well as methods of repairing damaged databases.

2.7. Precautions of InterBase database corruption
In order to prevent database corruption, one should always create backup copies (if you want to know more about backup then look the chapter “Backup and restore”). It’s the most trusted way against database corruption. Only backup gives 100% guarantee of database security. As it’s described above, as the result of backup we can get a useless copy (a copy that can’t be restored), that’s why restoring a base from the copy mustn’t be performed by writing over the script and backup must be done according to definite rules. Firstly, backup must be executed as more often as possible, secondly it must be serial and thirdly, backup copies must be checked for restoring capability.

Often backup means that it’s necessary to make a backup copy rather often, for example once in twenty-four hours. The less data period is between the database backup, the fewer data will be lost as a result of fault. Sequence of backup means that the number of backups must increase and must be stored at least for a week. If there is a possibility, it’s necessary to write backups to special devices like streamer, but if there is not – just copy them to the other computer. The history of backup copies will help to discover hidden corruptions and cope with the error that arose long ago and showed up unexpectedly. One has to check, whether it is possible to restore the received backup without errors or not. It can be checked only in one way - through the test restore process. It should be said that restore process takes 3 times more time than backup, and it’s difficult to execute restore validation every dayfor large databases, because it may interrupt the users’ work for a few ho urs (nightbreak may not be enough).

It wou ld be better if big organizations didn’t save on “matches” and left one computer for these aims.

In this case, if server must work with serious load 24 hours 7 days a week, we canuse SHADOW mechanism for taking snapshots from database and further backup operations from the immediate copy. Backup process and database restoring is described in detail in chapter “Backup and restore”. When creatinga backup and then restoring database from it, recreation of all data in database is happening.  This process (backup/restore or b/r) contributes to the correction of most non-fatal errorsin database, connected with hard disk corruptions, detecting problems with integrityin database, cleaning database from garbage (old versions and fragments of  records, incomplete transactions), decreasing database size considerably.

Regular b/r is a guarantee of InterBase database security. If database is working, then it is recommended to execute b/r every week. To tell the truth, there are some illustrations about InterBase databases that are intensively used for same years without backup/restore.

Nevertheless, to be on a safe side it’s desirable to perform this procedure, especially as it can be easily automated (look chapter “Backup”)
If it’s impossible to perform backup/restore often for some reasons, then one can use the tool gfix for checking and restoring database. gfix allows to check and remove many errors without b/r.

2.8. Command line tool gfix
Command line tool gfix is used for checking and restoring database. Besides, gfix can also execute various activities of database control: changing database dialect, setting and canceling the mode “read-only”, setting cache size for a concrete database and also some important functions (you can know about them in InterBase 6 Operations Guide [4]) gfix is committed in a command line mode and has the following syntax:
Gfix [ options] db name
Options – is a set of options for executing gfix , db name is a name of database over which operations will be performed, defined by set of options. Table 3 represents options gfix related to database repairing:
Option Description
–f[ull]  This option is used in combination with –v and
means it’s time to check all fragments of records
–i[gnore] Option makes gfix ignore checksums errors at
the time of validation or database cleaning
–m[end] Marks damaged records as not available, as a
result of what they will be deleted during the
following backup/restore. Option is used at the
time of preparing corrupted database to b/r.
–n[o_update] Option is used in combination with –v for
read-only database validation without correcting
corruptions
–pas[sword] Option allows to set the password when
connecting database. (Note that it is the error in
InterBase documentation -pa[ssword], but the
shortcut "-pa" will not work - use "-pas" )
–user Option allows to set user’s name connecting
database
–v[alidate] Option presetting database validation in the way
of which errors are discovered
-m[ode] Option setting the write mode for database – for
read only or read/write. This parameter can
accept 2 values – read write or read only.
–w[rite] {sync | async} Option that turns on and off the mode
synchronous/asynchronous forced writes to
database. sync – to turn synchronous writes on
(FW ON); async –to turn asynchronous writes
on (FW OFF);
Table 1: gfix tool options for database restoring

There are some typical examples of using gfix:

gfix –w sync –user SYSDBA –pass masterkey firstbase.gdb

In this example we set for our test database firstbase.gdb synchronous writes mode (FW ON). (Of course, it is useful before the corruption occurs). And below is the first command that you should use to check database after the corruption occurs:
gfix –v –full –user SYSDBA –pass masterkey firstbase.gdb

In this example we start checking our test database (option –v) and indicate that fragments of records must be checked as well (option -full). Of course, it is more convenient to set various options for checking and restoring process by any GUI, but we’ll consider the functions of database recovery using command line tools. These  tools are included to InterBase and you can be sure that their behavior will be the same on all OS running InterBase. It is very important that they always be near.
Besides, the existing tools, allowing to execute database administrating from a
client’s computer use Services API for it, that isn’t supported by InterBase server Classic architecture. That means you may use third party products with server’s architecture SuperServer.

2.9. The repairing of corrupted database
Let’s suppose there are some errors in our database. Firstly, we have to check the existence of these errors; secondly, we have to try to correct these errors. You should abide the following instructions.
You should stop the InterBase server if it’s still working and make a copy of file or database files. All the restore activities should be performed only with database copy, because the chosen way may lead to unfortunate result, and you’ll have to restart a restore procedure (from a starting point). After creating a copy we’ll perform the whole database validation (checking fragments of records).
We should execute the following command for it:
gfix –v – full corruptbase gdb –user SYSDBA - password

In this case corruptbase.gdb – is a copy of damaged database. A command will
check database for any structure corruption and give the list of unsolved problems. If such errors are detected, we’ll have to delete the damaged data and get ready for backup/restore using the following command:
gfix –mend –user SYSDBA –password your_masterkey corruptbase gdb

After committing a command you should check if there are some errors in database left. You must run gfix with options –v –full for it, and when the process is over, perform database backup:
gbak –b –v -ig –user SYSDBA –password corruptbase.gdb corruptbase.gbk

This command will perform database backup (option - b says about it) and we’ll get detailed information about backup process executing (option –v). Error regarding to checksums will be ignored (option - ig) If you want to know more information about the options of command line toolgbak, you can find it in the chapter “Backup and restore” If there are some errors with backup, you should start it in another configuration:
gbak –b –v –ig -g –user SYSDBA –password corruptbase.gdb
corruptbase.gbk

Where option – g will switch off garbage collection during backup. If often helps to solve a problem with backup.
Also it may be possible to make a backup of database, if before it we set database in read-only mode. This mode prevents from writing any modifications to database and sometimes helps to perform backup of damaged database. For setting database to read-only mode, you should use the following command: gfix –m read _only
–user SYSDBA –password masterkey Disk:\Path\file.gdb

After it you should try again to perform database backup using the parameters given above.
If backup was done successfully, you should restore the database from backup
copy. You should use the following command:

gbak –c –user SYSDBA –password masterkey Disk:\Path\backup.gbk
Disk:\Path\newbase,gdb
When you are restoring the database, you may have some problems, especially
when creating the indexes. In this case options –inactive and -one_at_a_time should be added to restore command. These options deactivate indexes in creating from database backup and commit data confirmation for every table.

2.10. How you can try to extract the data from a corrupted database
It is possible that the operations given above will not lead to database recovery.
 
It means that database is seriously damaged or it cannot be restored as a single
whole, or a great number of efforts must be made for it is recovery. For example, one can execute a modification of system metadata, use non-documented functions and so on. It is a very hard, long-lasting and ungrateful work with doubtful chances for success. And if it is possible, try to evade it and use other methods. If a damaged database opens and allows to perform reading and modification operations with some data, you should use this possibility and save the data by copying them to a new base, and “ say god-bye” to the old one for good.

So, before transferring the data from the old database, it’s necessary to create a abase destination. If database hasn’t been changed for a long time, then you can use the old backup, from which metadata can be extracted for creating a database destination. On a basis of these metadata one has to create a data destination and start copying the data. The main task is to extract the data from a damaged database. Then we’ll have to allocate the data in a new base, but it’s not very difficult, even if we’ll have to restore database structure from memory. When extracting data from tables, you should use the following algorithm of operations:
  • At first you should try to execute SELECT* from table N. If it went normallyyou could save the data you’ve got in the external source. It’s better to store data in script (almost all GUI give this function), if only the table doesn’t contain BLOB-fields. If there are BLOB-fields in the table, then data from them should be saved to another database by client program that will play a role of a mediator.Maybe you’ll have to write this trivial program especially for data recovery aims.
  • If you failed to retrieve all data, you should delete all the indexes and try again. Virtually, indexes can be deleted from all the tables from the beginning of restoring, because they won’t be needed any more. Of course, if you don’t have a structure of metadata, same to the corrupted, it’s necessary to input a protocol of all operations that you are doing with a damaged database-source.
  • If you don’t manage to read all the data from the table after deleting the indexes, one can try to do range query by primary key. It means to choose definite range of data. For example:
SELECT * FROM table N WHERE field_PK >=0 and field_PK <=10000 Field_PK
here is a primary key. InterBase has page data organization and that’s why range query of values may be rather effective, although it seems to be something like shamanism. Nevertheless it works because we can expel data from query from damaged pages and read fortunately the other ones. You can recall our thesis that there is no defined order of storing records in SQL. Really, nobody guarantees that not an ordered query during restarts will return the records in the same order, but nevertheless physical records are stored within the database in defined internal order. It’s obviously that server will not mix the records just for abiding SQL-standard. One can try to use this internal order extracting data from damaged database (if you want to know more information about data pages and their correlations, then look chapter “Structure of InterBase database”).

Vitaliy Barmin, one of the experienced Russian InterBase-developers reported that in this way he managed to restore up to 98% information from unrecoverable database (there were a great number of damaged pages). Thus, data from a damaged database must be moved to a new database or into external sources like SQL-scripts. When you copy the data, pay attention to generators values in damaged database (they must be saved for restarting a proper work in new database. If you don’t have a complete copy of metadata, you should extract the texts of stored procedures, triggers, constraints and definition of indexes.

2.11. Restoring of hopeless database
In general, restoring of database can be very troublesome and difficult and that’s why it’s better to make a backup copy of database than restore the damaged data and whatever happened, you shouldn’t get despaired because a solution can be found in the most difficult situations. And now we’ll consider 2 cases.

The first case (a classic problem). A backup that can’t be restored because of having NULL values in the column with constraints NOT NULL (restore process was run over the working file). The working file was erased and restore process was interrupted because of error. And as a result of thoughtless actions we got a great number of useless data (that can’t be restored) instead of backup copy. But the solution was found. The programmer managed to recollect what table and what column had constraints NOT NULL. Backup file was loaded to hexadecimal editor. And a combination of bytes, corresponded to definition of this column, was found there by searching. After innumerous experiments it turned out that constraint NOT NULL adds 1 somewhere near the column name. In HEX-editor this “1” was corrected to “0” and backup copy was restored. After that case programmer memorized once and for all how to execute backup process and restore.

The second case. The situation was catastrophic. Database corrupted on the phase of extension because of lack of disk space. When increasing the database size, server creates series of critically important pages (for example, transaction inventory page and page inventory page, additional pages for RDB$Pages relation) and writes them down to the end of database As a result, database didn’t open either by administration facilities or by utility GBAK. And when we tried to connect database, error message (“Unexpected end of file”) appeared.

When we run utility gfix strange things were happening: The program was working in an endless cycle. When gfix was working, server was writing errors to log (file InterBase log) with high speed (around 100 Kb per second). As a result, log file filled all the free disk space very quickly. We even had to write a program that erased this log by timer. This process lasted for a long time – gfix was working for more than 16 hours without any results. Log was filled up with errors of the following view: “Page XXX doubly allocated”. In starting InterBase sourses (in file val.#) there is a short description of this error. It says that this error appears when the same data page is used twice. It’s obviously that this error is a result of corruption of critically important pages.

As a result, after several days of unfortunate experiments attempts to restore the data in standard ways were left. And that’s why we had to use low-level analysis of data stored in damaged database.

Alexander Kozelskiy, a chief of Information technologies department East View Publications Inc, is the author of the idea how to extract information from similar unrecoverable databases.

The method of restoring that we got as a result of researches was based on the fact that database has page organization and data from every table are collected by data pages. Every data page contains identifier of the table for which it stores data. It was especially important to restore data from several critical tables. There were data from the similar tables, received from an old backup copy that worked perfectly and could be a pattern. Database-pattern was loaded to editor of hexadecimal sources and then we searched for the patterns of those data that interested us. These data were copied to buffer in hexadecimal format and then remains of damaged database were loaded to the editor. A sequence of bytes corresponded to the pattern was found in damaged database, and page was analyzed (on which this sequence was found).

At first we defined the beginning page, but it wasn’t difficult because the sizeof
database file is divisible by data page size. A number of current byte divided by page size – 8192 bytes, approximates the result to integer (and got the number of current page). Then multiplied the number of current page-by-page size and got the number of byte corresponded to the beginning of current page. Having analyzed the header, we defined the type of page (for pages with data the type is 5 – look file ods.h from set of starting InterBase sources and also the chapter “The structure of InterBase Database”) as well as identifier of necessary table.
Then a program was written, that analyzed the whole database, collected all the pages for necessary table into one single piece and move it to file.

Thus, when we got the data we needed in first term, we started analyzing contents of selected pages. InterBase is widely using data compression for saving place. For example, a string like VARCHAR containing “ABC” string, it stores sequence of following values: string length (2 bytes), in our case it is 0003, and then symbols themselves and then checksum. We had to write analyzer of string as well as other database types that converted data from hexadecimal format into ordinary view. We managed to extract up to 80% of information from several critical tables using a “manual” method of analyzing database contents. Later on the basis of experience Oleg Kulkov and Alexey Kovyazin, one of the authors of this book, developed the utility InterBase Surgeon that performs direct access to database, bypassing the InterBase engine and allows to read directly and interpret the data within InterBase database in a proper way.

Using InterBase Surgeon, we manage to detect causes of corruption and restore up to 90% of absolutely unrecoverable databases that can’t be open by InterBase and restored by standard methods.

You can download this program from official program site www.ib-aid.com.

3. Thanks

 I'd like to tender thanks to all who help me to create this guide:
Craig Stuntz, Alexander Nevsky, Konstantin Sipachev, Tatjana Sipacheva and all other kind and knowledgeable people of InterBase and Firebird comminity.
If you have any suggestions or questions about this chapter, please feel free to email.

© 2002 AIexey Kovyazin, Serge Vostrikov.
Copyright © 2004 IBSurgeon Team. All rights reserved.