bfl_util - Big File Utilities

The BFL_UTIL package provides a high-level interface for reading and writing logical files greater than 2GB in size (the typical UNIX limit on physical file size).

A logical "big file" is broken down into multiple physical files, each less than the maximum physical file size:

                             basePathname_bfl0
                             basePathname_bfl1
                             basePathname_bfl2
                             ...
                             basePathname_bflN

When the size of an individual file reaches the maximum allowable size (a configurable parameter), the next file in sequence is created. For example, suppose a logical write to xyz begins in its last physical file, xyz_bfl22. If xyz_bfl22 fills up before the write is complete, then xyz_bfl23 is created and the write continues.

The following code fragment creates a 7GB logical file filled with over a billion "Hello!" records:

    #include  "bfl_util.h"		-- Big file utilities.
    BigFile  file ;
    int  i ;
    ...
    bflOpen ("pathname", "-write", &file) ;
    for (i = 0 ;  i < 1*1024*1024*1024 ;  i++)
        bflWrite (file, strlen ("Hello!") + 1, "Hello!", NULL) ;
    bflClose (file, NULL) ;

and the following code fragment counts the number of records in the file:

    char  buffer[] = "Hello!" ;		-- Sized to hold a "Hello!" record.
    int  numRecords = 0 ;
    ...
    bflOpen ("pathname", "-read", &file) ;
    while (!bflRead (file, sizeof buffer, buffer, NULL))
        numRecords++ ;
    bflClose (file, NULL) ;

You can seek to an arbitrary position in the logical file as follows:

    QuadWord  position = qwCreate (0, 1234) ;
    ...
    bflSetPos (file, position) ;	-- Seek to offset 1234 in file.

Removing a logical file deletes all of its physical files:

    bflRemove ("pathname") ;

Public Procedures

bflClose() - closes a big file.

bflGetPos() - gets the current I/O position in a big file.

bflOpen() - opens/creates a big file.

bflRead() - reads data from a big file.

bflRemove() - removes a big file.

bflSetPos() - sets the current I/O position in a big file.

bflWrite() - writes data to a big file.

Source Files

bfl_util.c

bfl_util.h