The Clarion Advisor - Fast ASCII

Published 1999-05-25    Printer-friendly version

ASCII files aren't often used for storing application data for several reasons: they don't have any native support for indexing, you can't change the length of an individual record without rewriting the file, only STRING and GROUP data types are supported, and some functions like PREVIOUS aren't available. As well, anyone with a text editor can alter the contents of the file.

Still, ASCII files have their uses, particularly for importing/exporting data and analysing text files. If you're working a lot with ASCII files there are some tricks you can use to improve performance.

The ASCII driver reads in data one buffer at a time rather than one record at a time. You can specify the buffer size with the FILEDRIVERS buffer string (see Driver Strings in the Clarion help file) where each buffer allocated takes 512 bytes. If you use a driver string of

/FILEBUFFERS=64

you'll be reading in the data in 32k chunks which should give fairly good performance. The larger the buffer, the fewer disk reads. However, to avoid possible record contention the driver will reread the buffer before each record access unless you turn QUICKSCAN (another driver string) on. Another approach is to open the file denying others write access.

A more low-level approach is to use the DOS driver instead of the ASCII driver. The ASCII driver looks for records that end (by default) in CRLF, whereas the DOS driver simply reads in the amount of data you tell it to read. You could read in data in 32k chunks and break it down into records by looking for the CRLF (or whatever record terminator the file uses). If you do this keep in mind that the record terminator will almost never be at the end of your 32k chunk of data, and there may be any number of terminators within that 32k. You could even have one half of the terminator in the end of one block, and the second in the start of the next block. As well, the last chunk of data will almost certainly be less than 32k in length.

If you're willing to go right to the metal then you can use the Windows API to read the file. Allocate a large buffer with GlobalAlloc (say one megabyte) then read data into the buffer with CreateFile and ReadFile in 32 bit, or _lopen and _hread in 16 bit.

Finally, if all of this API magic has you reeling, you could always wait for the next release. The C5B ASCII driver has been tweaked, resulting in a significant increase in performance.

Special thanks to Jim Kane, Arnor Baldvinsson, Gordon Smith, and Scott Ferrett for their suggestions.

Printer-friendly version

Copyright © 1999-2008 by CoveComm Inc. All Rights Reserved. Reproduction in any form without the express written consent of CoveComm Inc., except as described in the subscription agreement, is prohibited.

Clarion Magazine ISSN 1718-9942