DAB - File Manager III

by David Bayliss

Published 1999-07-06    Printer-friendly version

(Part 3 of 3. Click to read Part 1 and Part 2)

Welcome to the third and final installment in the ABC FileManager Design series. In this article I'll get down to the meat of the class; the dictionary interaction and the file access itself. You may want to begin by reading the first and second installments if you haven't already done so.

Record Initialisation and Validation

Whilst initialisation and validation are logically distinct tasks they are grouped together here because in the fullness of time they will form the basis of what is really one concept: business rules. People can use business rules to mean just about anything they like. I use it to mean information about the data that is not contained explicitly within the data.

Some of these rules are already handled by the file driver. For example the DUP attribute in the Clarion language (upon a key) specifies something about the data you cannot get directly from the data (although you may be able to intuit it). The Clarion dictionary gains some of its power by specifying initialisation values and validation values within a single repository. In Clarion 2.003 this information was then scattered (at code generation time) throughout the application. Any form upon a file would contain code to initialise and then validate the record buffer. In ABC this code is contained in the (derived) FileManager object.

One of the ABC aims is to keep the initialisation and validation information in one place so that if the rule is dynamic (i.e. has to be done with an embed point) then that embed only needs to be placed once and all accesses to the data obey the new rule. This is particularly important as ABC was to have edit-in-place browses and automatic updates from drop-combos. In other words, we could no longer rely on forms being around to act as guardians of the file.

A consequence of the heavy dictionary tie to these routines and the need to pander to embed code is that a number of these methods are blank; they are placeholders for template generated code. In these instances I will describe the code that I envisage will go into the derived form of these methods (and which, purely coincidentally, the templates generate.)

CancelAutoInc
PROCEDURE(<RelationManager RM>),VIRTUAL,BYTE,PROC

Initialising a record with an autoincrement key results (in our implementation) in a record being stored on disk to act as a placeholder for the autoincrement value. If you cancel the operation then that record needs to be deleted. The CancelAutoInc is the clue to the FileManager that the record that was initialised is about to be thrown away. The RelationManager parameter is a solution to the problem often referred to as the "orphaned children problem." In a 2.003 application if you insert a record with an autoincrement key, go to the child tab, insert some child records and then cancel the form, the child records persist. Of course you can't see them until you insert a new record and find it automatically gains some children! This is really scary on forms with many children where the child tabs may not even be perused during the insert. ABC gets around this problem in that the form procedure passes the RelationManager in to the CancelAutoInc and the CancelAutoInc undertakes to delete any children (or refuse to cancel the autoincrement) as appropriate.

The implementation is much simpler than the description: if an autoincrement has happened then either DELETE or the RI DELETE are called. In the latter case the response is noted, as an RI relation of restrict means that the autoincrement cannot be cancelled whilst the children persist.

PrimeAutoIncServer PROCEDURE(BYTE HandleErrors),BYTE,PROC,PRIVATE, PrimeAutoInc and TryPrimeAutoInc are really just two virtual hooks on the AutoIncServer. Non-zero for HandleErrors implies the server will take all possible steps to ensure the autoincrement happens. The TryPrimeAutoInc only is called when TryInsert has been called. This is never done with the shipping templates but has been added at the request of a third party.

The routine starts by checking the guard variables. PrimeAutoInc can be called multiple times to allow for the cases where priming is done in the browse or in the form. It also allows for inserts to be done without pre-priming of the record. This is an instance of what I call "objects with attitude." Basically the FileManager knows that priming should be done once, and only once, and thus it does it at the first opportunity it is asked, or at the last minute if no-one asks it!

The algorithm for autoincrementing is essentially the one used to return a unique handle in SaveBuffer, find the last element and add on one. However, as you shall see, some of the little details make things a vast amount more complicated.

The first LOOP is a loop to allow for multiple reruns of the bulk of the procedure in the case where an attempt to autoincrement failed. The failure is most likely to be because between the reading (from disk) of the current highest element, adding on one, and then ADDing the record, another station got there first and ADDed the record with that number! The simplest solution to this failure is therefore to try again from the beginning, and the outer loop encodes that logic. Of course you can't keep doing that for ever because something may really be wrong. If you go to the end of the outer loop, after the ADD(File) you will see logic to trap the error.

If this is the third failure the error manager is invoked to see if the user wants to try again. If he does you simply try again (three times), and if he doesn't you break out with a failure. If the add was successful the method simply notes the autoincrement has been done and then returns Level:Benign (which is zero and means OK).

The second (or inner) loop introduces a new complexity. Since it is possible for there to be multiple autoincrement keys in one file, this code has to find incremented values for each of them. So it loops on each key, executing the body of the loop if it is autoincrement. The SaveBuffer is important because PrimeAutoInc should only change those fields which are tagged in the dictionary as autoincrement (or it won't be possible to support delaying the autoinc allocation until the insert point). The code then splits into two branches, the relatively easy one where the key has one component, and the multi-component case.

One component is handled by fetching the component any variable and assigning to AutoIncField. The file is SET to key order and the final record (or first for a descending key) is fetched. If no record is found (i.e. the file is empty) the new autoinc value is one, else it is the highest value plus one. It's necessary to use AutoIncField/ AutoValue rather than just using the underlying fields as the object will later restore the buffer (which will corrupt the field values) and then perform the autoincrement assignment.

The multi-component case uses exactly the same algorithm. The complexity is in finding the "final" record because you don't want the final record, you want the final record that matches the current record buffer in all components except the last.

ConcatGetComponents is a simple (and ugly) way of snapshotting the leading components of a key to later see if they have changed. The method then clears the minor-most (and thus autoincrement) key high and does a SET(K,K) followed by a Previous or Next as before. In the NoErrorcase it has to check that the record fetched did match in the leading components; if it did then it can use the AI value fetched and add on one, otherwise it knows that no records currently match the major components and thus can use one.

Having computed the new field value (using either method) it restores the buffer contents and then assigns the new autoincrement field value into place. Once that is done that for all autoincrement keys it can try ADDing the new record.

If this procedure looks long and horrible it is because it is. My general rule of thumb is that any procedure more than a page long is a bug. Again you can see engineering and efficiency overcoming science. I could remove the "OneComponent" arm of the IF, and the only effect would be a few more string compares (no big deal), but this also changes this code:

CLEAR(OnlyKeyComponent)
SET(K,K)
PREVIOUS(K)

into this:

SET(K)
PREVIOUS(K)

When you consider that the one component case is the standard third normal form case it was considered that the code verbosity was worth it. That said, the file drivers now spot the above optimisation in most instances so we may be able to simplify this code soon.

PrimeFields PROCEDURE,PROC,VIRTUAL The default implementation of this method is blank; in the derived form the templates insert an assignment for each field that has a non-blank initialisation value in the dictionary. The PrimeFields routine may not assume that autoincrement has been done. Any required blanking will have been performed.

PrimeRecord PROCEDURE(BYTE SuppressClear = 0),BYTE,PROC,VIRTUAL This method is called to prime the record whatever that means. In the current implementation that involves calling PrimeFields to prime the field values and then forcing the autoincrement to prime. Note that this method overrides the attitude built into PrimeAutoInc; when PrimeRecordis called a new autoincrement record will be made. The method has the facility to clear out any fields it doesn't explicitly prime or to leave them alone. This functionality is required to allow the ViewManager to place extra priming information into the record buffer (such as range-limited components of keys) and is controlled by the SuppressClear flag.

The primary case (where AliasedFile &= NULL) is quite straightforward. The interest comes when the file is an alias. Here you don't want to call the priming functions of the alias (because they don't have the required embed code); you want to call them in the "real" file. The code for this has changed in C5EEA; I am describing the new code.

First the "real" file has to be opened on this thread (it may not be), and then the file contents/position have to be snapshotted so that eventually the file can be restored. Now in the case where the clear is to be suppressed the code has to assume that the record (of the alias) contains interesting information that may be required by the field priming or autoincrement. So it has to get the information from the alias into the real file. This is done by the devious device of snap-shotting the alias file buffer and the restoring from the alias FileManager into the "real" file buffer. Having done this it can perform the PrimeRecord on the "real" file; if this is successful it needs to copy the result back to the alias file, done using the save/restore trick again. Finally the "real" file is restored to normality and closed. This may look odd: why restore a file then close it? Simply because Open/Close only increment/decrement counters. The Open only opens if this is the first open, similarly the Close only closes if this is the last close.

ValidateField PROCEDURE(UNSIGNED Id),BYTE,PROC,VIRTUAL This function returns Level:Benign if the field is ok, otherwise it returns an error level. The default implementation only handles the alias case; the actual field validation is handled in a derived method. The Id is the number that would come back from a WHERE statement. The template code simply generates a CASE statement on the field number, it would be possible to produce a more sophisticated version using WHAT. We went for simplicity as this is a very common place to put embed code and also ValidateFieldis hit quite frequently for control by control field validation and therefore performance was an issue.

ValidateFields PROCEDURE(UNSIGNED Low,UNSIGNED High,
    <*UNSIGNED Failed>),BYTE,PROTECTED,PROC,VIRTUAL

This method is simply an encapsulated way of calling a range of ValidateField calls. It simply spools over the field numbers contained within the (inclusive) range. If one fails then the failure number is assigned to Failed. Again the alias case is handled by re-vectoring through the "real" file. It should perhaps be noted that for efficiency the alias code does a SaveBuffer, not SaveFile. This implicitly assumes that the field validation code will not mess with the current file state (i.e. no file I/O will be done on the primary file).

ValidateRecord PROCEDURE(<*UNSIGNED Failed>),BYTE,VIRTUAL Another syntactic short-hand, this simply calls ValidateFields to ensure that every field in the record is validated.

File Driver Replacements

These methods are direct replacements for the equivalents in the file driver. They are generally there to perform advanced error handling or to ensure that other FileManager routines are called at the appropriate moment.

BindFields PROCEDURE,VIRTUAL This method is called at a suitable point to bind the fields. By default the code simply performs a bind on the record buffer. The templates further override this to perform binds on any memos that are available. The call can also be overridden by the user (in C5) to bind logical names (as opposed to labels) as required.

Close PROCEDURE,BYTE,PROC,VIRTUAL The close mechanism (tied to the open mechanism) is designed to avoid the needless opening and closing of files upon a thread. The FileManager therefore maintains a count of the number of times a file has been opened and closed. Upon a close it therefore decrements the counter. If this close has closed the final remaining open then the close is actually performed upon the file. The Used flag denotes if a file was really forced open (by an implicit or explicit UseFile) as opposed to just logically opened. Errors are not trapped by this routine as any real problem with a close will be picked up again when the file comes to be re-opened. For this reason ABC also eschews the TryClose.

Fetch PROCEDURE(KEY K),BYTE,PROC The Fetch routine is really just a wrapper for a file GET. The error case results in the buffer being cleared. Most of the work is done by re-vectoring through TryFetch, and though this doesn't save a great deal of code and is marginally less efficient than inline coding it does result in greater code integrity. Put another way, the code for fetching is in only one place so only has to be fixed in one place.

InsertServer PROCEDURE(BYTE HandleError),BYTE,PRIVATE Insert and TryInsert are just interface maps of this procedure. InsertServer is a fairly good illustration of the difference between the FileManager and the file driver equivalents. It is only really trying to do an add; all of the other code is there either to handle errors or to ensure other ABC methods get called as appropriate.

First UseFileis called. This registers that not only is this file logically open, it needs to be actually open. Then comes a call to ValidateRecord. If the record is not valid then the method returns. Note throughout ABC the fact that Level:Benign is zero is assumed to aid readability and brevity. There are then three cases:

1. No autoincrement keys. In this instance the record will not already exist so a new one can be added
2. There are autoincrement keys and the autoincrement has been pre-primed. In this case the record does exist resulting in a PUT rather than ADD.
3. There are autoincrement keys but they have not been pre-primed. Call the autoincrement logic to ADD the record (remember PrimeAutoIncrementdoes not corrupt the record buffer other than the autoinc components themselves).

There are then three error conditions to worry about:

1. NoError. In this case simply note that any previously primed autoincrementing has now been used and return. (Note this code assumes that PrimeAutoIncrementwill not have left a value in ErrorCode if it was successful; I suspect that technically this is a bug.)
2. DupKey. This is the only error the method attempts to recover from gracefully, stepping through the keys and alerting the end user of any duplications that this record causes.

Everything else. Post a general (cryptic) error message to the user and return.

NextServer PROCEDURE(BYTE HandleError,BYTE Prev),BYTE,PRIVATE Again Next and TryNext are just interfaces to this method. Since C5EEA Previous and TryPrevious have also become interfaces to this routine (the beauty of this method being private!). The NextServer and PreviousServermethods of earlier versions differed only in one line which has now been parameterised with the Prev byte.

There are only two real points of interest. Firstly the BadRecErr sets the EOF flag (see GetEOF). Secondly ABC has a facility whereby a held record error can be treated simply as a Skip rather than as an EOF (which it was in 2.003). You can argue back and forwards for hours as to whether it is better to display a browse with information missing or to abort the display. ABC takes the approach that that decision is best left in the hands of the developer (as the real answer is probably dataset specific) and so provides a property for her to register her decision.

OpenServer PROCEDURE(BYTE HandleError,BYTE IncrementUsage=True,
                     BYTE ForceOpen=False),BYTE,PROC,PRIVATE Open and TryOpen are interfaces to this method. The second and third parameters are really for UseFile. They allow an actual open to be forced without a logical open happening. Specifically, setting increment usage to false means that a corresponding close is not required. ForceOpen is used to force the file open even when the LazyOpen status would suggest the open can be deferred. Note that ForceOpen is quite safe as the routine takes a failure to open because the file is already open as a success. Again most of the work of the routine is in error handling, and in this case some moderately sophisticated recovery is allowed for.

1. NoError or FileOpen. Both treated as a success, internal state variables cleared (on this thread).
2. RecordLimitError. This code is just there for the evaluation edition. An attempt has been made to open the file in read/write mode with greater than the set number of records in the file. The code therefore sets the OpenMode to read-only and cycles (the loop will then cause another open attempt).
3. NoAccessError. Read-write access could not be acquired so the system tries to open the file in read-only mode (having first warned the end user)
4. NoFile. Provided the create mode has been set the routine will attempt to create the file, if that fails a fatal error is thrown.
5. BadKeyErr. For those file drivers with independent key files (notably Clarion) a corrupt key is non-fatal and the system will try to rebuild the keys so that processing can continue.

The UNTIL 1at the end of the loop means that any code falling through to the end of the loop will cause the loop to terminate. The LOOP is not a real loop; it is simply there to allow some of the recovery routines to attempt to open the file again without a GOTO statement. It should be noted that a CYCLE statement bypasses the loop tail termination condition but not the loop top termination condition.

Position PROCEDURE(),STRING This method differs from the file driver equivalent in that it will issue a UseFile and if a primary key is available it will use that to form the position, or if there is no primary key it will use the file itself. In general (and increasingly) the ABC system assumes (and functions most efficiently providing) that all files in the system have a primary key. Note that this position string can only be used to perform a TryReget; it is not as general as the Clarion language Position. Whilst this functionality restriction does not give us much presently it will eventually allow extra efficiencies within the forthcoming FileClass.

TryFetch PROCEDURE(KEY K),BYTE,PROC TryFetch only really does a UseFile before passing control to the file system, although since C5EEA it has also performed a SetKey in debug mode purely to verify that the key passed in is valid for this file. (The ? is one of my favourite C5 features; it allows you to write code very cleanly which will not be executed when debug is turned off. This allows you to put in quite a few safety checks with zero overhead in the final shipping code.)

TryReget PROCEDURE(STRING Position),BYTE,PROCPerform a Regetfrom a string provided by Position. The Reget/Position pair give the FileManager user one extra piece of encapsulation: independence from key structure. Without them you need to know from outside the class how to uniquely identify a record. This illustrates an important aim: localising information to reduce maintenance.

UpdateServer PROCEDURE(BYTE HandleError),BYTE,PROC,PRIVATE Update and TryUpdate are interfaces to this procedure. Clearly this is similar to InsertServer. The main extra comes from concurrency issues. ABC implements a technique called optimistic concurrency. Put simply, this means the algorithm assumes that no one else will ever change the record being edited locally, and then panics if they did. It relies upon a WATCH having been issued before the record (now being updated) was fetched. In standard ABC usage the WATCH is issued before the view REGET in the browse UpdateViewRecord method.

To handle this the code first takes the position of the current record, then it tries the PUT, if this returns a RecordChangedErrthen the user is notified. (To help the end-user the form template passes in 2 as the HandleError value which prompts the use of a fairly verbose error message which tells the user of such things as the history key). Then the record saved by the other station is loaded (i.e. corrupting the local file buffer) and control is handed back.

UseFile PROCEDURE(),BYTE,PROC The UseFile method is there simply to perform a real open (using OpenServer) if lazy open is currently on and the file is not open. This routine has one of the few bits of defensive coding in the whole of ABC; it actually preserves the file buffers across the Open call just in case the file driver (which could be supplied by a third party) corrupts the file buffer upon the open.

Conclusion

I hope these articles on the FileManager have helped you understand some of what we were aiming for (and have achieved) when we coded this class. It is one of our largest and most complex, and it also presently forms the base of what I call the spine of ABC : FileManager -> RelationManager -> ViewManager -> BrowseClass. FileManager is also the class the developer most frequently needs to interact with (at least as much as BrowseClass and WindowManager). As such I believe an understanding of the principles involved will send you well on your way towards mastery of the ABC system.

(Part 3 of 3. Click to read Part 1 and Part 2)

---

David Bayliss is a Systems Architect for The TopSpeed Development Center. He has worked upon TopSpeed's compiler and was the chief architect of the Application Builder Classes.

Printer-friendly version

Reader Comments

To add a comment to this article you must log in.

Copyright © 1999-2008 by CoveComm Inc. All Rights Reserved. Reproduction in any form without the express written consent of CoveComm Inc., except as described in the subscription agreement, is prohibited.

Clarion Magazine ISSN 1718-9942