David Bayliss On The FileManager

by David Bayliss

Published 1999-05-17    Printer-friendly version

(Part 1 of 3 - read Part 2 and Part 3)

To understand the design strategy behind the FileManager you first need to understand the name. So far the classes I've covered in this series (ErrorClass, ConstantClass and FieldPairsClass) all have names which end with "Class." That is to denote that they are logically complete entities in their own right. Clearly they are implemented using features of the Clarion language, but it is the class itself that logically provides the functionality.

When I came around to looking at the file part of the ABC structure I quickly realised that here was a very different problem. The largest bulk of file functionality already existed in the language in the form of the FILE structure support by the file drivers. In fact the file drivers can be viewed as objects (especially as Scott was keen to extend the file property syntax to expose some of the file driver data structures).

The file drivers have two major drawbacks when viewed from an ABC perspective:

1. They deal with files. At the template level Clarion files are uniquely gifted with all sorts of useful information supplied by the dictionary (including default values, validation, prompts, descriptions etc). The file drivers only (or mostly) support the information that is actually provided "down at the metal" of the file structure itself. Thus, if the whole of ABC were to use the file drivers directly, all of the value of having a data rich dictionary would be lost to the class hierarchy.
2. They are a black box. A potential beauty of the Clarion language is the way that the grunge of file access is hidden, thereby providing a nice clean programmer interface. The problem is that the wrapper prevents the underlying functionality from being readily extended or overridden.

From these problems arose the idea of a FileManager class. Essentially each file would be owned by a FileManager. The manager would provide a wrapper around the file entity, and the wrapper would embellish the functionality of the file by utilising (and acting as a repository for) the data about the file contained within the dictionary.

Future Proofing

The next aspect to consider is that the black box approach to file classes clearly cannot last for ever. If the ABC system really is to become a fully integrated development solution then, over time, it has to extend throughout the run-time system. The advantages to be gained by having a consistent object-oriented library that allows modification and expansion at all stages in the hierarchy are too compelling to be ignored. In fact, as Bruce Barrington announced at DevCon '98, the FileClass (note the name switch) project is under way and will be providing a major thrust (and raft of benefits) to our future products.

But this presents another problem. We have to be able to produce a FileManager interface now that will still be valid and supportable when the file drivers have changed completely. This is more challenging than the normal OOP problem. All classes need to encapsulate themselves, but the FileManager has to provide encapsulation for the file driver that is not in itself encapsulated. This is rather like the distinction between erecting a fence that will remind you not to walk on a flower bed and erecting a fence that will persuade an exuberant Labrador retriever not to walk on a flower bed. In particular it means that the FileManager has a higher proportion of PRIVATE and non-virtual methods than most of the other classes.

Static Usage

Another peculiarity of the FileManager (shared with the RelationManager) is that the instances of the class to be created will (at least by default) be static. Remember that the FileManager is really just the extension we would like to have made to the FILE itself (but couldn't). Thus it would have the same scope and persistence as a file buffer, i.e. global. But it's not quite global. Most files are actually threaded variables. This gives the FileManager a bit of a headache since some of the data it stores is global (the file name, for example, is the same for all threads), but some of it is threaded (the open state of a file is thread specific). Therefore it was decided to build thread management into the FileManager so that it could control inter-thread resources directly.

Aims & Issues

In tackling a class of the size and complexity of this one, it is worth splitting out and enumerating some of the tasks and requirements it is to meet.

1. Structure Storage. To provide services to return the keys of a file, fields of a file, number of components of a key etc.
2. File Snapshotting. A problem of having global buffers is that when some other procedure wants to use a file buffer it tends to corrupt the buffer contents. In the classic template chain, for example, the RI code actually corrupts buffer contents. A requirement of the manager class was to allow file buffers (and file state) to be snap-shotted and restored at a later date.
3. Auto-Increment. The manager is to provide support for automatically incrementing key components.
4. Retrieval and Update of records. This is fairly obvious.
5. Initialisation and validation. Whilst this is again fairly easy to specify it is one of the key capabilities of the Clarion data dictionary. FileManager must exploit these capabilities.

As well as providing some raw functionality there were some other issues and subtleties that significantly extended the work of implementation.

Firstly, the FileManager was to be smart and reliable. If you told it to open a file then it should go open the file, and do everything within its power to get the file open, including retrying, creating, building keys etc. The reason is to simplify the usage from within hand-code (or embed code). Because the methods will do all they possibly can to make the thing happen you can code as if it has happened. This meant the FileManager needed a tightly integrated ErrorClass . There was a problem, however, because sometimes you don't actually want the system to scream blue murder if it doesn't succeed. The fact the class failed is all the information you require. So most of the methods have "Try" equivalents, i.e. the Open method has a corresponding TryOpen method.. The Try means that the method simply returns an error code rather than taking action on its own (such as presenting an error message).

Another subtlety is aliases. In Clarion aliases are treated pretty much as separate files. Whilst this is good from the IDE end of things it makes some of the object orientation horrible if files are mapped one-to-one with FileManagers.

Suppose the FileManager provided a virtual method to initialise a record into which you drop some embed code. Then whenever you create a record your code gets executed, unless of course you happen to be using an alias, which results in an integrity problem. Now you need to embed the code for every alias as well. Bleh! So we decided to get creative. The FileManager for an alias class contains a reference to the FileManager for the "real" file. Then whenever the alias FileManager is called it simply recalls on to the actual FileManager to do the work. Of course to do this we need to make frequent use of the File snapshotting technology.

The final tweak, one which came up fairly late and which is a mixed blessing, is the notion of LazyOpen. The idea here is that if you use a lot of files in a procedure it is better to open them one by one when required than open them all in a burst at the head of the procedure (which can cause a delay in SQL). This is no problem for people using ABC (as the ABC classes always know if they are about to touch a file) but it causes a bit of a headache for people doing hand-code as they need to warn the ABC system (using the UseFile method) that they are about to drill underneath the ABC layer. (Incidentally, this is one of the problems that will go away once the FileClass is implemented).

Initialisation

Originally the initialisation of the FileManager was very complex. The template generated code called many Addxxxx routines building up within the FileManager a replica of the data stored within the file driver itself. From beta three onwards the file drivers were extended so that the information within them was more readily accessible. For this reason the initialisation interface withered down to two genuine methods, both used to tie the FileManager into the outside world.

I have added two others, Kill and SetThread, as they are both tied to FileManagerManagement rather than FileManagerment itself.

FileManager.Init PROCEDURE(File File,ErrorClass E)

There is a one-one correspondence between instances of the FileManager and files used in the program. The File parameter sets up this mapping. The ErrorClass defaults to the global error manager (when the template generated init code is called). The advantage of having this reference as protected is that a given file (or batch of files) can have a different error manager installed to allow (say) less extreme responses to a given set of files being missing.

The coding is fairly straight forward, the usual array of queues being NEWed and class variables being set.

The only real magic is in the AddFileMapping method. This is there to help with lazy open, or more specifically, to help when a view is used. The view driver can return a list of files that are used in a view, but this is not quite what we want. We want a list of FileManagers. The mapping code is thus there to provide a mapping from file references to FileManager references. The mapping takes advantage of the fact that both file references and FileManager references can be used as LONGs. The Filemappings will be able to die once the File class comes along, and are thus private. In fact they use Clarion's rather nice "in module" private technology to allow the private methods to be completely removed from the .inc file.

FileManager.AddKey PROCEDURE(KEY k,STRING Desc,BYTE AutoInc)

This is the one remaining Add method. It is required because there are two pieces of information stored within the dictionary and required by the FileManager that are not available to the FILE driver. The first is the textual description of each key, and the second is the auto-increment nature of the key. The AutoInc value specified is the component of the key which is auto-increment. Within the templates this will always be defined as the minor-most component of the key although this restriction is not imposed by the base classes. (That said, non-minor-most component auto-incrementation is much less tested than the more normal form!)

Whilst the method itself is of little strategic importance it does use a number of relatively new or advanced coding techniques so it is worth a little further investigation.

The principle job of AddKey is to form a new record in the FileKeyQueue. This is a private property. Because Clarion allows unresolved forward references (i.e. allows opaque types) the queue structure is not defined within the .inc file, only within ABFILE.CLW.

Assigning the key reference and description is fairly straightforward. Use of the description field as fixed length (rather than assigning a reference) is not as painful as you may think because Clarion queue buffers are compressed when stored in memory so any trailing spaces simply squash up.

The interesting code starts with this reference assignment:

self.keys.fields &= NEW KeyFieldQueue

This code is setting one of the fields of the FileKeyQueue to be a new instance of the KeyFieldQueue. That means there will be a KeyFieldQueue for every record of the FileKeyQueue. In other words this is a queue of queues. This is a very powerful technique, but it is also something to be careful of. In the Kill method for the FileManager we have to step through the FileKeyQueue freeing up each one of these sub-queues.

Next note that SELF.HasAutoInc is set if AutoInc is true. HasAutoInc is set on a per file basis, not per key. The information is really redundant since HasAutoInc could always be computed simply by stepping through the values in the key queue, but HasAutoInc is hit sufficiently often it was deemed worth the extra cost (1 byte!) of storing a copy. SELF.PrimaryKey is stored in the object for similar reasons.

Finally there is a loop filling in the nested queue with one entry for each key component. This queue has two gotchas to watch for. The first is it contains an ANY. You must always CLEAR a queue buffer containing ANYs before assigning to the fields (see the Language Reference Manual). The file driver can return the field number of a key component, and the FileManager has the record buffer in SELF.Buffer so we can use WHAT to return an ANY referring to the field of the key component. This then gets stored in the queue.

The other required piece of information is the "bind" name of the key component. This comes from PROP:Label, but the code does one more piece of pre-computation. If the key is case insensitive and the field is not numeric then we store UPPER(FieldName) rather than FieldName in the queue. This is neat as it means routines further down the line can simply use the stored field name without regard to case sensitivity and the issues are dealt with for them.

FileManager.Kill PROCEDURE

I suspect some would argue that Kill is not an initialisation method, but I disagree. Kill is really the mirror image of Init. I nearly always write them at the same time (literally a line of Init, then a line of Kill). The job of Kill is to undo anything that has been done during the Init, and sometimes it has to undo the work of later routines too, but acting as an un-Init is a must.

Notionally this routine is very straightforward but there is some quite involved picking apart of the queue of queues. The outer loop steps through each key and the inner loop steps through each key component. Finally for each key component the name is disposed and the ANY variable is freed up.

Failure to do any of these steps would result in a memory leak.

FileManager.SetThread PROCEDURE

As suggested in the Static Usage section one of the problems the FileManager has to contend with is that a file effectively has a new instance on each thread. This needs mirroring in the FileManager. Rather than clone the FileManager for each thread (and support an inter-communication layer) it was decided to support a queue within the FileManager where each thread was mirrored by a record in the queue. This may not please the object purists but as the thread specific data totalled 10 bytes compared to >1K for the whole object, the engineers will understand the logic.

The methodology is thus quite simple: each procedure starts by calling SetThread which pages in the correct data for the current thread; the procedure can then simply access the data using SELF.Info. There is a slight gotcha here. Because SELF.Info is a queue buffer, any method altering a value in SELF.Info must then do a corresponding PUT(SELF.Info) or the value altered will be lost upon the next call to SetThread.

The implementation is straightforward. If the file is threaded (early ABCs ignored this condition!) then the current thread is read from the system. The current list of thread data is looked up, and if it doesn't exist then a new record is cleared and created.

The final line of this procedure is actually a cheat. It has nothing to do with setting the thread; it simply takes advantage of the fact that all FileManager methods will call SetThread before commencing execution. SetThread thus because a suitable repository for always call items. In this case the error manager is being primed with the current file name so that any errors within the methods themselves can simply Throw, confident that pertinent details have been filled in ahead of time.

Summary

I hope this overview has been of some help The FileManagers are coded in a fairly unusual context which has greatly influenced some of the design decisions taken during their implementation. In my next article I will go into the remainder of the methods looking at their code and design ideas.

(Part 1 of 3. Read Part 2 and Part 3)

---

David Bayliss is a Systems Architect for The TopSpeed Development Center. He has worked upon TopSpeed's compiler and was the chief architect of the Application Builder Classes.

Printer-friendly version

Copyright © 1999-2008 by CoveComm Inc. All Rights Reserved. Reproduction in any form without the express written consent of CoveComm Inc., except as described in the subscription agreement, is prohibited.

Clarion Magazine ISSN 1718-9942