David Bayliss On The ConstantClass

Posted March 19 1999

Printer-friendly version

Behold, for peace I had great bitterness: but thou hast in love to my soul delivered it from the pit of corruption: for thou hast cast all my sins behind thy back.

One of the interesting differences between different people is their perception of events past. Two individuals can experience exactly the same thing, one remember all that is good, the other all that is bad. This difference is most marked when reviewing our own performance. The most interesting point is that we all tend to assume we have the right to our own unique slant on history. Yet we all have a phrase that sometimes troubles our sleep, the "skeleton in the closet".

I don't know where the expression came from but I actually did attend one of the first "schools for poor boys" in the UK (est. 1640) where they did have a skeleton in the closet. A boy who had been punished for cheating by having a lead pencil shoved up his nose. When he died of internal bleeding the headmaster tried to hide his deed by incarcerating the boy in a closet. He was discovered some years later. One of the more sobering messages of the bible is that one day those closets will be opened.

The issue facing me is rather less drastic but still problematic. I must confess that in writing this article I am not quite sure of the correct tack to take. Should I explain the design insights I should have had (as I can now see with perfect 20-20 hindsight) or should I detail the evolution of the Constant Class from the germ of an idea into one of the cornerstones of the system?

Given the Constant Class didn't appear in its current form until C5 (and the gaffe is therefore blown) it is probably instructive to take the latter approach. Therefore you will see how a particular need developed into a set of (widely dispersed) code. How this code was then extracted (and abstracted) to form a class that is now generally applicable.

I would also like to acknowledge that although the design of the class was mine the implementation was done by Roy Hawkes of the TopSpeed Development Center.

The Germ

The real germ of the constant class is actually the data set used by the ErrorClass. However to understand the full usage of the Constant Class it is necessary to look at the problem in a wider context.

Just about any program can be broken down into two distinct parts, the algorithm and the data it works upon. Now as database programmers you will automatically think of data-files when I use the term data. But in fact, commonly, the data in the data-files will not be provided by you the programmer, it will be provided by the end-user. (Of course the end-user may also be you!) The data in your actual program will usually be far more diffuse, possibly even hidden, but still vital.

One piece of data you may not think of is the Windows and Reports in your program. They are just data-slabs (resources actually). But there are also many constants "hard wired" into your programs, usually in the form of parameters. Take the following:

?MyString{PROP:Text} = 'The meaning of life the universe and everything is...'

That code obviously contains a string constant, right? Actually, no. It contains two string constants and two numeric constants. (Prop:Text is a string, ?MyString is a number as is the implicit Window $ that precedes it.)

Now at the top of a procedure with a moderately flexible display there will often be ten or more of these lines. In some common totally dynamic cases it is easy to end up with a hundred of these statements (consider setting the color, position, tooltip and icon of a set of buttons on a user-configurable toolbar). This is perfectly reasonable and easy to code but it does have some significant drawbacks:

• Code bloat. Each of the above statements compiles to about 50 bytes. 100 lines is 5K ...
• Inflexibility. Suppose there are two types of text string, for different languages perhaps. That means two sets of code (or CHOOSE statements).
• Alterations have to be done in code. To appreciate this one you need to subscribe to the Bayliss Mantra that says "new code is bad code." If you allow that >75% of coding effort happens during the maintenance phase then new code is <25% of the way there. Therefore you should always aim to let your source files get at old as possible. The older the date on the file the more likely it is to work. But if constants are in code then it is impossible to change the occasional misspelling without soiling the source code.
• Detail and complexity are interwoven. Complexity in programming is the sums and the clever bits (such as control positioning). It has to be done very slowly (or very often!). Detail is all the grungy stuff that has to be done well and thoroughly but can still be done quickly. Typically there will be 10x as much detail as complexity, but the complexity takes 10x longer per line to write. If you mix detail and complexity you can end up with 10x as much code that has to be written slowly. So you want detailed constants (such as prompts / messages) out of the main body of your code. 
• The "prompt" writer has to program. This is really just another facet of the above two but still important. In the ABC system I want the prompts to be alterable by people who may not be happy trying to penetrate my coding. Therefore I need to be able to identify the safe zone (the .trns) from the DAB zone.
• Lines full of constants can weigh heavily on the 16 bit compiler's internal data pools leading to the dreaded dynamic pool errors.

The First Solution

With the above somewhat forbidding list weighing on my mind I retired to slumber and awoke in the early hours with a partial solution. Why not use a statically initialised group to act as a data stream that can be parsed apart and then used. The group can reside anywhere and you have a separate piece of complexity to parse the data structure. One key extra thought is that you can use pstrings or cstrings in the data structure. This means there is no wasted space. Fairly rapidly I wrote the code for the error class (see ErrorClass.AddErrors, in aberror.clw). After about 20 lines I was home.

Have you ever noticed that when you get a new tool for your toolbox there are a whole host of jobs that arise to which the tool is perfectly suited? Within a few weeks we had five other similar pieces of code scattered around the ABC system. Then the synchroniser team spotted the trick and did the same. Come the spring of '98 the Wizatron team had also rolled out five or six copies of similar code.

Now you might think, hey, why not use procedures, then you only get one copy of the code! But the task is not that simple. The thing is that although each of these pieces of code used the same idea, they each did different things with the data, and their data was a different structure. For example, in abtoolba.trn the structure is simply ushorts followed by strings. The print-preview has two strings and a short. Sometimes the data was being used to pre-fill a queue, sometimes in property statements of a window. Although the germ was the same between each block of code, very little of the code was actually the same. Also each lump of code was quite easy to write so the easiest way for people to reuse the code was simply to copy it from somewhere else and hack it a little bit.

OK is not OK

This is the easiest approach, but not the cleanest. The problem is that the low-level of the constant class code (as it then wasn't) is actually extremely dirty. It makes assumptions about the layout of Clarion groups & the data structure of clarion data types. The thought of having >20 copies of that code floating around the system was not pleasant. Also the Wizatron team needed the code to go as fast as possible, that means spending time tweaking; we didn't want to do that many times. It came down to an engineering decision. Do you put up with a sub-optimal solution in order to make life easy or do you hammer at the problem until you get it right? There is a phrase I always bring up when interviewing people to see how they react: "Ok is not OK."

So at this point we sat down and thought, what actually can we abstract from this problem to enable us to write a class. We came up with the following: "The class should have the ability to parse apart an unknown but externally specified data structure into some suitable location or locations(s)".

That wasn't much to go on, but it was enough. We could now see how the class would be used. First there would be an init sequence specifying where the group was, what fields should be expected and in what order, and where the result for that field should go. Then there would be a next loop somehow reading the group record by record. Then some kind of closedown. One could still argue the spec is a little vague, and Roy did look extremely dis-chuffed when I told him my idea, but the result actually works very cleanly.

I will now go through the methods themselves in the rough sequence they would be used. I shall assume you have the actual code to look at (you can find it in abutil.clw).

The Implementation

For the following I will describe the group as the data structure and those fields pertaining to one "set" of data as a record. So supposing I had a group which consisted of 20 pairs of strings. Each string pair I will refer to as a record.

ConstantClass.Init

Internally the list of fields (that is to say the data types expected and where they should go) is to be stored in a queue, so the Init method constructs this. The parameter it takes gives the first level of flexibility built into the system. How do you decide when you have read all the records? You have four alternatives :

1. Term:EndGroup - Stop when you come to the end of the group. This is the cleanest and least accident prone (you can't run off the end of the list or loose items on the end). The main down-side is it doesn't allow you to arrange for only the first n items of a group to be used.
2. Term:Ushort - The first field of the group is unique (a ushort) which denotes the number of records (groups of fields) to follow. This is efficient and clean but suffers from two types of accident. If the number is too high you get a gpf as the parser trips off the end of the group. If the number is too low the last elements are silently ignored. On the plus side you can select to use only the first n elements.
3. Term:Byte - The same as (b) but limited to 255 records
4. Term:FieldValue - This is the old magic number trick. If you specify this then the parser keeps going until it finds a first field of a record that matches the termination sequence (which you have to put into the TerminatorValue field). This is a good lazy/safe solution if you are reading a sequence of strings. Just make the last value '**STOP**' and make that the terminator value. The only real down-side is a slight danger of picking a terminator that someone uses as a data value. The biggest perceived downside is a guilty twinge that proper programmers don't use magic values.

Note that part of the Init function is 'split out' into Reset to allow a group to be re-parsed without going through the whole set-up rigmarole again.

ConstantClass.AddItem PROCEDURE(BYTE ItemType,*? Dest)

This tiny little procedure is the hub of the class. It is called once for each field in the record. The sequence these are called does matter. It is assumed the first call to AddItem is the first field in the record. The *? Dest is a field pointer to the place you want the result of the field to go. The type of the variable passed in does not have to be the same as specified by ItemType. ItemType pertains to the type of the field in the group structure. (That said, using pstrings to store byte values would be a bit daft)

Note the CLEAR is important as the queue contains ANY variables. (If that sentence didn't make sense consult the LRM, the usage of ANY variables in queues is very delicate, but powerful).

ConstantClass.Set PROCEDURE(*STRING Src)

This is the procedure that hands the group over to the constant class and primes the constant class to read it. Note that you may do multiple sets, so one constant class can process many different groups provided each group has the same structure. Note too that the constant class copies the group so groups passed into the constant class do not need to be static. (This illustrates an interesting trade-off. The constant class is general purpose and therefore is coded to be much safer than the ErrorClass equivalent. The down side is that it is slightly more expensive in use).

ConstantClass.Next PROCEDURE

You may think of Next very much like the file driver equivalent: read in the next set of data. The only difference is that the data goes into those places defined by the AddItem, not into some separate record buffer. (Re-read those two sentences again 10-15 times. There may well come a time when the concept is important).

The coding is rendered more complex by the need to check for four different methods of termination. Note that upon termination Level:Notify is returned so a simple LOOP WHILE ~MyConst.Next() can be coded.

Leaving aside the CASE SELF.Termination (which is detail, not complexity) the real code starts with the GET of SELF.Descriptor. That is moving the queue so that the results of the first AddItem are available. Now dependant upon the type of the data field the suitable byte-shuffling is done to read out a value which is then placed into the required location. Then the next item in the queue is read to process the next field within the record. It is possible to get confused here. The LOOP inside next is looping through the fields of the record, not the records of the data structure.

Note too that the detail of the exact low-level format is removed from this complex procedure. This procedure is the controller procedure for making sure the right fields are read in the right sequence and sent to the right place. That is enough for one procedure to worry about.

ConstantClass.Reset PROCEDURE

This procedure is simply to allow the same group to be re-read without an intervening SET. It also handles the reading of the leading byte/ushort (if appropriate) allowing the Next procedure to stay clean (because there is no special case of first record).

ConstantClass.Kill PROCEDURE

This is the traditional way of closing off a class. Mainly there to dispose of the queue and the copy of the group that was taken. Note though the special handling required of the queue because it contains an ANY variable.

ConstantClass.Next PROCEDURE(QUEUE Q)
ConstantClass.Next PROCEDURE(FILE F)

The preceding procedures are the real interface. There are then a couple of procedures just to make life easier. One thing we noted was that very commonly all the fields of the AddItem were actually fields of a queue, and straight after the Next we would do an Add. So we made that case easier. You still pass in the fields of the queue but rather than code the loop yourself you pass in the queue and the work is done for you. This is done similarly in the file case.

ConstantClass.GetByte PROCEDURE()

Now, in all the above one vital little detail has been deliberately left out. How do you parse the group structure to give all this high level information? There are essentially two different problems, the first is keeping your place in the byte stream; the second is plucking the various data types out of this byte stream.

In typical fashion these two problems have been teased apart. GetByte tackles the first. After a Reset the first GetByte will return the first byte of data from the record stream after any leading byte or ushort (to give the record count) has been read. The next call to GetByte will get the second byte, etc. GetByte has dual usage. It is used by all the other Get functions as a primitive but can also be used in its own right to read in fields of type byte.

ConstantClass.GetUShort PROCEDURE()

Ushorts are stored with the least significant byte first so this is read (using GetByte), then the second byte is read, shifted left 8 and added to the first. You may wonder why we've used Rval. This is because the evaluation order of the operands of the addition operator is not specified so, as the GetByte calls are sequence specific, you need to split the expression to define which one comes first. In other words if you have the code GetByte()+BSHIFT(GetByte(),8) it is not defined whether the first or second GetByte call will happen first so the data could end up reversed.

ConstantClass.GetShort PROCEDURE()

This is precisely the kind of low level dirt that persuaded me this code had to be in a class. Constructing two bytes into a ushort then claiming it happens to work if you just assign that to a short is extremely ugly. But it works beautifully (for now). By encapsulating this code I can ensure it works in the future too.

ConstantClass.GetLong PROCEDURE()

That natural way to do this is with four GetBytes where you shift 8 bits for each iteration of the loop. It took me a while and quite a lot of squared paper to persuade Roy that this trick would work, but it does!

ConstantClass.GetPString PROCEDURE()

Twigging that constant pstrings could be parsed without wasting space was one of the key concepts behind this technology. It is actually very simple. You just read the leading byte (the length byte) and then you know how many more characters there are to read. Now here again we have traded some speed for some clean. It so happens that because of the way we store the group (in a copied string) we could code the pstring reader to be quicker by using a string slice and updating the pointer in one go. Instead we have chosen to persist with the GetByte interface reading the pstring one character at a time so that we could change the data storage mechanism with a minimal re-write.

ConstantClass.GetCString PROCEDURE()

Reading the cstring is less complex than the pstring and yields itself naturally to the GetByte approach, you simply sit in a loop reading characters until you hit the null terminator.

Conclusion

There you have the evolution of the constant class. I hope the insights into our development process have been of some help, so too the insights into the class itself. For people who like something concrete on which to hang their ideas the simplest example of using the constant class is probably the SetTips procedure in abtoolba.clw. InitializeColors in abeip.clw is fairly simple too. Using the class to fill a queue in one shot is shown in TranslatorClass.Init.

Article comments