A Simple Conversion Logging Template

by Don Reynolds

Published 1997-09-01    Printer-friendly version

Download the code here

Time to do a conversion

If the extent of your conversion needs is to copy files from location A to location B, this may not interest you. However, if you are converting files from one set of formats to another, if the specifications you've been given are incomplete, if you've been asked during the conversion to 'change' certain fields, if you've been asked to clean up dirty files, if you're converting non-coded fields into coded fields, if you've ever been accused of incorrectly converting data (erroneously of course), or if you've needed an audit trail of what's been done, then read on.

Use a Log file

Several times each year, I create a conversion program set. Each varies depending on the formats of the clients' files, the database I convert into, and the type of field conversions. A log file is created as part of the conversion process. It contains some or all of the following:

• The date and time of the conversion
• The record number and identifying information of the input file for transaction messages
• Error messages
  
  1. Missing required fields
  2. Coded value conversion errors-
     
• Special conversion notes -This varies depending on the client's needs
• Other messages
• Totals of records read, converted, etc.

Conversion into coded values

Frequently systems use a lot of coded values. This may involve a simple one to one conversion process, it may involve converting multiple values into a single coded value, or it may involve some combination of fields which, taken together, now results in a certain code.

Sometimes a client provides a set of values to convert into codes, double checks the set, insists the set is complete, only to find additional codes within the data. The log provides a place to show values that would not convert. Conversion programs are designed to be run two different ways.

• Go through all the edits but do not write to the database.
• Go through all the edits creating new records or updating old ones.

Both ways create a log file. Repeated running of the conversion without updating the database can locate missing values which should be put into our coded value conversion subsystem or handled via program changes. (The coded value conversion subsystem is beyond the scope of this article)

Sample log file

8/02/1997 9:14:55 Conversion program starting
8/02/1997 9:14:55 Record=12 Missing SSN, unable to convert
8/02/1997 9:14:56 Record=33 Unknown code BYBqqC
8/02/1997 9:14:56 Record=33 Checkout date blanked
8/02/1997 9:14:56 Record=42 Unknown code BYBqqC
8/02/1997 9:14:56 Record=44 Unknown code BYBqqC
8/02/1997 9:14:56 Record=51 Unknown code BYBqqC
8/02/1997 9:16:56 Record=333 Unknown code BYBqqC
8/02/1997 9:16:57 Conversion Statistics
8/02/1997 9:16:58 Records read: 349
8/02/1997 9:16:58 Records converted: 348
8/02/1997 9:16:59 Records not converted: 1
8/02/1997 9:16:58 Invalid codes: 5

Levels of Concern

Several concerns need to be addressed to provide a good solution for my needs:

• A single procedure, callable from any other procedure, seems the best way to go. I've seen similar functionality in a DOS program developed by Jon B. Bushey, President of RAPID LLC. It was very useful.
• If multiple programs are needed for a conversion, I want to maintain unique log file names for each program.
• I want the single procedure to be as simple as possible, essentially printing whatever is passed to it. The responsibility for formatting this text is in the calling procedure.
• The Logging procedure will, however, add the date and time to the beginning of each line.
• Sometimes I need to run a conversion multiple times, while I fine-tune the code conversion parameters. The log file should not be cleared; the records for another run will append to a former one. In this way, I can compare log messages from one run to another.
• Sometimes I need to run a conversion forcing a prior log file to clear.
• I want to capture as much data as possible if the conversion aborts for any reason. I have seen some systems where the last few records never appeared on the log file. I am willing to be less efficient, performance-wise, in order to get all the records.

Is a better solution possible?

What I need is a template to do all of the above things. This could then be easily dropped into an application. The solution should involve as little time as possible to set up the logging aspects of the system... preferably less than a minute. I can then focus my concentration on conversion aspects rather than logging ones

Template

Following is the code crom the template file SMPL0002.TPL. You can download this file by clicking on the "Save your fingers" icon at the top of this article, then register the template.

#TEMPLATE(SMPL0002,'Simple Logging')
#EXTENSION(SMPLD002,'Global Log file Declaration'),APPLICATION
#DISPLAY('')
#DISPLAY(' Extension template for O_LogMsg')
#DISPLAY('')
#AT(%BeforeFileDeclarations)
g:LogFileCleared BYTE
g:CommandLine    STRING(20)
#ENDAT
#AT(%ProgramSetup)
g:CommandLine = COMMAND('')
g:CommandLine = UPPER(g:CommandLine)
IF g:CommandLine = 'CLEAR'
  g:LogFileCleared = FALSE
ELSE
  g:LogFileCleared = TRUE
END
#ENDAT
#PROCEDURE(O_LogMsg,'Log Message Procedure')
#PROTOTYPE('(String)')
#DISPLAY(' This procedure requires the following:')
#DISPLAY(' Global Log File Declaration')
#DISPLAY(' O_FileErr Procedure')
#DISPLAY('')
#DISPLAY('Possible Calling Conventions:')
#DISPLAY(' O_LogMsg(''Conversion Program Starting'')')
#DISPLAY(' O_LogMsg(''errormsg'')')
#DISPLAY(' O_LogMsg(''Conversion Statistics:'')')
#DISPLAY('')
#DISPLAY('Logfile has date and time prepended to each line')
%Procedure PROCEDURE(PassedLogText)
L:LogMsgDate LONG
L:LogMsgTime LONG
LogFile      FILE,DRIVER('ASCII'),CREATE,PRE(LOG),NAME('%Application.Log')
LogRec         RECORD
LogText          STRING(250)
               END
             END

  CODE
  IF g:LogFileCleared = False
    g:LogFileCleared = True
    REMOVE(LogFile)
    IF ERRORCODE() AND ~ERRORCODE() = 2
      O_FileErr('O_LogMsg','Program Setup--Remove Logfile')
    END !If
  END !If
  OPEN(LogFile)
  IF ERRORCODE()
    CREATE(LogFile)
    IF ERRORCODE()
      O_FileErr('O_LogMsg','Create Logfile')
    END !IF
    OPEN(LogFile)
    IF ERRORCODE()
      O_FileErr('O_LogMsg','Open Logfile')
    END !If
  END !If
  L:LogMsgDate = TODAY()
  L:LogMsgTime = CLOCK()
  LOG:LogText = FORMAT(L:LogMsgDate,@D2) & ' ' & |
  FORMAT(L:LogMsgTime,@T4) & ' ' & |
  CLIP(PassedLogText)
  APPEND(LogFile)
  CLOSE(LogFile)
  IF ERRORCODE()
    O_FileErr('O_LogMsg','Close Logfile')
  END !If

The template in action

1. Add a global extension

• At the application tree press Global
• Press Extensions
• Press Insert
• Select SMPLD002 - Global Log File Declaration from the Class SMPL0002
• Press OK

It should look something like this:

Adding the Global Log File Declaration Template

2. Add a new procedure to your application

• At the application tree press Insert
• Enter O_LogMsg for the procedure name
• Select O_LogMsg for the procedure type
• Press OK

It should look something like this:

Adding the Logging Procedure

3. Insert procedure O_FileErr, described in Clarion Online Volume 1 Number 1. If you don't have this see Template Notes.

4. Within the conversion procedures enter code similar to the following:

• O_LogMsg('YOURPGM starting')
• O_LogMsg('An illuminating message to clarify tons of conversion problems')
• O_LogMsg('Records Processed:')

5. Run the conversion program one of the two following ways (assuming the program name is YOURPGM):

YOURPGM CLEAR

This clears the log file

YOURPGM

This does not clear the log file

Template Notes

This solution involves both an extension template and a Procedure Template. Both are required for the solution to work.

O_FileErr is a procedure discussed in Volume 1 Number 1 of Clarion Online. If you do not have this, you could replace each occurrence with a structure similar to the following, replacing 'Data File Problem' with the concatenated values in parentheses within the template.

IF MESSAGE('File processing error:' |
& '<10,10>' |
& 'File: ' & ERRORFILE() & '<10>' |
& 'Errorcode: ' & ERRORCODE() & '<10>' |
& 'Description: ' & ERROR(), |
'Data File Problem', |
ICON:EXCLAMATION,BUTTON:OK,BUTTON:OK,0).

Conclusion

Using O_LogMsg, I can add a logging procedure to a conversion program set in less than a minute. Then within the conversion logic, I can print any text I want using a simple syntax.

Printer-friendly version

Copyright © 1999-2008 by CoveComm Inc. All Rights Reserved. Reproduction in any form without the express written consent of CoveComm Inc., except as described in the subscription agreement, is prohibited.

Clarion Magazine ISSN 1718-9942