Clarion, the Web, OOP and You (Part 2)

by Tom Hebenstreit, Review Editor

Published 1997-11-01    Printer-friendly version

Download the code here

Welcome back! In the first installment of this series, we examined some of the basic concepts of the World Wide Web (such as HTTP and HTML), the pros and cons of the various ways your programs can interface with a Web Server (CGI, Win-CGI and ISAPI) and the tools you will need to help you build and test your web program (Web servers, etc.). If you missed that article, or have forgotten what the preceding alphabet soup of acronyms mean, you might want to re-read it to refresh your mind.

This month, we will be examining in detail each of the three methods you can use to interface your programs with a web server: CGI, Win-CGI and ISAPI via the Tornado OCX. To help get you started on your own projects, there are code examples, CGI/Win-CGI OOP templates and a Tornado shell application which can be downloaded from right here on Clarion Online.

One quick point about the demos and sample code provided here needs to be clarified before we can proceed, though. Unlike a 'normal' demo, where you can just stick it somewhere and run it, these web examples rely on the particular directory structure and naming convention being used by the web server you are testing with. To complicate matters, web paths are usually specified as 'relative' paths, i.e., paths which exist in relation to the current directory or the root directory of your domain. Those paths might have very little to do with the 'absolute' path, or actual directory structure of your ISP's hard drive. The advantage of relative paths is that they work the same on your machine as on the actual web server. Regardless of what the domain or actual disk structure is, the server will understand that '/bin/myprog.exe' means "from the root directory of this domain, look in the '/bin/' directory to find this program".

To illustrate, on my machine, when testing with Web-Site, the 'domain' for the server is 'localhost' (just like the domain for Clarion Online is 'www.clariononline.com' on the actual World Wide Web). So a program called 'myprog.exe' would be specified like this:

When using MS Personal Web Server and a domain of 'desktop', it looks like this:

The moral of the story here is that when you see "/bin/demo.exe" in any of the following examples, just keep in mind that you will need to substitute the relative paths and directory names as they are set up on your PC and web server.

All right! We have a lot of material to cover here, so put back on your webmaster cap and let's get started with delving deeper into the interfaces, templates and code...

Setting the scene

Let's begin by establishing a common scenario which will be used by all of the examples:

Our goal will be to process the input from a page of the 'Big Secret Club' web site. The page itself has two primary items of interest on it for us - a 'list members' link to show all members (this will act like a menu option, directly invoking our program), and an HTML Form which has a question prospective members must answer before submitting the Form to our program for processing. In case you are wondering, the Submit button on a Web Form is rather like the OK button on a CW form - it tells the system that you are done with the Form and that you want it processed now, i.e., sent back to the server (and from there to your program). Our example page looks like this:

Our example web page

When someone clicks on the first link (the one that says 'click here'), our program will return a list of members. When someone submits the Form, we will retrieve the Form data, check their answer and then either let them join up or send back a 'Sorry, try again' type of message. This combination of requirements should allow us to explore the most common actions performed by web programs (processing a Form, returning a page, and returning a page with a table).

Note: we're not going to discuss how to design web pages and Forms here, as there are a million books on that topic already out there. Also, if you are using a modern HTML editor like MS FrontPage or Claris HomePage, that process has become pretty much a purely visual experience similar to the CW Window Formatter, i.e., it can be done without any knowledge of HTML by filling in prompts (which in turn generate the HTML). What does concern us directly is how to fill in those prompts, and that depends on the interface we are using.

If we were working on a program called 'xdemo', here is how it would be called (depending on the interface). For the 'click here' link which lists members, we would specify the link destination (URL) as follows (only without the quotes):

The basic format is first the program name (and relative path), optionally followed by a question mark and then what we want the program to do. The question mark is the delimiter - everything on the left is used by the server to find and run the program, everything on the right side of it is passed on to your program. If your program only performs one single task and doesn't need any parameters, you may not need to pass it anything at all (in which case you just enter the program name and path without anything else). Other times, you may want to pass both instructions and data.

And before you ask, the parameters, etc., that we are passing to the demo programs are completely arbitrary - in other words, I just made them up. The only requirement is that your program understands it since the browser and web server couldn't care less what you put there as long as you stay within the rules for something called 'URL-encoding' (which we'll get to later).

As for our Form, there is a special HTML tag (called FORM, amazingly enough) that tells the browser that there is a Form on the page it is displaying, and that tag has an attribute called 'action' which tells the server what to do. It would be filled out the same way as the link except for what we pass to the program. Why? Because this time we want it to process the Form, not list members. For example, when used with the Tornado OCX, the action would be "/bin/xdemo.dll?processform".

Finally, you need to be aware that when you design an HTML Form, you need to give each field on the Form a label just like you have to give all variables a label in CW. This label is how you will refer to the field later so that you can retrieve the values entered by the user. For our example, we'll label our Form field 'answer'. (Original, aren't we?)

Ok, now we've seen a couple ways to have the web page start our program, but how does our program know what to do once it is running? Glad you asked...

Getting to know all about you

In addition to the HTTP variables discussed last month in Part 1, there is another set of similar variables which accompany every request which executes a script or program. These variables, often called CGI variables or Environment variables, provide all sorts of information about the request itself, and although there are quite a few of them, there are only two that you really need to understand in order to achieve your goal of writing web programs. They are:

REQUEST_METHOD

This one tells your program just what kind of request it is dealing with, and through that, where look for any data and/or instructions which might have been passed to your program from the web server. There are many types of requests, but for our purposes we only will be dealing with two of them: GET and POST. Like all of these server variables, REQUEST_METHOD is a field=value pair, as in: REQUEST_METHOD=POST.

GET - A GET request tells your program that all instructions and data have been passed to your program in what is called the QUERY_STRING (the second of our critical CGI variables, which we'll get to next). GET requests are usually used when initiating an action directly (like our 'List Members' link), though it can also be used to process a Form.

POST - POST is the standard method for processing Forms, and a POST request indicates that in addition to the QUERY_STRING, there are other data fields available for your program to process. The data is always in the form of field=value pairs, and is retrieved one pair at a time just like HTTP variables or the REQUEST_METHOD itself. (Details on how to actually retrieve field=value pairs are provided later.)

QUERY_STRING

As stated above, the QUERY_STRING is one of the two ways to have information from the user passed to your program. It is where everything to the right of the question mark is put when the server sets up the interface to talk to your program (remember the question mark as in "cgidemo.exe?listmembers"). If the REQUEST_METHOD is GET, and you are processing a Form, it will also include all of the field=value pairs in a special format called 'URL-encoded'.

Side Note: We won't go into the details of decoding URL-encoded strings here other than to say that it is a specification where spaces are turned into plus signs, field=value pairs separated by ampersands, and most punctuation marks, etc., are turned into hexadecimal preceded by a percent sign. Trust me, you DON'T want to mess with this if you don't have to!

Fortunately for you, in all three of the interfaces we are discussing the decoding is already taken care of for you by either the web server, the templates or the wrapper classes. If you'd like to see an example of URL-encoded data, just go to a web search site, start a search and then take a look at the URL line in your browser - all the stuff to the right of the question mark is URL encoded...and have fun trying to figure it out!

Back to business... Using our example web page, let's say a user pressed the 'click here' link to list members. When our program starts up, it will find a REQUEST_METHOD of 'GET', and a QUERY_STRING of 'listmembers'. If the user filled in an answer of '42' and pressed the Submit button, the program would see a REQUEST_METHOD of 'POST' and a QUERY_STRING of 'processform'. Additionally, it would need to ask for and get the value for the Form entry field which we named 'answer' ('42' in this case).

So, to finish up with CGI variables, you'll find that web programs will almost always begin the same way: get the REQUEST_METHOD, the QUERY_STRING, and then, if it is a POST request, start retrieving field=value pairs for processing. Note: The Win-CGI and CGI templates and Tornado OCX automatically take care of all of this start up work for you - isn't that nice of them?

You talking to me? I'm talking to you...

The final key point you need to understand when discussing the various interfaces is that no matter which interface is being used (CGI, Win-CGI or ISAPI/Tornado), it is filling the same overall role - acting as a middle-man or translator between the Web server which received the request and the program you wrote to fulfill it. Also, for all of the interfaces, the general sequence of events is identical: Sever to Interface to Your Program back to Interface back to Server.

Where the interfaces differ is in how they format and pass the information back and forth. In simple terms, they break down like this:

• Win-CGI - The information is written into an ASCII text file in a format just like an INI file (sections in brackets followed by field=value pairs). This file contains both the standard HTTP/CGI variables (such as REQUEST_METHOD) as well as various sections for Form data, etc. The server then runs your program, passing two pieces of information to your program via the command line: the name of the input file to read, and the name of a temp file for you to write your output to. Your program reads and writes to those files using the stock CW ASCII file driver.
• CGI - When the server allocates a memory space for your program to run in, it places the HTTP variables into the Environment for that space, once again as field=value pairs. While this may sound arcane, what we're talking about here is really just standard DOS environment type variables like the ones that fill your Autoexec.bat file. Your program uses Windows API calls to read them, and then, if the REQUEST_ACTION is POST, uses more API calls to read from what is known as 'StdIn' (standard in) to get the field=value pairs for the Form. The web page you create is written back to the server using 'StdOut' and even more API calls. Don't worry, however, if you don't understand what these last few sentences mean - once again, the templates completely hide that complexity with the class methods.
• ISAPI - The ISAPI DLL notifies your program (which is always running) that there is something to do via an event generated by the Tornado OCX. All the usual HTTP and CGI variables are already available, and your output is written directly back to the OCX (and through it, the ISAPI interface to the server).

In each case, when your program either terminates (CGI and Win-CGI) or flags the server that it has finished processing (ISAPI), the server sends the page you created back to the users web browser and the entire cycle is complete.

Whew! A lot more background there, but we should be at a point now where you have a good grasp of how the interfaces connect with your programs (and vice versa). Time to dive into the templates and applications themselves.

Inside the Win-CGI and CGI templates

We will deal with these two templates sets at the same time, because they are functionally identical from your (the template user) point of view. How is this accomplished? Basically, it is done by wrapping the external interface (CGI with its Environment variables and StdIn/StdOut API calls; Win-CGI with its INI and temp files) within an OOP Class wrapper. This class encapsulates (OOP-speak for hides) the low down nuts and bolts of the actual interface, so that you only need to talk to the class/object, not the interface itself.

On the practical side, this means that you can write essentially the same program and, depending on which templates you use, have it end up as either standard CGI or Win-CGI. You don't have to sweat the details of the interface or the Windows API - you only need to write whatever standard Clarion code you need to accomplish the basic task and let the templates take care of the rest.

Note: From here on in when discussing these two templates I will be using the term 'CGI' rather generically to cover both standard CGI and Win-CGI. If specific differences need to be pointed out, I will note the actual interface in question.

Overview

In a nutshell, when your program is run by the server the templates automatically invoke the CGI class methods which perform the common tasks like retrieving the standard REQUEST_METHOD and QUERY_STRING variables. If the REQUEST_METHOD is POST, they also take care of retrieving all of the Form field=value pairs. This Form data is parsed out, decoded (if necessary) and then loaded into a global Queue, which is where you'll find it ready and waiting when the program finally gets to your code. The class/object automatically takes care of initializing the required HTTP headers for the page you are going to send back as well.

Welcome to the Class

First off, I highly recommend that you spend some time looking at the generated source code for your CGI program. We will only be looking at snippets here and there due to space constraints in the article, but you will get a much more thorough understanding by looking at the generated source for the CGI Class methods and properties.

The following are a few of the most important CGI class methods and properties (the ones you will use all of the time). Note the use of the SELF keyword to indicate that the method should act on the current instance of the class (it is actually the ONLY instance):

SELF.GetValue()

This is one of the core functions, and its purpose is to retrieve a value from the Queue of field=value pairs. You pass it the name of the field you want to retrieve, and it returns either the value for the field or an empty string if the field was not found. In the example below, the string variable 'UserAnswer' would get a value of '42' if that was what the user entered into the 'answer' field on our sample Web Form.

UserAnswer = GetValue('answer') !From the sample web page

SELF.HTMLOut()

Here is the other main procedure which you will use all of the time. It simply writes whatever string you pass to it out to the web page that is being created to return to the user. Once again, you don't need to worry whether that string is being written to a temporary ASCII file (Win-CGI) or written via an API call to StdOut (CGI). Note: When using this, you DO need to write valid HTML, since what you are doing is basically just creating a web page line by line. Examples:

SELF.HTMLOut('<h1>Members</H1>')
SELF.HTMLOut('<hr>')
SELF.HTMLOut('<p>' & clip(fil:text) & '</p>')

These three lines would write out 1) a large bold header, 2) a horizontal line, and 3) a paragraph containing the contents of a field called 'fil:text'. The field could contain anything from a few characters to a large memo, and the user's browser will wrap it to fit.

The Preformatted Table Methods

These are a really nice set of methods which are used together to easily create an HTML table of data. In the following example, we will create a table with three columns, one each for last name, first name, and phone fields from a file called 'member' (MEM) - all without writing any HTML!:

SELF.TableHeader('First|Last|Phone')
SET(Mem:By_LastNameKey)
LOOP
  NEXT(Member)
  IF ERRORCODE() THEN BREAK.
  SELF.TableItem(CLIP(MEM:First) & '|' & CLIP(MEM:Last) |
   & '|' & MEM:Phone)
END
SELF.TableFooter

The Table methods are using the vertical bar character as a delimiter within the strings you pass to them, so that is how it knows where one column/field ends and the next begins. The TableHeader() method begins a table and automatically makes the header text bold, while the TableFooter method signals the end of the table. If you were creating the table by hand, you would need to write out a ton of the begin/end tags that are used to build tables in HTML (tag pairs for the table, each row, each column, etc.), so you can see how these methods simplify the process. In fact, the above code is all the code we would really need to create our basic list of members as requested by our sample page.

Finally, there are also properties (and prompts on the templates to set them) which allow you pretty much complete control of the borders and other details of the generated table.

Controlling Program Flow

Per our sample web page, here is how our program would look if we were using routines to do the work. This code would be placed in the 'CGI Processing' embed in our web app. Basically, we check the QUERY_STRING, and then proceed based on what's there.

CASE SELF.GetValue('QUERY_STRING')
OF 'listmembers'
!-- same code as the above Table example
  DO ListMembers
OF 'processform'
!-- Get the 'answer' field
  Temp" = SELF.GetValue('answer')
  IF Temp" = '42'
    DO WelcomeToTheClub
  ELSE
    DO SorryTryAgain
  END
ELSE
  DO UnknownQueryMessage
END

Not so scary, is it? The key is that this is basically the entire program (other than some SELF.HTMLOut() statements in the appropriate routines) and it would work equally well in the CGI templates or the Win-CGI templates.

Creating a CGI or Win-CGI Program

Now that you have learned something about the CGI Class methods and how to use them, here are the exact steps needed to create a new web application using either the CGI or Win-CGI templates

[Note: It is assumed that you have downloaded and registered the CGI and/or Win-CGI templates already. If you haven't done that yet, don't worry - it should still be easy to follow what is going on.]

1. Open CW and create a new Application, then name it. Don't use any wizards, etc. - all we want is an empty App.
2. Select 'Project', 'Edit', 'Properties', set the Target-OS to 32-bit, then save the project.
3. Press the Global button, then Extensions and insert the 'CGI Global Extensions' template under the desired class 'WinCGI' or 'CGI'. This Extension will set up the Class definitions, API prototypes and other details required for it all to work. What do you have do here? How about just enter 'CGI' in the Class Name prompt, and, if using Win-CGI, put a check in the 'Call SELF.GetAllCGI' option..
4. Press OK - OK - OK until you are back to the procedure tree. Double click on 'Main - (ToDo)' and when the list of procedure types pops up, select 'CGIProcedure - Main CGI Procedure' (make sure you choose from the SAME CLASS that you chose for the Global Extension - the two classes will NOT mix!).
5. Click on Embeds, then select the 'CGI Processing' embed, and tell it you want it to be 'Source' (in other words, you will provide the code). Once in the editor, enter whatever code you need to accomplish your goals.

That's it. Save the embed, save the procedure, press the lightning bolt to compile it and bingo - you have a program which is ready to run on a web server! (Don't press the blue cloud - the program won't behave correctly if it isn't being run by a web server).

To test it, you will need to copy it to the executable directory of the web server you are using for your testing, load your browser, and then use a web page to run the program using either a link or a Form/Submit button.

This concludes our discussion of the CGI and Win-CGI templates. Be sure to check out the code and comments in the sample CGI and Win-CGI apps... and have fun!

Using ISAPI and the Tornado OCX

We now move on to a much different way of interfacing with a web server - using ISAPI and the Tornado OCX. Part 1 of this series discussed the extra requirements for this method (installing an OCX on the ISP's server, etc.), as well as what you get for those extra efforts (speed and the ability to use HTX files). Here's how this method contrasts with the CGI/Win-CGI interfaces we just discussed.

What's different

In some ways, the mechanics of this interface are the exact opposite of the CGI/Win-CGI interfaces. For example, where CGI programs never open a screen, a Tornado application must have a Window and an Accept loop active so that the OCX can communicate with your programs and the server via events. Instead of being run on demand, the application is always running and ready to go, and because of that, your files are not being constantly opened and closed every time a request is handled. Also, instead of communicating directly with the web server's interface, the OCX communicates through the Tornado 'Administrator' application which mediates between the server, an ISAPI DLL and your program.

One other difference is more of a design consideration. Generally, CGI/Win-CGI programs are kept as small as possible to minimize load times, etc., on the server. The result is that you commonly have a lot of small programs, each one handling one or a few specific functions. With Tornado, I find it better to have fewer programs which can handle a greater number of functions, since the application is always resident and running. Also, combining similar functions into one app can make better use of common code which can be shared between the various procedures.

What's the same

Despite those differences, all of the basics of web processing which we have already covered up to this point don't change when it comes to processing a web request using Tornado/ISAPI. Information is still passed via the standard variables such as REQUEST_METHOD and QUERY_STRING. Form data from a POST operation is still gathered individually as field=value pairs, and HTML is still written back out to the server using a process similar to the CGI template 'HTMLOut() methods.

So, despite the apparent differences, virtually everything we have discussed still applies to your Tornado based programs. They may have different names, but the functions remain the same.

Using the sample shell application

Instead of templates, I have provided a well commented 'shell' application for use with Tornado/ISAPI. All of the basics are already in place - the screen, OCX event handler, cases to handle the various requests and so on. Like the CGI/Win-CGI templates, all that you need to do is fill in the code to perform whatever actions you require.

To create a new Tornado/ISAPI application, just open the 'shell' application in CW (it is named 'Tdemo.app'), then do a 'File - Save As' and save it under a new name. At that point, you are ready to go with your new app.

Getting and using Tornado

Obviously, in order to test a Tornado application, you will need to have the Tornado OCX and Administrator program installed on your machine. If you do not have it already, you can download it from http://www.eudev.com/tornado.htm. The package comes with complete documentation (including some nice diagrams of how the pieces all fit together), so I am not going to go over it again here. You should, however, study the shell application for examples of how things are done in CW since most of the examples in the Tornado documentation are for Visual Basic.

Summary

Well, that's all for this topic, folks! I hope you've enjoyed taking our little journey through web-land as much as I have, and that it has encouraged you to expand your CW programming horizons out to the World Wide Web. The templates and Tornado examples should give you a huge boost in that direction, and with a bit of perseverance you, too, can wear that web dude (or dudette) hat with confidence.

Next month, a special bonus edition of this series which will focus on the various methods for using FTP (File Transfer Protocol) to transfer files across the Internet, both from within your CW programs and by using other available utilities.

See you then!

- Extra Stuff -

Special credit (drum roll, please...)

I want to give major thanks to Dave Berton, the original author of the Win-CGI and CGI templates which accompany this article. He and a few other hardy souls pioneered the initial work in getting CW to work with Win-CGI and standard CGI, and Dave was generous enough to donate the initial version of these templates to the Clarion community at large. I have modified and enhanced them since then to fit my own needs, but I just can't stress enough how much time they have saved me in the process of developing my own Web applications.

Thanks again, Dave!!

The big disclaimer (no small type here!)

I usually figure people already understand this kind of thing, but just in case...

NOTICE: The templates and examples which accompany this article are intended for educational use and neither I nor Dave Berton make any claims whatsoever as to their completeness or suitability for any actual business use.

In other words, kids, have fun with them, learn from them, use them if you want to (I have many applications running on the web right now based on this stuff) but understand that you DO need to do your own extensive testing to ensure that any program you write using these free tools will function the way you planned (or function at all).

So, words to the wise: never bet the farm on a freebie!<G>

Printer-friendly version

Copyright © 1999-2008 by CoveComm Inc. All Rights Reserved. Reproduction in any form without the express written consent of CoveComm Inc., except as described in the subscription agreement, is prohibited.

Clarion Magazine ISSN 1718-9942