The Common Gateway Interface (CGI) is a standard that governs how external applications are interfaced with Web servers. The reasoning behind the invention of CGI is simple: Without it, the HTTP specification and all Web servers would have become a
patchwork of ad-hoc extensions.
CGI provides a way to write programs that will run on the server when they are invoked by the client Web browser through HTML code. These programs can be written in C language. But C is just one possibility. For a discussion of other options, see the
section titled "Choosing a CGI Programming Language."
At this point, the astute reader might have noticed that there are no fewer than four areas of programming prowess needed to get this dog to hunt! Just count 'em: CGI, HTTP, HTML, and C (or some other programming language). And just for good measure,
you might want to throw in the Win32 API and SQL, depending on what your Web program will actually do after you finish laying the required groundwork.
The reason we choose to run this challenging gauntlet is that CGI opens the door to great new opportunities. CGI programs are often associated with Web forms. When the user finishes filling out an HTML form and submits it, the data stream that is
returned to the server is called the form data. Keep in mind that just because you send a blank HTML form to the client Web browser, nothing is going to happen with the form data when it is submitted—unless you make it happen. (In fact, the
form data would just land in the bit bucket if not for CGI.) For example, CGI is a necessity if you want to save the form data into a database on the server. Perhaps the form data should be e-mailed to the Webmaster or some other party. Maybe the intent of
the form is to have some data faxed or e-mailed back to the client. Or the form could be used to obtain a database query from the user, which is then sent to a database engine, before the formatted results are finally returned to the client as an HTML
file. These are just some of the possibilities available to anyone brave enough to master the details of client/server Web programming with CGI. (Actually, tools exist so that much of this can be done without programming—and we're going to tell you
about several of them.)
Although all of these things can be accomplished with traditional programming, doing it on the Web makes applications platform-independent, distributed, easier to update, and easier to develop.
The purpose of this chapter is to give you the fundamentals of CGI and show you a couple of simple CGI examples. Programming knowledge is not required, but it would be helpful. If you don't yet know about programming, you might just want to skim this
chapter to get a glimpse of the possibilities. On the other hand, if you want to utilize CGI at your Web site, this chapter lays the groundwork for Part V, which covers CGI in much more depth.
CGI programs are also called CGI scripts or applications. The reason they are called scripts is that they can be written in Perl or at the UNIX command shell, in which case they are interpreted
rather than compiled. When the C language or Visual Basic is used for CGI, the terms CGI program or CGI application are preferred to the term CGI script because those languages are not interpreted in the traditional sense
of script files. Some people just refer to all such things as CGIs.
Figure 11.1 shows a high-level overview of how CGI forms-processing works. There are many other details of HTTP and TCP/IP than what are shown here, but we omit those in order to concentrate on the basic concepts of CGI.
Figure 11.1. How CGI processes Web forms.
The annotated steps corresponding to Figure 11.1 follow. (We assume you are familiar with the way an HTML file gets created and displayed in the Web browser, which is where we pick up the story with step 1.)
Allowing any person with a Web browser to execute applications on your server is a security concern. It is a good idea to ensure that all the CGI applications are isolated to one directory and that no one else
has access to that directory. On the Final Bit Solutions server, all CGI applications are kept in a directory called cgi-bin under the server root. Be careful about using public-domain CGI applications that have not been tested over time to be secure.
Both the Purveyor Desktop server and the FolkWeb server conform to the CGI 1.1 standard. The CGI application must be a console-mode program and be located within the HTTP data directory tree. By saying console-mode, we mean that CGI applications
cannot be Windows API programs, or GUI programs.
The Purveyor Desktop server supports another form of CGI, which Process Software calls buffered CGI. Buffered CGI does permit you to develop Windows API programs in languages such as Delphi or Visual Basic. Similarly, the FolkWeb server supports
a popular standard on the Web called WinCGI 1.2. (See Appendix I.) WinCGI is very well suited for developing Web applications in Visual Basic.
In UNIX, which is where the Web got its start, CGI applications are frequently written in C, Perl, or the UNIX shell command language. In Windows NT, you can use C/C++ or Perl with most servers, and Visual Basic with some. Many Windows NT Webmasters run
a public-domain Perl 4 interpreter for CGI and site statistics. Perl 5, which includes some nice object-oriented extensions, has recently arrived on the scene. We will walk you through a really cool Perl 5 program in Chapter 21 that will help you track
statistics on your Web site.
Both Perl and C have their advocacy camps. Perl offers great file and string handling, and the code is fairly easy to write and modify. On the other hand, because C is a compiled language, it offers better efficiency, both from the optimization of the
compiled code and the fact that the interpreter is not launched for every client submission of form data. In addition, many claim that compiled programs provide better security than scripts because hackers can more easily modify the text of a script just
before its execution.
We will be using C language in this chapter and in Chapter 22, "Power CGI Programming with C/C++." The programs have been compiled with Borland C++ 4.x and Visual C++ 2.x. The program in this chapter is only a demonstration; but in Chapter 22,
we'll show you how to construct a very useful CGI application. In Chapter 23 we will compile a similar program in C++ and integrate it with Visual Basic to talk to any Access or ODBC database.
It's okay if you don't plan to learn programming. Most of the examples are already compiled on the CD-ROM and will run without you knowing how to program.
The server uses environment variables to pass information to the CGI application. The environment variables are set after the HTTP GET or POST request is received (see the next section) and before the server executes the CGI application. You will find
that most environment variables are fairly standard from server to server, but be aware that there are some differences. Nothing stops the makers of a Web server from adding environment variables for their own use.
The CGI standard specifies certain environment variables that are used for conveying information to a CGI script. The following subset of those environment variables are supported by most HTTP servers. Please don't despair; most CGI programs don't need
to use all of these environment variables:
CONTENT_LENGTH The length of the content as given by the client.
CONTENT_TYPE For queries that have attached information, such as POST and PUT, this is the content type of the data.
GATEWAY_INTERFACE The revision of the CGI specification to which this server complies. Format: CGI/revision
HTTP_ACCEPT The MIME types that the client will accept. Format: type/subtype, type/subtype.
PATH_INFO The extra path information, as given by the client. This enables scripts to be accessed by their virtual pathname.
QUERY_STRING The information that follows the ? in the URL that referenced this script. This is the query information.
REMOTE_ADDR The IP address of the remote host making the request.
REQUEST_METHOD The method with which the request was made, such as GET, HEAD, and POST.
SCRIPT_NAME A virtual path to the script being executed.
SERVER_NAME The server's hostname, DNS alias, or IP address.
SERVER_PROTOCOL The name and revision of the information protocol this request came in with. Format: protocol/revision.
SERVER_PORT The port number to which the request was sent.
SERVER_SOFTWARE The name and version of the server software answering the request. Format: name/version.
Other HTTP headers received from the client are available in environment variables of the form HTTP_*. For instance, the User-Agent: header value is available in HTTP_USER_AGENT. Note that - in the header names is replaced by _ in the corresponding
environment variable names. An understanding of the HTTP specification is probably a prerequisite to a full understanding of the purpose of some of these environment variables.
The CGI application accesses information about how it was invoked through the environment variables initialized by the Web server; it reads any information supplied by the client (in a POST request) via stdin and sends output to the client through
stdout. It's actually pretty simple, once you get the hang of it.
GET and POST are two HTTP methods of sending form data to the Web server. When you write a form in HTML 2.0, you should specify which HTTP method the browser will use when the form data is sent back to the server.
Take a look at a short block of HTML code that comprises a complete form. See Listing 11.1. The line numbers are not a part of the HTML code. Note in line 2 that the form is using Method="POST". We could just as easily change it to
"GET". The main difference between GET and POST is that the CGI application will receive the POST data by reading the stdin device, whereas GET data would be received on the command line and in the QUERY_STRING environment variable.
1. <HTML><HEAD><TITLE>Simple Form</TITLE></HEAD><BODY> 2. <FORM Method="POST" 3. Action="http://url\cgi-bin\prog.exe"> 4. Your Name: <INPUT Name="user" SIZE="30"><P> 5. <INPUT Type=submit Value="Click here to send"> 6. </FORM></BODY></HTML>
Usually, your forms will be much more complex than this one, which only contains one input field. Because many operating systems impose some limit to the length of the command line, it is usually best to use POST. On the other hand, if you know your
form data is small, you can use GET.
In the case of a GET request (or ISINDEX, discussed in Chapter 19), the form data will be on the command line and in the QUERY_STRING environment variable. The command line will contain a question mark after the application name as the delimiter that
marks the beginning of the form data. For example, suppose you change the HTML code in Listing 11.1 to use Method="GET", and the user types in the string User's Name in the text field named user.
The command line of the CGI application would look like this:
\cgi-bin\prog.exe?user=User%27s+Name
The QUERY_STRING environment variable would look like this:
user=User%27s+Name.
Our first observation is that this looks somewhat strange. Our second observation is that the QUERY_STRING data appears somewhat more friendly looking than the command line data, so let's definitely work with the QUERY_STRING.
Now let's try to figure out what's going on with all those funny characters. Recall from line 4 of Listing 11.1 that we named the input field user. Now that label is being sent back to us as the first word of QUERY_STRING. Everything after the equals
sign in the QUERY_STRING represents the data that the user typed into that particular field. Because more than one field could be used, each one must be named uniquely in the HTML form and in the QUERY_STRING data that is sent back to the CGI application.
Remember that our example assumes the user typed User's Name with no period on the end. (If he had typed a period, that would be another story—more about that later.) Checking the QUERY_STRING above, we notice that we almost have exactly what the
user typed, except for the %27, which replaces the apostrophe and the plus sign, which replaces the space character.
HTTP calls for these translations because of operating system conventions for reserved characters in filenames. The same mechanism is used by HTTP to pass URLs, so the server needs to be able to distinguish between the two.
The percent sign is a hex escape character, and the two digits that follow it are used to indicate the ASCII code of a reserved character. Here, the apostrophe sign has a hex code of 27. If a period were typed by the user, it would be replaced by %2E.
Not all servers encode these characters, because whether they are reserved or not depends on the operating system. For example, the apostrophe and the period are legal in some UNIX systems.
The plus sign is simply the convention for encoding space characters. Another common translation is the dash character encoded as an underscore.
Finally, if there were other input fields in the HTML form, they would follow the data of the user field. Each name=value pair would be separated by an ampersand character.
Table 11.1 is a quick review of the special characters you will come across in CGI. Some of these conventions make up what is known as URL-encoding.
|
+ (plus sign) |
Used in place of space characters in user input. |
|
= (equals sign) |
Used to separate the field name from the field value. |
|
? (question mark) |
Used to mark the beginning of the form data on the command line. |
|
_ (underscore) |
Used to replace dash characters. |
|
% (percent sign) |
Used to encode reserved ASCII characters, followed by two hex digits. |
|
& (ampersand) |
Used as the boundary between name/value pairs for each field in the HTML form. |
|
|
|
Recall that QUERY_STRING is not used for the POST method. Because POST is probably more typical, you need to understand how to read stdin to retrieve form data.
First, the server will set the CONTENT_LENGTH environment variable to tell how many bytes to read from stdin. You must not read more than that amount.
Other than that, the POST-invoked program will read and parse the form data from the stdin device instead of the QUERY_STRING environment variable. Either way, you'll want to have some standard routines in C or Perl to help you perform standard
decoding. The C programs in this chapter (as well as Chapter 22) include several useful functions for that purpose. Feel free to customize them and use them in your own programs.
When the CGI application is done parsing and processing the input data, it must send a reply to the server. The server will forward the reply to the client after applying a header as per the rules of the HyperText Transfer Protocol.
The server will be listening to the stdout device of the CGI application while the latter is executing. The CGI program can generate HTML code on the fly or refer the server to another document that it would like to have sent instead. Either you want to
compose an HTML document on the fly, or you want to refer to another document—via HTTP, FTP, or Gopher—anywhere on the Web.
See the section below titled "A Simple CGI Example in HTML and C" for all the details about composing an HTML response document from within the CGI application.
If you want the server to send another document that already exists, you can use the Location code. In C, you would execute a printf that looks something like this:
printf("Location: ftp://FQDN/dir/filename.txt\n\n");
You must follow the header information with a blank line; thus, there are two newline characters in the example.
It is very important that your CGI program prints out an extra blank line after the HTTP header and before the document contents. This is a common source of trouble when trying to debug CGI systems.
How To Learn More About CGI
This section covers a complete CGI transaction, from server to client, back to server, and back to client. This example is not intended to be useful, except as a template from which you could build a more sophisticated CGI application.
The CGI system you are going to build here starts with an HTML file that contains a form. When the form data is submitted by the user, the server will determine that the Action attribute for the form refers to a CGI application. The server will start
the application and send it the form data on stdin. Then the server will listen for stdout from the CGI application.
The CGI program is written in C. The program will show you how to retrieve the form data, parse it, and send back an HTML document. The HTML response is constructed within the CGI application because you should embed part of the form data in your
response. You don't always have to create HTML on the fly from inside the CGI program, but doing so will make your Web pages more dynamic.
In order to demonstrate CGI, we need to start with an HTML page that contains a URL to a CGI application. See Figure 11.2 for how the data entry form appears in the browser as the user is filling it out.
Figure 11.2. The data entry form that the user fills out.
The HTML code that gets the ball rolling with our sample CGI program is shown in Listing 11.2. This file, and the C program below, are available on the CD-ROM if you want to experiment. Note that you will want to change the ULR in the FORM ACTION to
refer to your site (or use localhost).
<HTML> <HEAD> <!-- Created: 8/21/95 7:48:53 PM --> <TITLE>CGI Application Example</TITLE> </HEAD> <BODY> <H1>CGI Application Example</H1> This is an example of a cgi application handling the data from a form. <BR> <FORM ACTION="http://www.fbsolutions.com/cgi-bin/cgisamp.exe" METHOD="Post"> Please enter your first name: <INPUT NAME="name" TYPE="text"><p> <input type=submit value="When done, click here!"> </FORM> </BODY> </HTML>
Before we show you the C program that will process the form data, we'll discuss the output of the C program. Listing 11.3 is the HTML code that is sent back to the client after the server obtains it from stdout of the CGI application.
<HEAD><TITLE>Submitted OK</TITLE></HEAD> <BODY><h2>The information you supplied has been accepted.<br> Thank You Scott</h2> <h3><A href="http://www.fbsolutions.com/book/cgisamp.htm">[Return]</a></h3></BODY>
Figure 11.3 shows the browser on the client side after the CGI application has finished processing the form data. Note in Figure 11.3 below (and Listing 11.3) that the HTML response sent by the CGI application is customized for each set of form data; it
includes the name the user supplied.
Figure 11.3. The result of the CGI application as seen by the client.
Listing 11.4 shows the complete C program called cgisamp, which is executed by the server when the client submits the form data. Here is a quick list of the five functions in cgisamp:
/***************************************************************************
* File: cgisamp.c
*
* Use: CGI Example Script.
*
* Notes: Assumes it is invoked from a form and that REQUEST_METHOD is POST.
* Ensure that you compile this script as a console mode app.
*
* This script is a modified version of the script that comes with EMWAC
* HTTPS.
*
* Date: 8/21/95
* Christopher L. T. Brown clbrown@netcom.com
*
***************************************************************************/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include <io.h>
char InputBuffer[4096];
static char * field;
static char * name;
/* Convert all cOld characters */
/* in cStr into cNew characters. */
void strcvrt(char *cStr, char cOld, char cNew)
{
int i = 0;
while(cStr[i])
{
if(cStr[i] == cOld)
cStr[i] = cNew;
i++;
}
}
/* The string starts with two hex */
/* characters. Return an integer */
/* formed from them. */
static int TwoHex2Int(char *pC)
{
int Hi, Lo, Result;
Hi = pC[0];
if('0' <= Hi && Hi <= '9')
Hi -= '0';
else if('a' <= Hi && Hi <= 'f')
Hi -= ('a' - 10);
else if('A' <= Hi && Hi <= 'F')
Hi -= ('A' - 10);
Lo = pC[1];
if('0' <= Lo && Lo <= '9')
Lo -= '0';
else if('a' <= Lo && Lo <= 'f')
Lo -= ('a' - 10);
else if('A' <= Lo && Lo <= 'F')
Lo -= ('A' - 10);
Result = Lo + 16 * Hi;
return(Result);
}
/* Decode the given string in-place */
/* by expanding %XX escapes. */
void urlDecode(char *p)
{
char *pD = p;
while(*p)
{
if (*p == '%') /* Escape: next 2 chars are hex */
{ /* representation of the actual character.*/
p++;
if(isxdigit(p[0]) && isxdigit(p[1]))
{
*pD++ = (char)TwoHex2Int(p);
p += 2;
}
}
else
*pD++ = *p++;
}
*pD = '\0';
}
/* Parse out and store field=value items. */
/* Don't use strtok! */
void StoreField(char *f, char *Item)
{
char *p;
p = strchr(Item, '=');
*p++ = '\0';
urlDecode(Item);
urlDecode(p);
strcvrt(p, '\n', ' ');
strcvrt(p, '+', ' '); /* Get rid of those nasty +'s */
field = f; /* Hold on to the field just in case. */
name = p; /* Hold on to the name to print*/
}
int main(void)
{
int ContentLength, x, i;
char *p,
*pRequestMethod,
*URL,
*f;
/* Turn buffering off for stdin.*/
setvbuf(stdin, NULL, _IONBF, 0);
/* Tell the client what we're going to send */
printf("Content-type: text/html\n\n");
/* What method were we invoked through? */
pRequestMethod = getenv("REQUEST_METHOD");
/* Get the data from the client */
if(strcmp(pRequestMethod,"POST") == 0)
{
/* according to the requested method.*/
/* Read in the data from the client. */
p = getenv("CONTENT_LENGTH");
if(p != NULL)
ContentLength = atoi(p);
else
ContentLength = 0;
if(ContentLength > sizeof(InputBuffer) -1)
ContentLength = sizeof(InputBuffer) -1;
i = 0;
while(i < ContentLength)
{
x = fgetc(stdin);
if(x == EOF)
break;
InputBuffer[i++] = x;
}
InputBuffer[i] = '\0';
ContentLength = i;
p = getenv("CONTENT_TYPE");
if(p == NULL)
return(0);
if(strcmp(p, "application/x-www-form-urlencoded") == 0)
{
p = strtok(InputBuffer, "&"); /* Parse the data */
while(p != NULL)
{
StoreField(f, p);
p = strtok(NULL, "&");
}
}
}
URL = getenv("HTTP_REFERER"); /* What url called me.*/
printf("<HEAD><TITLE>Submitted OK</TITLE></HEAD>\n");
printf("<BODY><h2>The information you supplied has been accepted.");
printf("<br> Thank You %s</h2>\n", name);
printf("<h3><A href=\"%s\">[Return]</a></h3></BODY>\n", URL);
return(0);
}
Notice the calls in the main routine to the C library function getenv. That is how the program can determine if the REQUEST_METHOD is equal to POST and how many bytes it should read by checking CONTENT_LENGTH.
Another very important point to make about the main function is that it must output a partial HTTP header to go with the HTML document that it creates. This line appears near the top of the function:
printf("Content-type: text/html\n\n");
You might want to add error handling later, in which case you would probably create an alternative HTML response document. This HTTP header would need to be printed in any case. The CGI convention requires that the header be followed by a blank line
before the HTML code that is sent. That is why the printf includes two newlines at the end. Please forgive our frequent reminders, but this is an important point.
Content type indicates a MIME encoding that tells the client browser that the data stream to follow is HTML code in ASCII format. There are several standard MIME encoding types. See the CGI specification or the Purveyor Desktop Web Server documentation
for further information.
There is a standard MIME type for plain ASCII text, Content type: text/plain. This is useful in a trivial but interesting example of CGI, which is often used as proof that the Web server and CGI are installed and running properly. The idea is to invoke
a DOS batch file that echoes the values of the CGI environment variables on the server back to the Web browser. All you need to do is save the following text into a file named test.cmd in your cgi-bin directory:
echo content-type: text/plain
echo.
echo script_name = %script_name%
Then write a line in your home page that links to test.cmd. The program should tell you its name. You could easily modify this to output other environment variables as well.
Getting CGI systems to work properly obviously requires the ability to integrate several sophisticated tools. And what should a good software engineer do when faced with the challenge of building a complex system? Establish clear milestones to reach the
overall goal, build the software one piece at a time (preferably as black boxes with as few interfaces as possible), and test each module separately as you go to prove that the milestones are met successfully.
For example, test the HTML form independently from the CGI program. You might even take the time to build a test environment for the CGI application so that you can verify its input/output completely independent of any interaction with the Web server.
Doing this could yield a great payback when it comes time to debug or enhance the system, especially if it is a large application or if it interfaces with a database. The goal is to reduce the edit/compile/link/test cycle down to as tight a loop as
possible. A test environment that doesn't involve running the server, launching the browser, and filling out the form will yield significant time savings over the long run.
Now that we have covered all the basics of CGI, Part V will get into several more realistic examples. The programs we will build in Chapters 21 through 24 are already compiled on the CD and can provide you with very useful Web applications, even without you getting involved in programming.