|
|
|
CSC sure knows how to throw a party! The St. Patrick's Day party was a lot of fun. When John Buell broke out of the pack of 3 competitors with his fine performance of "H-A-R-R-I-G-A-N" and was awarded Honorable Mention in the Entertainment category, I felt sure they were saving the Third Place award for the MITU/MCCSIM relay crew!
Did you see Bill's latest list of action items? Don is supposed to "Start a trace on the X11R4 shipment", while Steve is supposed to "Build X.11.4 server". Did I miss something here? Poor Nancy has to "Find out why time is messed up"; next week, she'll probably have to "Find out why space is curved".
This memo discusses a number of topics useful to TPOCC software developers: the XDR standard, processing command line options, fast lookups using hash tables, and counting lines of code.
Speaking of parties, I hope you didn't miss the biweekly Alex Measday Roast, euphemistically called a status meeting, that was held a couple of weeks ago. Almost everyone had a roaring good time - Steve was even handing out marshmallows (sharing his thoughts?).
Although dumping on my code is a daily occurrence (and my code core dumps on
itself regularly), I do take umbrage at the suggestion that my code doesn't
do what it says it does; e.g., xnet_write_xdr_string() doesn't
write out an XDR string to the network. Actually,
net_write_xdr_string() does exactly what its name implies.
There was, however, some confusion on the part of myself and others about
Sun's implementation of network-based XDR.
Sun's XDR (eXternal Data Representation) Standard defines common representations for a number of different data types: integers, floating point numbers, character strings, etc. The common representation of data simplifies the exchange of information between dissimilar computers. Note that the XDR Standard per se does not deal with distributed computers communicating over TCP/IP networks. In fact, Sun's Standard C Library provides XDR data streams (1) through the Standard I/O facility (e.g., files and TTY ports), (2) in memory (e.g., shared memory between co-processors with incompatible architectures), and (3) over network connections (e.g., TPOCC).
Sun's network-based XDR (the xdrrec_xxx(3) functions)
differs from the standard I/O- and memory-based XDR in that an additional
protocol is implemented underneath the XDR data stream. Sun's Record Marking
(RM) Standard breaks the XDR data stream up into record fragments; a group of
fragments make up a record. The programmer determines where the record
boundaries fall in the XDR data stream; the number and lengths of the record
fragments are invisible to the programmer.
Each record fragment begins with a four-byte fragment header which specifies
the length of the fragment and if the fragment is the last fragment in the
record (indicated by setting the most significant bit in the header). For
example, a record containing a single XDR string, "PAGE ISECHK2",
would look as follows if dumped:
00000000: 80000010 0000000C 50414745 "........PAGE"
0000000C: 20495345 43484B32 " ISECHK2"
The 0x80000010 header indicates that the incoming 16-byte record fragment is
the last and only fragment in the record. If the string happened to be longer
than the size of the buffer allocated by xdrrec_create(), the
string (and the record) would be strung out over multiple record fragments.
0x0000000C is the length field from the XDR string; the remainder of the
record is the string itself.
It should be pointed out that the Record Marking standard is not part of the XDR standard. Sun documents the Record Marking standard in the Remote Procedure Call (RPC) protocol specification and says: "Note that this record specification is NOT in XDR standard form!" (capitalization and exclamation point by Sun). Ironically, use of the RM standard makes network-based XDR streams incompatible with standard I/O- and memory-based XDR streams, as the following example illustrates.
inetd(8) is a Unix daemon known as the "Internet super-server".
At boot time, inetd reads a list of servers from its
/etc/inetd.conf configuration file and then monitors the servers'
network ports for connection requests. When a request arrives at a port,
inetd accepts the request and fork()/exec()s
the corresponding server program in such a way that the network connection is
attached to file descriptors 0 and 1 (stdin and
stdout) of the server process. Now, assume that the client
program is speaking XDR using the xdrrec_xxx() functions.
If the server program also uses the xdrrec_xxx() functions,
the two programs will get along fine. If, instead, the server program opened
the XDR stream using xdrstdio_create() (it's communicating through
stdin and stdout after all), communications would
break down.
Why did Sun slip the Record Marking protocol in under XDR in its network-based
implementation of XDR? Someone with more smarts than me - sit down, Steve,
this is a rhetorical statement - will have to explain why. The
xdrrec_xxx() routines do provide buffered I/O,
but so does XDR done through the standard I/O facilities. The Record Marking
protocol also allows a program to recover if it gets out of sync at the XDR
level, but what happens if it gets out of sync at the record marking level?
First of all, our Detailed Design document should probably be updated to reference the RPC Record Marking Standard (XDR is already referenced). Second, change your program to use the "real" XDR functions.
The TSTOL parser now talks to Display, State Manager, and everyone else using
XDR records; each record contains exactly one XDR string. The talknet
program has also been updated to use the real XDR routines. The
"-X" option puts talknet in XDR mode, where each line of
input from the operator is sent out across the network as a one-string XDR
record. Since talknet is intended as a diagnostic tool, the data
received back from the network is dumped straight out to the operator's
screen; it is not interpreted as an XDR stream. (The upstart challenger,
talkxdr, has a bug in its network reads, by the way, so use
talknet instead; see the description of the buffered I/O problem
a little later on.)
How do you perform "real" XDR I/O? Two ways: the hard way and the easy way.
The hard way is to call the Sun XDR routines directly. To output a record
of XDR data, set the operation field in the XDR I/O handle to
XDR_ENCODE, call the appropriate XDR routines (e.g.,
xdr_string()) for the data types in the record, and, finally,
call xdrrec_endofrecord() to mark the end of the record and
flush it out to the network. To input a record of XDR data, set the operation
field in the I/O handle to XDR_DECODE, call the appropriate XDR
routines for the expected data types in the incoming record, and then call
xdrrec_skiprecord() to position to the start of the next input
record.
The easy way to perform "real" XDR I/O is to call the
xnet_xxx()
utilities in the TPOCC library (found in source file xnet_util.c):
xnet_answer() - sets up a server to listen for and accept
connection requests.
xnet_call() - on behalf of a client task, requests a network
connection to a server.
xnet_close() - closes a network connection.
xnet_read_string() - inputs a record containing an XDR string
from a network connection.
xnet_write_string() - outputs a record containing an XDR
string to a network connection.
xnet_end_record() - outputs the contents of the current
output record to the network connection and begins a new record.
xnet_next_record() - discards the current input record and
positions to the beginning of the next input record.
xnet_socket() - returns the socket number from the I/O handle.
These utilities provide a simplified interface to XDR functionality. They are particularly useful if you're just passing ASCII strings back and forth, but they're still usable even if you're doing fancy stuff (e.g., non-blocking I/O, obscure or composite data types, etc.).
xnet_answer() combines calls to
net_answer() and
Nancy's stream_svr(). xnet_call() does the same
for net_call().
Each of them returns an I/O handle which is to be passed into subsequent
xnet_xxx() calls. An I/O handle is just a void *
pointer to a svr_stream structure (see stream_svr.c
and strm_svr.h). xnet_socket() returns the socket
number found in the svr_stream structure.
The network connection is set up for blocking I/O and a very long timeout.
If you need to change these parameters, allocate your own
svr_stream structure, do your own net_call() or
net_answer(), and call stream_svr() directly.
The address of your svr_stream structure, cast as a
"void *" pointer, can be passed into the various
xnet_xxx() routines.
xnet_end_record() flushes the contents of the current output
record out to the network connection. xnet_next_record() discards
the contents of the current input record and positions to the beginning of the
next input record. A number of xnet_rtype() and
xnet_wtype() functions (not listed above) retrieve and
store various data types in the input and output buffers, respectively.
xnet_read_string() and xnet_write_string() allow you
to ignore the details of skipping input records and ending output records.
xnet_read_string() retrieves an XDR string from the current input
record and then skips forward to the next input record.
xnet_write_string() adds an XDR string to the current output
record and flushes the record out to the network.
A simple client:
#include <stdio.h>
main ()
{
char buffer[256] ;
int more_input ;
void *handle ;
xnet_call ("host", "server", &handle) ;
for ( ; ; ) {
xnet_write_string (handle, "Hello Server!", 0) ;
xnet_read_string (handle, buffer, sizeof buffer, &more_input) ;
printf ("Client: received \"%s\"\n", buffer) ;
}
}
and a simple server:
#include <stdio.h>
main ()
{
char buffer[256] ;
int more_input, server_socket ;
void *handle ;
xnet_answer ("server", &server_socket, &handle) ;
for ( ; ; ) {
xnet_read_string (handle, buffer, sizeof buffer, &more_input) ;
printf ("Server: received \"%s\"\n", buffer) ;
xnet_write_string (handle, "Hello Client!", 0) ;
}
}
What if your program has to monitor multiple network connections for input?
With network connections carrying non-XDR traffic, you can just wait on a
select(2)() of all your sockets, read a record from an active
socket, and loop back to select() again. This doesn't work for
XDR connections.
select() tells you if any data is waiting to be read from a
socket. When you try to read an XDR record from the socket,
xdrrec_xxx()'s buffered I/O may actually input and save
several XDR records. If you simply return to the select() call
after reading and processing a single record, you might miss the buffered
records that follow (select() won't catch them since they've
already been read). If you're in the middle of a conversation with another
process, the two programs are liable to hang, each waiting on their respective
select() calls.
The solution? If you're using the low-level XDR functions, call
xdrrec_eof() before calling xdrrec_skiprecord().
If you're using the xnet_xxx() utilities, just keep track
of the more_data flag returned by xnet_next_record()
and xnet_read_string().
The following example is a server program that accepts connections from two clients, A and B, and displays any messages the clients send:
#include <errno.h>
#include <stdio.h>
#include <sys/types.h>
main ()
{ char buffer[256] ;
fd_set read_mask ;
int more_input, server_socket ;
void *client_A, *client_B ;
/* Establish connections with clients. */
server_socket= -1 ;
xnet_answer ("server", &server_socket, &client_A) ;
xnet_answer ("server", &server_socket, &client_B) ;
/* Read and display input from clients. */
for ( ; ; ) {
FD_ZERO(&read_mask) ;
FD_SET(xnet_socket (client_A), &read_mask) ;
FD_SET(xnet_socket (client_B), &read_mask) ;
/* Wait for input.*/
while (select (FD_SETSIZE, &read_mask, NULL, NULL, NULL) < 0) {
if (errno == EINTR) continue ;
perror ("Error selecting input.\nselect") ;
exit (errno) ;
}
/* Input from client A? */
if (FD_ISSET(xnet_socket (client_A), &read_mask))
do {
xnet_read_string (client_A, buffer, sizeof buffer, &more_input) ;
printf ("from Client A: \"%s\"\n", buffer) ;
} while (more_input) ;
/* Input from client B? */
if (FD_ISSET(xnet_socket (client_B), &read_mask))
do {
xnet_read_string (client_B, buffer, sizeof buffer, &more_input) ;
printf ("from Client B: \"%s\"\n", buffer) ;
} while (more_input) ;
}
}
At one point during our demonstration last September, everyone took turns re-compiling and re-linking their programs in order to turn debug output on. It had all the earmarks of a How-Many-X-Windows-Programmers-Does-It-Take-To-Open-A-Window situation. Conditionally-compiled debug code is a no-no in many software organizations and I believe Henry Ledgard warns against it in his Programming Proverbs book. The main reason: you're not running the exact same program. Code optimization, memory layout, etc. may be radically altered by the inclusion/exclusion of compilation-dependent debug code.
Fortunately, there's another way - add a global debug flag to your program and set it via a command line option. You might also want to add a global debug output file descriptor, too, so that debug output can easily be redirected to a file:
#include <stdio.h>
int debug = 0 ; /* Off, by default. */
FILE *dbg = stdout ;
main (...)
...
{
...
if (debug) fprintf (dbg, "Debug Text\n") ;
...
}
The additional memory and CPU cycles consumed by the "if (debug)"
code are probably insignificant in most people's programs. If your program
warrants it, you can even set debug to different values to turn
on varying levels of debug.
Okay, how do I set the debug switch from my program's command line? How do I
retrieve command line arguments in general? Very easily, thanks to
getopt(3)(). Although getopt() is available in the
Standard C Library, an improved (but compatible) version is available in the
TPOCC Library. getopt.h, a header file that should be used with
getopt(), is found in the TPOCC include directory.
getopt() scans the command line and returns, on repeated calls,
the options in the line. A Unix command line argument can be one of three
types, as shown in the following example:
% tlmgen -d -m ICE -r 256 history_file
This command line invokes the telemetry generator. "-d" is a
single-letter option that turns debug on. "-m ICE", a
single-letter option that expects an argument, specifies the mission name.
("-r 256", another single-letter option with an argument, sets
the telemetry rate to 256 BPS.) history_file is a non-option argument;
i.e., the argument is not associated with an option.
A list of legal options for a program is passed into getopt()
as a string. The string of legal options for tlmgen is
"dm:r:"; a colon following an option indicates that the option
expects an argument. Each call to getopt() returns the next
option letter from the command line. If the option expects an argument, a
pointer to the argument string is stored in global variable optarg.
If an illegal option is detected, getopt() returns a question
mark, '?'. If a non-option argument is encountered,
getopt() stores a pointer to the argument in optarg
and returns NONOPT as its function value.
The following code processes tlmgen's command line:
#include <stdio.h> /* Standard I/O definitions. */
#include "getopt.h" /* GETOPT(3) definitions. */
int debug = 0 ; /* Default parameters. */
FILE *dbg = stdout ;
char *history_file = "Ancient" ;
char *mission = "Impossible" ;
int rate = 32*1024 ;
main (argc, argv)
int argc ;
char *argv[] ;
{ /* Local variables. */
int errflg, option ;
errflg = 0 ;
while (((option = getopt (argc, argv, "dm:r:")) != NONOPT) ||
(optarg != NULL)) {
switch (option) {
case 'd': debug = -1 ; break ;
case 'm': mission = optarg ; break ;
case 'r': rate = atoi (optarg) ; break ;
case '?': errflg++ ; break ;
case NONOPT:
history_file = optarg ; break ;
default : break ;
}
}
if (errflg) {
fprintf (stderr, "Usage: tlmgen [-d] [-m <mission>] [-r <rate>] [<history_file>]\n") ;
exit (-1) ;
}
... continue with regular processing ...
}
The code above can be used as a template for other programs - just change the string of legal options and the case statements for the individual options.
If you're working with VxWorks, don't despair - the Unix command line is
easily emulated. sgetopt(), another routine in the TPOCC
library, scans an arbitrary string for options. Enclose all the desired
options in a string when spawning your VxWorks program:
-> sp tlmgen, "-d -m ICE -r 256 history_file"
sgetopt() is then used to parse the options:
... #include VxWorks header files ...
#define NONOPT ' ' /* SGETOPT definition. */
extern char *str_dupl () ; /* External function. */
int debug = 0 ; /* Default parameters. */
FILE *dbg = stdout ;
char *history_file = "Ancient" ;
char *mission = "Impossible" ;
int rate = 32*1024 ;
main (argc, argv)
int argc ;
char *argv[] ;
{
tlmgen (argv[1]) ;
}
tlmgen (command_line)
char *command_line ;
{
char *optarg, option ;
int errflg, next ;
errflg = 0 ;
if (command_line == NULL) command_line = "" ;
next = 1 ;
while (sgetopt (command_line, "dm:r:", &next, &option, &optarg)) {
switch (option) {
case 'd': debug = -1 ; break ;
case 'm': mission = str_dupl (optarg, 0) ; break ;
case 'r': rate = atoi (optarg) ; break ;
case '?': errflg++ ; break ;
case NONOPT:
history_file = str_dupl (optarg, 0) ; break ;
default : break ;
}
}
... etc., etc., ...
}
Note that if you use sgetopt(), you must duplicate the string
pointed to by optarg if you want to save it. See the
data_setter program in the TPOCC tools directory for an
example of combining getopt() and sgetopt() code
using #ifdefs.
Should you ever have the need to perform fast lookups of mnemonics or whatever, there are now some general purpose hash table functions in the TPOCC library. While they are not worthy to stand in the shadow of the original TPOCC shared memory hash functions, you might find these new functions useful:
hash_create() - creates an empty hash table.
hash_destroy() - deletes a hash table.
hash_dump() - prints out the contents of a hash table.
hash_add() - adds a key-data pair to a hash table.
hash_delete() - deletes a key-data pair from a hash table.
hash_search() - locates a key in a hash table and returns the
data value associated with the key.
The advantage of the TPOCC hash functions over the Standard C Library
hsearch(3)() functions is that the TPOCC functions are portable
(e.g., they'll run under VxWorks) and they allow a program to have more than
one hash table active at a time.
The TSTOL parser currently uses two hash tables: one for reserved keywords and one for directive keywords. The system variable utilities in the TPOCC library have also been converted to use the new hash functions, which means system variable lookups can be hashed under VxWorks, too. The hash functions work pretty well: a mean number of 1.3 or 1.4 string comparisons is required to lookup a key in a hash table built for 700+ ICE system variables and telemetry mnemonics.
Ever wonder how much you really stretched the truth in the lines-of-code
estimates you gave to Ed? Put the following aliases in your .cshrc
file and find out:
alias loc "echo -n \!* ':tab' ; egrep '^#|;|{|%' \!* | wc -l"
alias loctot awk "'"'{s+=$3} END {print s}'"'" loc.count
(tab in the first alias is a tab character; "-l" is
dash-ell, not dash-one.) loc counts the number of
lines of C code in a file; it works on C source files, header files, LEX
files, and YACC files. To run loc on all the files in your directory,
type in the following from the C Shell ("%" and "?"
are C Shell prompts):
% foreach file (*.c *.h)
? (loc $file) >>! loc.count
? end
The code count for each file is written seriatim to file
loc.count. (By leaving out ">>! loc.count",
you can have the code counts output to your screen.) Type loctot
to sum the individual code counts and display a total.
loc does a reasonable job of counting lines of code - it'll even work on Gordon's code. loctot did, however, core dump with an underflow error when I tried running it on the GMT server.