EECS 473 - MP3: Using Flex

Due: March 27, 2001

Update - 3/9/2001

The following tokens are missing from the below description. These tokens will be needed for MP 4 (at least the operators will be needed).

operator: minus -
operator: comma ,
keyword: cin
keyword: cout

Original Write-up

For this assignment, you are to write a Flex program that will take as input a "program" and determine the tokens contained in it. This program will be expanded to allow for the creation of the parser, but can be a simple as wanted/needed to solve the given requirements.

Flex can be found at the following location on the EECS machines

	/usr/local/gnu/bin/flex

You will need to add this to your path, if it does not already exist (and you don't what to type out the entire pathname everytime you wish to run the program).

The tokens for this assignment are as follows:

identifer - a sequence of alphabetic, numeric and underscore characters, not starting with a numeric character.
keyword - an identifer with one of the following values. Note that not all keywords are included in our list.

char class delete else false if int new private public return true void while
character - a single character delimited by matching single quotes ('). Special characters can be represented by multiple characters (that are still delimited by matching single quotes) using escape notation:
- \' - single quote
- \" - double quote
- \? - question mark
- \\ - back slash
- \f - formfeed
- \n - newline
- \r - carriage return
- \t - tab
- \ddd - character with octal value ddd
string - characters delimited by matching double quotes ("). Other special characters may be embedded within a string using the same escape sequence as for characters.
integer - a sequence of digits.
float - two sequences of digits separated by a decimal point (.). Either sequnce of digits (but not both) may be empty.
operator - any of the following. Note that not all operators have been included in our list.

{ } [ ] ( ) ; :
. + * / %
& && || ! = == <
<= > >= != -> >> <<

The flex program is to be given the name of the program/file to analyse through the command line. The program is to print out information (like mp1 did) for each token encountered. When reaching the end of the file, print out the number of total tokens found and the number found of each type of token. The program is to ignore and blanks, tabs, newlines, formfeeds and comments (either /* .. */ style or // style comments) found in the program.

For each token, you are to print out the token name (i.e. identifier, keyword, character, etc.), the token's value, the line number containing the start of the token and the column where the token started. If you encounter a character that is not part of any token, print the token name as "Unknown" and the value as "\xxx" where xxx is the octal value of the character.

The for following input line (assume it is line 24)

     x = 3 + 5.31;

Your program should produce something similar to:

   identifier     x    	24    	 1
   operator       =    	24    	 3
   integer        3    	24  	 5
   operator       +	24	 7
   float	  5.31	24	 9
   operator	  ;	24	13

Your program will be submitted electronically using turnin and must run on the EECS department computers. You must also submit a make file to compile your program. Your program is to be the result of individual work and is expected to be written using good programming style.