.TH FLEX 1 "26 May 1990" "Version 2.3"

.SH NAME

flexdoc - fast lexical analyzer generator

.SH SYNOPSIS

.B flex

.B [-bcdfinpstvFILT8 -C[efmF] -Sskeleton]

.I [filename ...]

.SH DESCRIPTION

.I flex

is a tool for generating

.I scanners:

programs which recognized lexical patterns in text.

.I flex

reads

the given input files, or its standard input if no file names are given,

for a description of a scanner to generate. The description is in

the form of pairs

of regular expressions and C code, called

.I rules. flex

generates as output a C source file,

.B lex.yy.c,

which defines a routine

.B yylex().

This file is compiled and linked with the

.B -lfl

library to produce an executable. When the executable is run,

it analyzes its input for occurrences

of the regular expressions. Whenever it finds one, it executes

the corresponding C code.

.SH SOME SIMPLE EXAMPLES

.LP

First some simple examples to get the flavor of how one uses

.I flex.

The following

.I flex

input specifies a scanner which whenever it encounters the string

"username" will replace it with the user's login name:

.nf

%%

username printf( "%s", getlogin() );

.fi

By default, any text not matched by a

.I flex

scanner

is copied to the output, so the net effect of this scanner is

to copy its input file to its output with each occurrence

of "username" expanded.

In this input, there is just one rule. "username" is the

.I pattern

and the "printf" is the

.I action.

The "%%" marks the beginning of the rules.

.LP

Here's another simple example:

.nf

int num_lines = 0, num_chars = 0;

%%

\\n ++num_lines; ++num_chars;

. ++num_chars;

%%

main()

{

yylex();

printf( "# of lines = %d, # of chars = %d\\n",

num_lines, num_chars );

}

.fi

This scanner counts the number of characters and the number

of lines in its input (it produces no output other than the

final report on the counts). The first line

declares two globals, "num_lines" and "num_chars", which are accessible

both inside

.B yylex()

and in the

.B main()

routine declared after the second "%%". There are two rules, one

which matches a newline ("\\n") and increments both the line count and

the character count, and one which matches any character other than

a newline (indicated by the "." regular expression).

.LP

A somewhat more complicated example:

.nf

/* scanner for a toy Pascal-like language */

%{

/* need this for the call to atof() below */

#include <math.h>

%}

DIGIT [0-9]

ID [a-z][a-z0-9]*

%%

{DIGIT}+ {

printf( "An integer: %s (%d)\\n", yytext,

atoi( yytext ) );

}

{DIGIT}+"."{DIGIT}* {

printf( "A float: %s (%g)\\n", yytext,

atof( yytext ) );

}

if|then|begin|end|procedure|function {

printf( "A keyword: %s\\n", yytext );

}

{ID} printf( "An identifier: %s\\n", yytext );

"+"|"-"|"*"|"/" printf( "An operator: %s\\n", yytext );

"{"[^}\\n]*"}" /* eat up one-line comments */

[ \\t\\n]+ /* eat up whitespace */

. printf( "Unrecognized character: %s\\n", yytext );

%%

main( argc, argv )

int argc;

char **argv;

{

++argv, --argc; /* skip over program name */

if ( argc > 0 )

yyin = fopen( argv[0], "r" );

else

yyin = stdin;

yylex();

}

.fi

This is the beginnings of a simple scanner for a language like

Pascal. It identifies different types of

.I tokens

and reports on what it has seen.

.LP

The details of this example will be explained in the following

sections.

.SH FORMAT OF THE INPUT FILE

The

.I flex

input file consists of three sections, separated by a line with just

.B %%

in it:

.nf

definitions

%%

rules

%%

user code

.fi

The

.I definitions

section contains declarations of simple

.I name

definitions to simplify the scanner specification, and declarations of

.I start conditions,

which are explained in a later section.

.LP

Name definitions have the form:

.nf

name definition

.fi

The "name" is a word beginning with a letter or an underscore ('_')

followed by zero or more letters, digits, '_', or '-' (dash).

The definition is taken to begin at the first non-white-space character

following the name and continuing to the end of the line.

The definition can subsequently be referred to using "{name}", which

will expand to "(definition)". For example,

.nf

DIGIT [0-9]

ID [a-z][a-z0-9]*

.fi

defines "DIGIT" to be a regular expression which matches a

single digit, and

"ID" to be a regular expression which matches a letter

followed by zero-or-more letters-or-digits.

A subsequent reference to

.nf

{DIGIT}+"."{DIGIT}*

.fi

is identical to

.nf

([0-9])+"."([0-9])*

.fi

and matches one-or-more digits followed by a '.' followed

by zero-or-more digits.

.LP

The

.I rules

section of the

.I flex

input contains a series of rules of the form:

.nf

pattern action

.fi

where the pattern must be unindented and the action must begin

on the same line.

.LP

See below for a further description of patterns and actions.

.LP

Finally, the user code section is simply copied to

.B lex.yy.c

verbatim.

It is used for companion routines which call or are called

by the scanner. The presence of this section is optional;

if it is missing, the second

.B %%

in the input file may be skipped, too.

.LP

In the definitions and rules sections, any

.I indented

text or text enclosed in

.B %{

and

.B %}

is copied verbatim to the output (with the %{}'s removed).

The %{}'s must appear unindented on lines by themselves.

.LP

In the rules section,

any indented or %{} text appearing before the

first rule may be used to declare variables

which are local to the scanning routine and (after the declarations)

code which is to be executed whenever the scanning routine is entered.

Other indented or %{} text in the rule section is still copied to the output,

but its meaning is not well-defined and it may well cause compile-time

errors (this feature is present for

.I POSIX

compliance; see below for other such features).

.LP

In the definitions section, an unindented comment (i.e., a line

beginning with "/*") is also copied verbatim to the output up

to the next "*/". Also, any line in the definitions section

beginning with '#' is ignored, though this style of comment is

deprecated and may go away in the future.

.SH PATTERNS

The patterns in the input are written using an extended set of regular

expressions. These are:

.nf

x match the character 'x'

. any character except newline

[xyz] a "character class"; in this case, the pattern

matches either an 'x', a 'y', or a 'z'

[abj-oZ] a "character class" with a range in it; matches

an 'a', a 'b', any letter from 'j' through 'o',

or a 'Z'

[^A-Z] a "negated character class", i.e., any character

but those in the class. In this case, any

character EXCEPT an uppercase letter.

[^A-Z\\n] any character EXCEPT an uppercase letter or

a newline

r* zero or more r's, where r is any regular expression

r+ one or more r's

r? zero or one r's (that is, "an optional r")

r{2,5} anywhere from two to five r's

r{2,} two or more r's

r{4} exactly 4 r's

{name} the expansion of the "name" definition

(see above)

"[xyz]\\"foo"

the literal string: [xyz]"foo

\\X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',

then the ANSI-C interpretation of \\x.

Otherwise, a literal 'X' (used to escape

operators such as '*')

\\123 the character with octal value 123

\\x2a the character with hexadecimal value 2a

(r) match an r; parentheses are used to override

precedence (see below)

rs the regular expression r followed by the

regular expression s; called "concatenation"

r|s either an r or an s

r/s an r but only if it is followed by an s. The

s is not part of the matched text. This type

of pattern is called as "trailing context".

^r an r, but only at the beginning of a line

r$ an r, but only at the end of a line. Equivalent

to "r/\\n".

<s>r an r, but only in start condition s (see

below for discussion of start conditions)

<s1,s2,s3>r

same, but in any of start conditions s1,

s2, or s3

<<EOF>> an end-of-file

<s1,s2><<EOF>>

an end-of-file when in start condition s1 or s2

.fi

The regular expressions listed above are grouped according to

precedence, from highest precedence at the top to lowest at the bottom.

Those grouped together have equal precedence. For example,

.nf

foo|bar*

.fi

is the same as

.nf

(foo)|(ba(r*))

.fi

since the '*' operator has higher precedence than concatenation,

and concatenation higher than alternation ('|'). This pattern

therefore matches

.I either

the string "foo"

.I or

the string "ba" followed by zero-or-more r's.

To match "foo" or zero-or-more "bar"'s, use:

.nf

foo|(bar)*

.fi

and to match zero-or-more "foo"'s-or-"bar"'s:

.nf

(foo|bar)*

.fi

.LP

Some notes on patterns:

.IP -

A negated character class such as the example "[^A-Z]"

above

.I will match a newline

unless "\\n" (or an equivalent escape sequence) is one of the

characters explicitly present in the negated character class

(e.g., "[^A-Z\\n]"). This is unlike how many other regular

expression tools treat negated character classes, but unfortunately

the inconsistency is historically entrenched.

Matching newlines means that a pattern like [^"]* can match an entire

input (overflowing the scanner's input buffer) unless there's another

quote in the input.

.IP -

A rule can have at most one instance of trailing context (the '/' operator

or the '$' operator). The start condition, '^', and "<<EOF>>" patterns

can only occur at the beginning of a pattern, and, as well as with '/' and '$',

cannot be grouped inside parentheses. A '^' which does not occur at

the beginning of a rule or a '$' which does not occur at the end of

a rule loses its special properties and is treated as a normal character.

.IP

The following are illegal:

.nf

foo/bar$

<sc1>foo<sc2>bar

.fi

Note that the first of these, can be written "foo/bar\\n".

.IP

The following will result in '$' or '^' being treated as a normal character:

.nf

foo|(bar$)

foo|^bar

.fi

If what's wanted is a "foo" or a bar-followed-by-a-newline, the following

could be used (the special '|' action is explained below):

.nf

foo |

bar$ /* action goes here */

.fi

A similar trick will work for matching a foo or a

bar-at-the-beginning-of-a-line.

.SH HOW THE INPUT IS MATCHED

When the generated scanner is run, it analyzes its input looking

for strings which match any of its patterns. If it finds more than

one match, it takes the one matching the most text (for trailing

context rules, this includes the length of the trailing part, even

though it will then be returned to the input). If it finds two

or more matches of the same length, the

rule listed first in the

.I flex

input file is chosen.

.LP

Once the match is determined, the text corresponding to the match

(called the

.I token)

is made available in the global character pointer

.B yytext,

and its length in the global integer

.B yyleng.

The

.I action

corresponding to the matched pattern is then executed (a more

detailed description of actions follows), and then the remaining

input is scanned for another match.

.LP

If no match is found, then the

.I default rule

is executed: the next character in the input is considered matched and

copied to the standard output. Thus, the simplest legal

.I flex

input is:

.nf

%%

.fi

which generates a scanner that simply copies its input (one character

at a time) to its output.

.SH ACTIONS

Each pattern in a rule has a corresponding action, which can be any

arbitrary C statement. The pattern ends at the first non-escaped

whitespace character; the remainder of the line is its action. If the

action is empty, then when the pattern is matched the input token

is simply discarded. For example, here is the specification for a program

which deletes all occurrences of "zap me" from its input:

.nf

%%

"zap me"

.fi

(It will copy all other characters in the input to the output since

they will be matched by the default rule.)

.LP

Here is a program which compresses multiple blanks and tabs down to

a single blank, and throws away whitespace found at the end of a line:

.nf

%%

[ \\t]+ putchar( ' ' );

[ \\t]+$ /* ignore this token */

.fi

.LP

If the action contains a '{', then the action spans till the balancing '}'

is found, and the action may cross multiple lines.

.I flex

knows about C strings and comments and won't be fooled by braces found

within them, but also allows actions to begin with

.B %{

and will consider the action to be all the text up to the next

.B %}

(regardless of ordinary braces inside the action).

.LP

An action consisting solely of a vertical bar ('|') means "same as

the action for the next rule." See below for an illustration.

.LP

Actions can include arbitrary C code, including

.B return

statements to return a value to whatever routine called

.B yylex().

Each time

.B yylex()

is called it continues processing tokens from where it last left

off until it either reaches

the end of the file or executes a return. Once it reaches an end-of-file,

however, then any subsequent call to

.B yylex()

will simply immediately return, unless

.B yyrestart()

is first called (see below).

.LP

Actions are not allowed to modify yytext or yyleng.

.LP

There are a number of special directives which can be included within

an action:

.IP -

.B ECHO

copies yytext to the scanner's output.

.IP -

.B BEGIN

followed by the name of a start condition places the scanner in the

corresponding start condition (see below).

.IP -

.B REJECT

directs the scanner to proceed on to the "second best" rule which matched the

input (or a prefix of the input). The rule is chosen as described

above in "How the Input is Matched", and

.B yytext

and

.B yyleng

set up appropriately.

It may either be one which matched as much text

as the originally chosen rule but came later in the

.I flex

input file, or one which matched less text.

For example, the following will both count the

words in the input and call the routine special() whenever "frob" is seen:

.nf

int word_count = 0;

%%

frob special(); REJECT;

[^ \\t\\n]+ ++word_count;

.fi

Without the

.B REJECT,

any "frob"'s in the input would not be counted as words, since the

scanner normally executes only one action per token.

Multiple

.B REJECT's

are allowed, each one finding the next best choice to the currently

active rule. For example, when the following scanner scans the token

"abcd", it will write "abcdabcaba" to the output:

.nf

%%

a |

ab |

abc |

abcd ECHO; REJECT;

.|\\n /* eat up any unmatched character */

.fi

(The first three rules share the fourth's action since they use

the special '|' action.)

.B REJECT

is a particularly expensive feature in terms scanner performance;

if it is used in

.I any

of the scanner's actions it will slow down

.I all

of the scanner's matching. Furthermore,

.B REJECT

cannot be used with the

.I -f

or

.I -F

options (see below).

.IP

Note also that unlike the other special actions,

.B REJECT

is a

.I branch;

code immediately following it in the action will

.I not

be executed.

.IP -

.B yymore()

tells the scanner that the next time it matches a rule, the corresponding

token should be

.I appended

onto the current value of

.B yytext

rather than replacing it. For example, given the input "mega-kludge"

the following will write "mega-mega-kludge" to the output:

.nf

%%

mega- ECHO; yymore();

kludge ECHO;

.fi

First "mega-" is matched and echoed to the output. Then "kludge"

is matched, but the previous "mega-" is still hanging around at the

beginning of

.B yytext

so the

.B ECHO

for the "kludge" rule will actually write "mega-kludge".

The presence of

.B yymore()

in the scanner's action entails a minor performance penalty in the

scanner's matching speed.

.IP -

.B yyless(n)

returns all but the first

.I n

characters of the current token back to the input stream, where they

will be rescanned when the scanner looks for the next match.

.B yytext

and

.B yyleng

are adjusted appropriately (e.g.,

.B yyleng

will now be equal to

.I n

). For example, on the input "foobar" the following will write out

"foobarbar":

.nf

%%

foobar ECHO; yyless(3);

[a-z]+ ECHO;

.fi

An argument of 0 to

.B yyless

will cause the entire current input string to be scanned again. Unless you've

changed how the scanner will subsequently process its input (using

.B BEGIN,

for example), this will result in an endless loop.

.IP -

.B unput(c)

puts the character

.I c

back onto the input stream. It will be the next character scanned.

The following action will take the current token and cause it

to be rescanned enclosed in parentheses.

.nf

{

int i;

unput( ')' );

for ( i = yyleng - 1; i >= 0; --i )

unput( yytext[i] );

unput( '(' );

}

.fi

Note that since each

.B unput()

puts the given character back at the

.I beginning

of the input stream, pushing back strings must be done back-to-front.

.IP -

.B input()

reads the next character from the input stream. For example,

the following is one way to eat up C comments:

.nf

%%

"/*" {

register int c;

for ( ; ; )

{

while ( (c = input()) != '*' &&

c != EOF )

; /* eat up text of comment */

if ( c == '*' )

{

while ( (c = input()) == '*' )

;

if ( c == '/' )

break; /* found the end */

}

if ( c == EOF )

{

error( "EOF in comment" );

break;

}

}

}

.fi

(Note that if the scanner is compiled using

.B C++,

then

.B input()

is instead referred to as

.B yyinput(),

in order to avoid a name clash with the

.B C++

stream by the name of

.I input.)

.IP -

.B yyterminate()

can be used in lieu of a return statement in an action. It terminates

the scanner and returns a 0 to the scanner's caller, indicating "all done".

Subsequent calls to the scanner will immediately return unless preceded

by a call to

.B yyrestart()

(see below).

By default,

.B yyterminate()

is also called when an end-of-file is encountered. It is a macro and

may be redefined.

.SH THE GENERATED SCANNER

The output of

.I flex

is the file

.B lex.yy.c,

which contains the scanning routine

.B yylex(),

a number of tables used by it for matching tokens, and a number

of auxiliary routines and macros. By default,

.B yylex()

is declared as follows:

.nf

int yylex()

{

... various definitions and the actions in here ...

}

.fi

(If your environment supports function prototypes, then it will

be "int yylex( void )".) This definition may be changed by redefining

the "YY_DECL" macro. For example, you could use:

.nf

#undef YY_DECL

#define YY_DECL float lexscan( a, b ) float a, b;

.fi

to give the scanning routine the name

.I lexscan,

returning a float, and taking two floats as arguments. Note that

if you give arguments to the scanning routine using a

K&R-style/non-prototyped function declaration, you must terminate

the definition with a semi-colon (;).

.LP

Whenever

.B yylex()

is called, it scans tokens from the global input file

.I yyin

(which defaults to stdin). It continues until it either reaches

an end-of-file (at which point it returns the value 0) or

one of its actions executes a

.I return

statement.

In the former case, when called again the scanner will immediately

return unless

.B yyrestart()

is called to point

.I yyin

at the new input file. (

.B yyrestart()

takes one argument, a

.B FILE *

pointer.)

In the latter case (i.e., when an action

executes a return), the scanner may then be called again and it

will resume scanning where it left off.

.LP

By default (and for purposes of efficiency), the scanner uses

block-reads rather than simple

.I getc()

calls to read characters from

.I yyin.

The nature of how it gets its input can be controlled by redefining the

.B YY_INPUT

macro.

YY_INPUT's calling sequence is "YY_INPUT(buf,result,max_size)". Its

action is to place up to

.I max_size

characters in the character array

.I buf

and return in the integer variable

.I result

either the

number of characters read or the constant YY_NULL (0 on Unix systems)

to indicate EOF. The default YY_INPUT reads from the

global file-pointer "yyin".

.LP

A sample redefinition of YY_INPUT (in the definitions

section of the input file):

.nf

%{

#undef YY_INPUT

#define YY_INPUT(buf,result,max_size) \\

{ \\

int c = getchar(); \\

result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \\

}

%}

.fi

This definition will change the input processing to occur

one character at a time.

.LP

You also can add in things like keeping track of the

input line number this way; but don't expect your scanner to

go very fast.

.LP

When the scanner receives an end-of-file indication from YY_INPUT,

it then checks the

.B yywrap()

function. If

.B yywrap()

returns false (zero), then it is assumed that the

function has gone ahead and set up

.I yyin

to point to another input file, and scanning continues. If it returns

true (non-zero), then the scanner terminates, returning 0 to its

caller.

.LP

The default

.B yywrap()

always returns 1. Presently, to redefine it you must first

"#undef yywrap", as it is currently implemented as a macro. As indicated

by the hedging in the previous sentence, it may be changed to

a true function in the near future.

.LP

The scanner writes its

.B ECHO

output to the

.I yyout

global (default, stdout), which may be redefined by the user simply

by assigning it to some other

.B FILE

pointer.

.SH START CONDITIONS

.I flex

provides a mechanism for conditionally activating rules. Any rule

whose pattern is prefixed with "<sc>" will only be active when

the scanner is in the start condition named "sc". For example,

.nf

<STRING>[^"]* { /* eat up the string body ... */

...

}

.fi

will be active only when the scanner is in the "STRING" start

condition, and

.nf

<INITIAL,STRING,QUOTE>\\. { /* handle an escape ... */

...

}

.fi

will be active only when the current start condition is

either "INITIAL", "STRING", or "QUOTE".

.LP

Start conditions

are declared in the definitions (first) section of the input

using unindented lines beginning with either

.B %s

or

.B %x

followed by a list of names.

The former declares

.I inclusive

start conditions, the latter

.I exclusive

start conditions. A start condition is activated using the

.B BEGIN

action. Until the next

.B BEGIN

action is executed, rules with the given start

condition will be active and

rules with other start conditions will be inactive.

If the start condition is

.I inclusive,

then rules with no start conditions at all will also be active.

If it is

.I exclusive,

then

.I only

rules qualified with the start condition will be active.

A set of rules contingent on the same exclusive start condition

describe a scanner which is independent of any of the other rules in the

.I flex

input. Because of this,

exclusive start conditions make it easy to specify "mini-scanners"

which scan portions of the input that are syntactically different

from the rest (e.g., comments).

.LP

If the distinction between inclusive and exclusive start conditions

is still a little vague, here's a simple example illustrating the

connection between the two. The set of rules:

.nf

%s example

%%

<example>foo /* do something */

.fi

is equivalent to

.nf

%x example

%%

<INITIAL,example>foo /* do something */

.fi

.LP

The default rule (to

.B ECHO

any unmatched character) remains active in start conditions.

.LP

.B BEGIN(0)

returns to the original state where only the rules with

no start conditions are active. This state can also be

referred to as the start-condition "INITIAL", so

.B BEGIN(INITIAL)

is equivalent to

.B BEGIN(0).

(The parentheses around the start condition name are not required but

are considered good style.)

.LP

.B BEGIN

actions can also be given as indented code at the beginning

of the rules section. For example, the following will cause

the scanner to enter the "SPECIAL" start condition whenever

.I yylex()

is called and the global variable

.I enter_special

is true:

.nf

int enter_special;

%x SPECIAL

%%

if ( enter_special )

BEGIN(SPECIAL);

<SPECIAL>blahblahblah

...more rules follow...

.fi

.LP

To illustrate the uses of start conditions,

here is a scanner which provides two different interpretations

of a string like "123.456". By default it will treat it as

as three tokens, the integer "123", a dot ('.'), and the integer "456".

But if the string is preceded earlier in the line by the string

"expect-floats"

it will treat it as a single token, the floating-point number

123.456:

.nf

%{

#include <math.h>

%}

%s expect

%%

expect-floats BEGIN(expect);

<expect>[0-9]+"."[0-9]+ {

printf( "found a float, = %f\\n",

atof( yytext ) );

}

<expect>\\n {

/* that's the end of the line, so

* we need another "expect-number"

* before we'll recognize any more

* numbers

*/

BEGIN(INITIAL);

}

[0-9]+ {

printf( "found an integer, = %d\\n",

atoi( yytext ) );

}

"." printf( "found a dot\\n" );

.fi

Here is a scanner which recognizes (and discards) C comments while

maintaining a count of the current input line.

.nf

%x comment

%%

int line_num = 1;

"/*" BEGIN(comment);

<comment>[^*\\n]* /* eat anything that's not a '*' */

<comment>"*"+[^*/\\n]* /* eat up '*'s not followed by '/'s */

<comment>\\n ++line_num;

<comment>"*"+"/" BEGIN(INITIAL);

.fi

Note that start-conditions names are really integer values and

can be stored as such. Thus, the above could be extended in the

following fashion: