3.3 Lexical Tokens
The syntax of VBA programs is most easily described in terms of lexical tokens rather than individual Unicode characters. In particular, the occurrence of whitespace or line-continuations between most syntactic elements is usually irrelevant to the syntactic grammar. The syntactic grammar is significantly simplified if it does not have to describe such possible whitespace occurrences. This is accomplished by using lexical tokens (also referred to simply as tokens) that abstract away whitespace as the terminal symbols of the syntactic grammar.
The lexical grammar defines the interpretation of a <module-body-lines> as a set of such lexical tokens.
The terminal elements of the lexical grammar are Unicode characters and the <LINE-START> and <LINE-END> elements. Generally any rule name of the lexical grammar that is written in all upper case characters is also a lexical token and terminal element of the VBA syntactic grammar. ABNF quoted literal text rules are also considered to be lexical tokens of the syntactic grammar. Lexical tokens encompass any white space characters that immediate precede them. Note that when used within the lexical grammar, quoted literal text rules are not treated as tokens and hence any preceding whitespace characters are significant.