Added kReturnUnknownLexFl configuration flag. If this flag is set
tokens which are not recognized will be returned with the
token id set to kUnknownLexTId. This is useful for returning all text.
Added kUserDefPriorityLexFl configuration flag. User defined tokens
take priority even if an identifier (kIdentLexTId) has a longer match.
Added cmLexEnableToken(). This function allows tokens recognizers to be
enabled and disabled.
