Lines Matching defs:token
118 # When a macro is created, the macro replacement token sequence is
167 tok = self.lexer.token()
185 # the token types of symbols that are important to the preprocessor.
192 # Determine the token type for identifiers
194 tok = self.lexer.token()
200 # Determine the token type for integers
202 tok = self.lexer.token()
209 # Determine the token type for strings enclosed in double quotes
211 tok = self.lexer.token()
217 # Determine the token type for whitespace--if any
219 tok = self.lexer.token()
225 # Determine the token type for newlines
227 tok = self.lexer.token()
230 print "Couldn't determine token for newlines"
240 tok = self.lexer.token()
278 tok = lex.token()
292 # Remove leading/trailing whitespace tokens from a token list
371 # Examine the macro value (token sequence) and identify patch points
414 # Given a Macro and list of arguments (each a token list), this method
415 # returns an expanded version of a macro. The return value is a token sequence
420 # Make a copy of the macro token sequence
531 # Evaluate an expression token sequence for the purposes of evaluating
860 # token()
864 def token(self):
885 tok = p.token()