CHANGES revision 6498
1 2Version 3.2 3----------------------------- 403/24/09: beazley 5 Added an extra check to not print duplicated warning messages 6 about reduce/reduce conflicts. 7 803/24/09: beazley 9 Switched PLY over to a BSD-license. 10 1103/23/09: beazley 12 Performance optimization. Discovered a few places to make 13 speedups in LR table generation. 14 1503/23/09: beazley 16 New warning message. PLY now warns about rules never 17 reduced due to reduce/reduce conflicts. Suggested by 18 Bruce Frederiksen. 19 2003/23/09: beazley 21 Some clean-up of warning messages related to reduce/reduce errors. 22 2303/23/09: beazley 24 Added a new picklefile option to yacc() to write the parsing 25 tables to a filename using the pickle module. Here is how 26 it works: 27 28 yacc(picklefile="parsetab.p") 29 30 This option can be used if the normal parsetab.py file is 31 extremely large. For example, on jython, it is impossible 32 to read parsing tables if the parsetab.py exceeds a certain 33 threshold. 34 35 The filename supplied to the picklefile option is opened 36 relative to the current working directory of the Python 37 interpreter. If you need to refer to the file elsewhere, 38 you will need to supply an absolute or relative path. 39 40 For maximum portability, the pickle file is written 41 using protocol 0. 42 4303/13/09: beazley 44 Fixed a bug in parser.out generation where the rule numbers 45 where off by one. 46 4703/13/09: beazley 48 Fixed a string formatting bug with one of the error messages. 49 Reported by Richard Reitmeyer 50 51Version 3.1 52----------------------------- 5302/28/09: beazley 54 Fixed broken start argument to yacc(). PLY-3.0 broke this 55 feature by accident. 56 5702/28/09: beazley 58 Fixed debugging output. yacc() no longer reports shift/reduce 59 or reduce/reduce conflicts if debugging is turned off. This 60 restores similar behavior in PLY-2.5. Reported by Andrew Waters. 61 62Version 3.0 63----------------------------- 6402/03/09: beazley 65 Fixed missing lexer attribute on certain tokens when 66 invoking the parser p_error() function. Reported by 67 Bart Whiteley. 68 6902/02/09: beazley 70 The lex() command now does all error-reporting and diagonistics 71 using the logging module interface. Pass in a Logger object 72 using the errorlog parameter to specify a different logger. 73 7402/02/09: beazley 75 Refactored ply.lex to use a more object-oriented and organized 76 approach to collecting lexer information. 77 7802/01/09: beazley 79 Removed the nowarn option from lex(). All output is controlled 80 by passing in a logger object. Just pass in a logger with a high 81 level setting to suppress output. This argument was never 82 documented to begin with so hopefully no one was relying upon it. 83 8402/01/09: beazley 85 Discovered and removed a dead if-statement in the lexer. This 86 resulted in a 6-7% speedup in lexing when I tested it. 87 8801/13/09: beazley 89 Minor change to the procedure for signalling a syntax error in a 90 production rule. A normal SyntaxError exception should be raised 91 instead of yacc.SyntaxError. 92 9301/13/09: beazley 94 Added a new method p.set_lineno(n,lineno) that can be used to set the 95 line number of symbol n in grammar rules. This simplifies manual 96 tracking of line numbers. 97 9801/11/09: beazley 99 Vastly improved debugging support for yacc.parse(). Instead of passing 100 debug as an integer, you can supply a Logging object (see the logging 101 module). Messages will be generated at the ERROR, INFO, and DEBUG 102 logging levels, each level providing progressively more information. 103 The debugging trace also shows states, grammar rule, values passed 104 into grammar rules, and the result of each reduction. 105 10601/09/09: beazley 107 The yacc() command now does all error-reporting and diagnostics using 108 the interface of the logging module. Use the errorlog parameter to 109 specify a logging object for error messages. Use the debuglog parameter 110 to specify a logging object for the 'parser.out' output. 111 11201/09/09: beazley 113 *HUGE* refactoring of the the ply.yacc() implementation. The high-level 114 user interface is backwards compatible, but the internals are completely 115 reorganized into classes. No more global variables. The internals 116 are also more extensible. For example, you can use the classes to 117 construct a LALR(1) parser in an entirely different manner than 118 what is currently the case. Documentation is forthcoming. 119 12001/07/09: beazley 121 Various cleanup and refactoring of yacc internals. 122 12301/06/09: beazley 124 Fixed a bug with precedence assignment. yacc was assigning the precedence 125 each rule based on the left-most token, when in fact, it should have been 126 using the right-most token. Reported by Bruce Frederiksen. 127 12811/27/08: beazley 129 Numerous changes to support Python 3.0 including removal of deprecated 130 statements (e.g., has_key) and the additional of compatibility code 131 to emulate features from Python 2 that have been removed, but which 132 are needed. Fixed the unit testing suite to work with Python 3.0. 133 The code should be backwards compatible with Python 2. 134 13511/26/08: beazley 136 Loosened the rules on what kind of objects can be passed in as the 137 "module" parameter to lex() and yacc(). Previously, you could only use 138 a module or an instance. Now, PLY just uses dir() to get a list of 139 symbols on whatever the object is without regard for its type. 140 14111/26/08: beazley 142 Changed all except: statements to be compatible with Python2.x/3.x syntax. 143 14411/26/08: beazley 145 Changed all raise Exception, value statements to raise Exception(value) for 146 forward compatibility. 147 14811/26/08: beazley 149 Removed all print statements from lex and yacc, using sys.stdout and sys.stderr 150 directly. Preparation for Python 3.0 support. 151 15211/04/08: beazley 153 Fixed a bug with referring to symbols on the the parsing stack using negative 154 indices. 155 15605/29/08: beazley 157 Completely revamped the testing system to use the unittest module for everything. 158 Added additional tests to cover new errors/warnings. 159 160Version 2.5 161----------------------------- 16205/28/08: beazley 163 Fixed a bug with writing lex-tables in optimized mode and start states. 164 Reported by Kevin Henry. 165 166Version 2.4 167----------------------------- 16805/04/08: beazley 169 A version number is now embedded in the table file signature so that 170 yacc can more gracefully accomodate changes to the output format 171 in the future. 172 17305/04/08: beazley 174 Removed undocumented .pushback() method on grammar productions. I'm 175 not sure this ever worked and can't recall ever using it. Might have 176 been an abandoned idea that never really got fleshed out. This 177 feature was never described or tested so removing it is hopefully 178 harmless. 179 18005/04/08: beazley 181 Added extra error checking to yacc() to detect precedence rules defined 182 for undefined terminal symbols. This allows yacc() to detect a potential 183 problem that can be really tricky to debug if no warning message or error 184 message is generated about it. 185 18605/04/08: beazley 187 lex() now has an outputdir that can specify the output directory for 188 tables when running in optimize mode. For example: 189 190 lexer = lex.lex(optimize=True, lextab="ltab", outputdir="foo/bar") 191 192 The behavior of specifying a table module and output directory are 193 more aligned with the behavior of yacc(). 194 19505/04/08: beazley 196 [Issue 9] 197 Fixed filename bug in when specifying the modulename in lex() and yacc(). 198 If you specified options such as the following: 199 200 parser = yacc.yacc(tabmodule="foo.bar.parsetab",outputdir="foo/bar") 201 202 yacc would create a file "foo.bar.parsetab.py" in the given directory. 203 Now, it simply generates a file "parsetab.py" in that directory. 204 Bug reported by cptbinho. 205 20605/04/08: beazley 207 Slight modification to lex() and yacc() to allow their table files 208 to be loaded from a previously loaded module. This might make 209 it easier to load the parsing tables from a complicated package 210 structure. For example: 211 212 import foo.bar.spam.parsetab as parsetab 213 parser = yacc.yacc(tabmodule=parsetab) 214 215 Note: lex and yacc will never regenerate the table file if used 216 in the form---you will get a warning message instead. 217 This idea suggested by Brian Clapper. 218 219 22004/28/08: beazley 221 Fixed a big with p_error() functions being picked up correctly 222 when running in yacc(optimize=1) mode. Patch contributed by 223 Bart Whiteley. 224 22502/28/08: beazley 226 Fixed a bug with 'nonassoc' precedence rules. Basically the 227 non-precedence was being ignored and not producing the correct 228 run-time behavior in the parser. 229 23002/16/08: beazley 231 Slight relaxation of what the input() method to a lexer will 232 accept as a string. Instead of testing the input to see 233 if the input is a string or unicode string, it checks to see 234 if the input object looks like it contains string data. 235 This change makes it possible to pass string-like objects 236 in as input. For example, the object returned by mmap. 237 238 import mmap, os 239 data = mmap.mmap(os.open(filename,os.O_RDONLY), 240 os.path.getsize(filename), 241 access=mmap.ACCESS_READ) 242 lexer.input(data) 243 244 24511/29/07: beazley 246 Modification of ply.lex to allow token functions to aliased. 247 This is subtle, but it makes it easier to create libraries and 248 to reuse token specifications. For example, suppose you defined 249 a function like this: 250 251 def number(t): 252 r'\d+' 253 t.value = int(t.value) 254 return t 255 256 This change would allow you to define a token rule as follows: 257 258 t_NUMBER = number 259 260 In this case, the token type will be set to 'NUMBER' and use 261 the associated number() function to process tokens. 262 26311/28/07: beazley 264 Slight modification to lex and yacc to grab symbols from both 265 the local and global dictionaries of the caller. This 266 modification allows lexers and parsers to be defined using 267 inner functions and closures. 268 26911/28/07: beazley 270 Performance optimization: The lexer.lexmatch and t.lexer 271 attributes are no longer set for lexer tokens that are not 272 defined by functions. The only normal use of these attributes 273 would be in lexer rules that need to perform some kind of 274 special processing. Thus, it doesn't make any sense to set 275 them on every token. 276 277 *** POTENTIAL INCOMPATIBILITY *** This might break code 278 that is mucking around with internal lexer state in some 279 sort of magical way. 280 28111/27/07: beazley 282 Added the ability to put the parser into error-handling mode 283 from within a normal production. To do this, simply raise 284 a yacc.SyntaxError exception like this: 285 286 def p_some_production(p): 287 'some_production : prod1 prod2' 288 ... 289 raise yacc.SyntaxError # Signal an error 290 291 A number of things happen after this occurs: 292 293 - The last symbol shifted onto the symbol stack is discarded 294 and parser state backed up to what it was before the 295 the rule reduction. 296 297 - The current lookahead symbol is saved and replaced by 298 the 'error' symbol. 299 300 - The parser enters error recovery mode where it tries 301 to either reduce the 'error' rule or it starts 302 discarding items off of the stack until the parser 303 resets. 304 305 When an error is manually set, the parser does *not* call 306 the p_error() function (if any is defined). 307 *** NEW FEATURE *** Suggested on the mailing list 308 30911/27/07: beazley 310 Fixed structure bug in examples/ansic. Reported by Dion Blazakis. 311 31211/27/07: beazley 313 Fixed a bug in the lexer related to start conditions and ignored 314 token rules. If a rule was defined that changed state, but 315 returned no token, the lexer could be left in an inconsistent 316 state. Reported by 317 31811/27/07: beazley 319 Modified setup.py to support Python Eggs. Patch contributed by 320 Simon Cross. 321 32211/09/07: beazely 323 Fixed a bug in error handling in yacc. If a syntax error occurred and the 324 parser rolled the entire parse stack back, the parser would be left in in 325 inconsistent state that would cause it to trigger incorrect actions on 326 subsequent input. Reported by Ton Biegstraaten, Justin King, and others. 327 32811/09/07: beazley 329 Fixed a bug when passing empty input strings to yacc.parse(). This 330 would result in an error message about "No input given". Reported 331 by Andrew Dalke. 332 333Version 2.3 334----------------------------- 33502/20/07: beazley 336 Fixed a bug with character literals if the literal '.' appeared as the 337 last symbol of a grammar rule. Reported by Ales Smrcka. 338 33902/19/07: beazley 340 Warning messages are now redirected to stderr instead of being printed 341 to standard output. 342 34302/19/07: beazley 344 Added a warning message to lex.py if it detects a literal backslash 345 character inside the t_ignore declaration. This is to help 346 problems that might occur if someone accidentally defines t_ignore 347 as a Python raw string. For example: 348 349 t_ignore = r' \t' 350 351 The idea for this is from an email I received from David Cimimi who 352 reported bizarre behavior in lexing as a result of defining t_ignore 353 as a raw string by accident. 354 35502/18/07: beazley 356 Performance improvements. Made some changes to the internal 357 table organization and LR parser to improve parsing performance. 358 35902/18/07: beazley 360 Automatic tracking of line number and position information must now be 361 enabled by a special flag to parse(). For example: 362 363 yacc.parse(data,tracking=True) 364 365 In many applications, it's just not that important to have the 366 parser automatically track all line numbers. By making this an 367 optional feature, it allows the parser to run significantly faster 368 (more than a 20% speed increase in many cases). Note: positional 369 information is always available for raw tokens---this change only 370 applies to positional information associated with nonterminal 371 grammar symbols. 372 *** POTENTIAL INCOMPATIBILITY *** 373 37402/18/07: beazley 375 Yacc no longer supports extended slices of grammar productions. 376 However, it does support regular slices. For example: 377 378 def p_foo(p): 379 '''foo: a b c d e''' 380 p[0] = p[1:3] 381 382 This change is a performance improvement to the parser--it streamlines 383 normal access to the grammar values since slices are now handled in 384 a __getslice__() method as opposed to __getitem__(). 385 38602/12/07: beazley 387 Fixed a bug in the handling of token names when combined with 388 start conditions. Bug reported by Todd O'Bryan. 389 390Version 2.2 391------------------------------ 39211/01/06: beazley 393 Added lexpos() and lexspan() methods to grammar symbols. These 394 mirror the same functionality of lineno() and linespan(). For 395 example: 396 397 def p_expr(p): 398 'expr : expr PLUS expr' 399 p.lexpos(1) # Lexing position of left-hand-expression 400 p.lexpos(1) # Lexing position of PLUS 401 start,end = p.lexspan(3) # Lexing range of right hand expression 402 40311/01/06: beazley 404 Minor change to error handling. The recommended way to skip characters 405 in the input is to use t.lexer.skip() as shown here: 406 407 def t_error(t): 408 print "Illegal character '%s'" % t.value[0] 409 t.lexer.skip(1) 410 411 The old approach of just using t.skip(1) will still work, but won't 412 be documented. 413 41410/31/06: beazley 415 Discarded tokens can now be specified as simple strings instead of 416 functions. To do this, simply include the text "ignore_" in the 417 token declaration. For example: 418 419 t_ignore_cppcomment = r'//.*' 420 421 Previously, this had to be done with a function. For example: 422 423 def t_ignore_cppcomment(t): 424 r'//.*' 425 pass 426 427 If start conditions/states are being used, state names should appear 428 before the "ignore_" text. 429 43010/19/06: beazley 431 The Lex module now provides support for flex-style start conditions 432 as described at http://www.gnu.org/software/flex/manual/html_chapter/flex_11.html. 433 Please refer to this document to understand this change note. Refer to 434 the PLY documentation for PLY-specific explanation of how this works. 435 436 To use start conditions, you first need to declare a set of states in 437 your lexer file: 438 439 states = ( 440 ('foo','exclusive'), 441 ('bar','inclusive') 442 ) 443 444 This serves the same role as the %s and %x specifiers in flex. 445 446 One a state has been declared, tokens for that state can be 447 declared by defining rules of the form t_state_TOK. For example: 448 449 t_PLUS = '\+' # Rule defined in INITIAL state 450 t_foo_NUM = '\d+' # Rule defined in foo state 451 t_bar_NUM = '\d+' # Rule defined in bar state 452 453 t_foo_bar_NUM = '\d+' # Rule defined in both foo and bar 454 t_ANY_NUM = '\d+' # Rule defined in all states 455 456 In addition to defining tokens for each state, the t_ignore and t_error 457 specifications can be customized for specific states. For example: 458 459 t_foo_ignore = " " # Ignored characters for foo state 460 def t_bar_error(t): 461 # Handle errors in bar state 462 463 With token rules, the following methods can be used to change states 464 465 def t_TOKNAME(t): 466 t.lexer.begin('foo') # Begin state 'foo' 467 t.lexer.push_state('foo') # Begin state 'foo', push old state 468 # onto a stack 469 t.lexer.pop_state() # Restore previous state 470 t.lexer.current_state() # Returns name of current state 471 472 These methods mirror the BEGIN(), yy_push_state(), yy_pop_state(), and 473 yy_top_state() functions in flex. 474 475 The use of start states can be used as one way to write sub-lexers. 476 For example, the lexer or parser might instruct the lexer to start 477 generating a different set of tokens depending on the context. 478 479 example/yply/ylex.py shows the use of start states to grab C/C++ 480 code fragments out of traditional yacc specification files. 481 482 *** NEW FEATURE *** Suggested by Daniel Larraz with whom I also 483 discussed various aspects of the design. 484 48510/19/06: beazley 486 Minor change to the way in which yacc.py was reporting shift/reduce 487 conflicts. Although the underlying LALR(1) algorithm was correct, 488 PLY was under-reporting the number of conflicts compared to yacc/bison 489 when precedence rules were in effect. This change should make PLY 490 report the same number of conflicts as yacc. 491 49210/19/06: beazley 493 Modified yacc so that grammar rules could also include the '-' 494 character. For example: 495 496 def p_expr_list(p): 497 'expression-list : expression-list expression' 498 499 Suggested by Oldrich Jedlicka. 500 50110/18/06: beazley 502 Attribute lexer.lexmatch added so that token rules can access the re 503 match object that was generated. For example: 504 505 def t_FOO(t): 506 r'some regex' 507 m = t.lexer.lexmatch 508 # Do something with m 509 510 511 This may be useful if you want to access named groups specified within 512 the regex for a specific token. Suggested by Oldrich Jedlicka. 513 51410/16/06: beazley 515 Changed the error message that results if an illegal character 516 is encountered and no default error function is defined in lex. 517 The exception is now more informative about the actual cause of 518 the error. 519 520Version 2.1 521------------------------------ 52210/02/06: beazley 523 The last Lexer object built by lex() can be found in lex.lexer. 524 The last Parser object built by yacc() can be found in yacc.parser. 525 52610/02/06: beazley 527 New example added: examples/yply 528 529 This example uses PLY to convert Unix-yacc specification files to 530 PLY programs with the same grammar. This may be useful if you 531 want to convert a grammar from bison/yacc to use with PLY. 532 53310/02/06: beazley 534 Added support for a start symbol to be specified in the yacc 535 input file itself. Just do this: 536 537 start = 'name' 538 539 where 'name' matches some grammar rule. For example: 540 541 def p_name(p): 542 'name : A B C' 543 ... 544 545 This mirrors the functionality of the yacc %start specifier. 546 54709/30/06: beazley 548 Some new examples added.: 549 550 examples/GardenSnake : A simple indentation based language similar 551 to Python. Shows how you might handle 552 whitespace. Contributed by Andrew Dalke. 553 554 examples/BASIC : An implementation of 1964 Dartmouth BASIC. 555 Contributed by Dave against his better 556 judgement. 557 55809/28/06: beazley 559 Minor patch to allow named groups to be used in lex regular 560 expression rules. For example: 561 562 t_QSTRING = r'''(?P<quote>['"]).*?(?P=quote)''' 563 564 Patch submitted by Adam Ring. 565 56609/28/06: beazley 567 LALR(1) is now the default parsing method. To use SLR, use 568 yacc.yacc(method="SLR"). Note: there is no performance impact 569 on parsing when using LALR(1) instead of SLR. However, constructing 570 the parsing tables will take a little longer. 571 57209/26/06: beazley 573 Change to line number tracking. To modify line numbers, modify 574 the line number of the lexer itself. For example: 575 576 def t_NEWLINE(t): 577 r'\n' 578 t.lexer.lineno += 1 579 580 This modification is both cleanup and a performance optimization. 581 In past versions, lex was monitoring every token for changes in 582 the line number. This extra processing is unnecessary for a vast 583 majority of tokens. Thus, this new approach cleans it up a bit. 584 585 *** POTENTIAL INCOMPATIBILITY *** 586 You will need to change code in your lexer that updates the line 587 number. For example, "t.lineno += 1" becomes "t.lexer.lineno += 1" 588 58909/26/06: beazley 590 Added the lexing position to tokens as an attribute lexpos. This 591 is the raw index into the input text at which a token appears. 592 This information can be used to compute column numbers and other 593 details (e.g., scan backwards from lexpos to the first newline 594 to get a column position). 595 59609/25/06: beazley 597 Changed the name of the __copy__() method on the Lexer class 598 to clone(). This is used to clone a Lexer object (e.g., if 599 you're running different lexers at the same time). 600 60109/21/06: beazley 602 Limitations related to the use of the re module have been eliminated. 603 Several users reported problems with regular expressions exceeding 604 more than 100 named groups. To solve this, lex.py is now capable 605 of automatically splitting its master regular regular expression into 606 smaller expressions as needed. This should, in theory, make it 607 possible to specify an arbitrarily large number of tokens. 608 60909/21/06: beazley 610 Improved error checking in lex.py. Rules that match the empty string 611 are now rejected (otherwise they cause the lexer to enter an infinite 612 loop). An extra check for rules containing '#' has also been added. 613 Since lex compiles regular expressions in verbose mode, '#' is interpreted 614 as a regex comment, it is critical to use '\#' instead. 615 61609/18/06: beazley 617 Added a @TOKEN decorator function to lex.py that can be used to 618 define token rules where the documentation string might be computed 619 in some way. 620 621 digit = r'([0-9])' 622 nondigit = r'([_A-Za-z])' 623 identifier = r'(' + nondigit + r'(' + digit + r'|' + nondigit + r')*)' 624 625 from ply.lex import TOKEN 626 627 @TOKEN(identifier) 628 def t_ID(t): 629 # Do whatever 630 631 The @TOKEN decorator merely sets the documentation string of the 632 associated token function as needed for lex to work. 633 634 Note: An alternative solution is the following: 635 636 def t_ID(t): 637 # Do whatever 638 639 t_ID.__doc__ = identifier 640 641 Note: Decorators require the use of Python 2.4 or later. If compatibility 642 with old versions is needed, use the latter solution. 643 644 The need for this feature was suggested by Cem Karan. 645 64609/14/06: beazley 647 Support for single-character literal tokens has been added to yacc. 648 These literals must be enclosed in quotes. For example: 649 650 def p_expr(p): 651 "expr : expr '+' expr" 652 ... 653 654 def p_expr(p): 655 'expr : expr "-" expr' 656 ... 657 658 In addition to this, it is necessary to tell the lexer module about 659 literal characters. This is done by defining the variable 'literals' 660 as a list of characters. This should be defined in the module that 661 invokes the lex.lex() function. For example: 662 663 literals = ['+','-','*','/','(',')','='] 664 665 or simply 666 667 literals = '+=*/()=' 668 669 It is important to note that literals can only be a single character. 670 When the lexer fails to match a token using its normal regular expression 671 rules, it will check the current character against the literal list. 672 If found, it will be returned with a token type set to match the literal 673 character. Otherwise, an illegal character will be signalled. 674 675 67609/14/06: beazley 677 Modified PLY to install itself as a proper Python package called 'ply'. 678 This will make it a little more friendly to other modules. This 679 changes the usage of PLY only slightly. Just do this to import the 680 modules 681 682 import ply.lex as lex 683 import ply.yacc as yacc 684 685 Alternatively, you can do this: 686 687 from ply import * 688 689 Which imports both the lex and yacc modules. 690 Change suggested by Lee June. 691 69209/13/06: beazley 693 Changed the handling of negative indices when used in production rules. 694 A negative production index now accesses already parsed symbols on the 695 parsing stack. For example, 696 697 def p_foo(p): 698 "foo: A B C D" 699 print p[1] # Value of 'A' symbol 700 print p[2] # Value of 'B' symbol 701 print p[-1] # Value of whatever symbol appears before A 702 # on the parsing stack. 703 704 p[0] = some_val # Sets the value of the 'foo' grammer symbol 705 706 This behavior makes it easier to work with embedded actions within the 707 parsing rules. For example, in C-yacc, it is possible to write code like 708 this: 709 710 bar: A { printf("seen an A = %d\n", $1); } B { do_stuff; } 711 712 In this example, the printf() code executes immediately after A has been 713 parsed. Within the embedded action code, $1 refers to the A symbol on 714 the stack. 715 716 To perform this equivalent action in PLY, you need to write a pair 717 of rules like this: 718 719 def p_bar(p): 720 "bar : A seen_A B" 721 do_stuff 722 723 def p_seen_A(p): 724 "seen_A :" 725 print "seen an A =", p[-1] 726 727 The second rule "seen_A" is merely a empty production which should be 728 reduced as soon as A is parsed in the "bar" rule above. The use 729 of the negative index p[-1] is used to access whatever symbol appeared 730 before the seen_A symbol. 731 732 This feature also makes it possible to support inherited attributes. 733 For example: 734 735 def p_decl(p): 736 "decl : scope name" 737 738 def p_scope(p): 739 """scope : GLOBAL 740 | LOCAL""" 741 p[0] = p[1] 742 743 def p_name(p): 744 "name : ID" 745 if p[-1] == "GLOBAL": 746 # ... 747 else if p[-1] == "LOCAL": 748 #... 749 750 In this case, the name rule is inheriting an attribute from the 751 scope declaration that precedes it. 752 753 *** POTENTIAL INCOMPATIBILITY *** 754 If you are currently using negative indices within existing grammar rules, 755 your code will break. This should be extremely rare if non-existent in 756 most cases. The argument to various grammar rules is not usually not 757 processed in the same way as a list of items. 758 759Version 2.0 760------------------------------ 76109/07/06: beazley 762 Major cleanup and refactoring of the LR table generation code. Both SLR 763 and LALR(1) table generation is now performed by the same code base with 764 only minor extensions for extra LALR(1) processing. 765 76609/07/06: beazley 767 Completely reimplemented the entire LALR(1) parsing engine to use the 768 DeRemer and Pennello algorithm for calculating lookahead sets. This 769 significantly improves the performance of generating LALR(1) tables 770 and has the added feature of actually working correctly! If you 771 experienced weird behavior with LALR(1) in prior releases, this should 772 hopefully resolve all of those problems. Many thanks to 773 Andrew Waters and Markus Schoepflin for submitting bug reports 774 and helping me test out the revised LALR(1) support. 775 776Version 1.8 777------------------------------ 77808/02/06: beazley 779 Fixed a problem related to the handling of default actions in LALR(1) 780 parsing. If you experienced subtle and/or bizarre behavior when trying 781 to use the LALR(1) engine, this may correct those problems. Patch 782 contributed by Russ Cox. Note: This patch has been superceded by 783 revisions for LALR(1) parsing in Ply-2.0. 784 78508/02/06: beazley 786 Added support for slicing of productions in yacc. 787 Patch contributed by Patrick Mezard. 788 789Version 1.7 790------------------------------ 79103/02/06: beazley 792 Fixed infinite recursion problem ReduceToTerminals() function that 793 would sometimes come up in LALR(1) table generation. Reported by 794 Markus Schoepflin. 795 79603/01/06: beazley 797 Added "reflags" argument to lex(). For example: 798 799 lex.lex(reflags=re.UNICODE) 800 801 This can be used to specify optional flags to the re.compile() function 802 used inside the lexer. This may be necessary for special situations such 803 as processing Unicode (e.g., if you want escapes like \w and \b to consult 804 the Unicode character property database). The need for this suggested by 805 Andreas Jung. 806 80703/01/06: beazley 808 Fixed a bug with an uninitialized variable on repeated instantiations of parser 809 objects when the write_tables=0 argument was used. Reported by Michael Brown. 810 81103/01/06: beazley 812 Modified lex.py to accept Unicode strings both as the regular expressions for 813 tokens and as input. Hopefully this is the only change needed for Unicode support. 814 Patch contributed by Johan Dahl. 815 81603/01/06: beazley 817 Modified the class-based interface to work with new-style or old-style classes. 818 Patch contributed by Michael Brown (although I tweaked it slightly so it would work 819 with older versions of Python). 820 821Version 1.6 822------------------------------ 82305/27/05: beazley 824 Incorporated patch contributed by Christopher Stawarz to fix an extremely 825 devious bug in LALR(1) parser generation. This patch should fix problems 826 numerous people reported with LALR parsing. 827 82805/27/05: beazley 829 Fixed problem with lex.py copy constructor. Reported by Dave Aitel, Aaron Lav, 830 and Thad Austin. 831 83205/27/05: beazley 833 Added outputdir option to yacc() to control output directory. Contributed 834 by Christopher Stawarz. 835 83605/27/05: beazley 837 Added rununit.py test script to run tests using the Python unittest module. 838 Contributed by Miki Tebeka. 839 840Version 1.5 841------------------------------ 84205/26/04: beazley 843 Major enhancement. LALR(1) parsing support is now working. 844 This feature was implemented by Elias Ioup (ezioup@alumni.uchicago.edu) 845 and optimized by David Beazley. To use LALR(1) parsing do 846 the following: 847 848 yacc.yacc(method="LALR") 849 850 Computing LALR(1) parsing tables takes about twice as long as 851 the default SLR method. However, LALR(1) allows you to handle 852 more complex grammars. For example, the ANSI C grammar 853 (in example/ansic) has 13 shift-reduce conflicts with SLR, but 854 only has 1 shift-reduce conflict with LALR(1). 855 85605/20/04: beazley 857 Added a __len__ method to parser production lists. Can 858 be used in parser rules like this: 859 860 def p_somerule(p): 861 """a : B C D 862 | E F" 863 if (len(p) == 3): 864 # Must have been first rule 865 elif (len(p) == 2): 866 # Must be second rule 867 868 Suggested by Joshua Gerth and others. 869 870Version 1.4 871------------------------------ 87204/23/04: beazley 873 Incorporated a variety of patches contributed by Eric Raymond. 874 These include: 875 876 0. Cleans up some comments so they don't wrap on an 80-column display. 877 1. Directs compiler errors to stderr where they belong. 878 2. Implements and documents automatic line counting when \n is ignored. 879 3. Changes the way progress messages are dumped when debugging is on. 880 The new format is both less verbose and conveys more information than 881 the old, including shift and reduce actions. 882 88304/23/04: beazley 884 Added a Python setup.py file to simply installation. Contributed 885 by Adam Kerrison. 886 88704/23/04: beazley 888 Added patches contributed by Adam Kerrison. 889 890 - Some output is now only shown when debugging is enabled. This 891 means that PLY will be completely silent when not in debugging mode. 892 893 - An optional parameter "write_tables" can be passed to yacc() to 894 control whether or not parsing tables are written. By default, 895 it is true, but it can be turned off if you don't want the yacc 896 table file. Note: disabling this will cause yacc() to regenerate 897 the parsing table each time. 898 89904/23/04: beazley 900 Added patches contributed by David McNab. This patch addes two 901 features: 902 903 - The parser can be supplied as a class instead of a module. 904 For an example of this, see the example/classcalc directory. 905 906 - Debugging output can be directed to a filename of the user's 907 choice. Use 908 909 yacc(debugfile="somefile.out") 910 911 912Version 1.3 913------------------------------ 91412/10/02: jmdyck 915 Various minor adjustments to the code that Dave checked in today. 916 Updated test/yacc_{inf,unused}.exp to reflect today's changes. 917 91812/10/02: beazley 919 Incorporated a variety of minor bug fixes to empty production 920 handling and infinite recursion checking. Contributed by 921 Michael Dyck. 922 92312/10/02: beazley 924 Removed bogus recover() method call in yacc.restart() 925 926Version 1.2 927------------------------------ 92811/27/02: beazley 929 Lexer and parser objects are now available as an attribute 930 of tokens and slices respectively. For example: 931 932 def t_NUMBER(t): 933 r'\d+' 934 print t.lexer 935 936 def p_expr_plus(t): 937 'expr: expr PLUS expr' 938 print t.lexer 939 print t.parser 940 941 This can be used for state management (if needed). 942 94310/31/02: beazley 944 Modified yacc.py to work with Python optimize mode. To make 945 this work, you need to use 946 947 yacc.yacc(optimize=1) 948 949 Furthermore, you need to first run Python in normal mode 950 to generate the necessary parsetab.py files. After that, 951 you can use python -O or python -OO. 952 953 Note: optimized mode turns off a lot of error checking. 954 Only use when you are sure that your grammar is working. 955 Make sure parsetab.py is up to date! 956 95710/30/02: beazley 958 Added cloning of Lexer objects. For example: 959 960 import copy 961 l = lex.lex() 962 lc = copy.copy(l) 963 964 l.input("Some text") 965 lc.input("Some other text") 966 ... 967 968 This might be useful if the same "lexer" is meant to 969 be used in different contexts---or if multiple lexers 970 are running concurrently. 971 97210/30/02: beazley 973 Fixed subtle bug with first set computation and empty productions. 974 Patch submitted by Michael Dyck. 975 97610/30/02: beazley 977 Fixed error messages to use "filename:line: message" instead 978 of "filename:line. message". This makes error reporting more 979 friendly to emacs. Patch submitted by Fran�ois Pinard. 980 98110/30/02: beazley 982 Improvements to parser.out file. Terminals and nonterminals 983 are sorted instead of being printed in random order. 984 Patch submitted by Fran�ois Pinard. 985 98610/30/02: beazley 987 Improvements to parser.out file output. Rules are now printed 988 in a way that's easier to understand. Contributed by Russ Cox. 989 99010/30/02: beazley 991 Added 'nonassoc' associativity support. This can be used 992 to disable the chaining of operators like a < b < c. 993 To use, simply specify 'nonassoc' in the precedence table 994 995 precedence = ( 996 ('nonassoc', 'LESSTHAN', 'GREATERTHAN'), # Nonassociative operators 997 ('left', 'PLUS', 'MINUS'), 998 ('left', 'TIMES', 'DIVIDE'), 999 ('right', 'UMINUS'), # Unary minus operator 1000 ) 1001 1002 Patch contributed by Russ Cox. 1003 100410/30/02: beazley 1005 Modified the lexer to provide optional support for Python -O and -OO 1006 modes. To make this work, Python *first* needs to be run in 1007 unoptimized mode. This reads the lexing information and creates a 1008 file "lextab.py". Then, run lex like this: 1009 1010 # module foo.py 1011 ... 1012 ... 1013 lex.lex(optimize=1) 1014 1015 Once the lextab file has been created, subsequent calls to 1016 lex.lex() will read data from the lextab file instead of using 1017 introspection. In optimized mode (-O, -OO) everything should 1018 work normally despite the loss of doc strings. 1019 1020 To change the name of the file 'lextab.py' use the following: 1021 1022 lex.lex(lextab="footab") 1023 1024 (this creates a file footab.py) 1025 1026 1027Version 1.1 October 25, 2001 1028------------------------------ 1029 103010/25/01: beazley 1031 Modified the table generator to produce much more compact data. 1032 This should greatly reduce the size of the parsetab.py[c] file. 1033 Caveat: the tables still need to be constructed so a little more 1034 work is done in parsetab on import. 1035 103610/25/01: beazley 1037 There may be a possible bug in the cycle detector that reports errors 1038 about infinite recursion. I'm having a little trouble tracking it 1039 down, but if you get this problem, you can disable the cycle 1040 detector as follows: 1041 1042 yacc.yacc(check_recursion = 0) 1043 104410/25/01: beazley 1045 Fixed a bug in lex.py that sometimes caused illegal characters to be 1046 reported incorrectly. Reported by Sverre J�rgensen. 1047 10487/8/01 : beazley 1049 Added a reference to the underlying lexer object when tokens are handled by 1050 functions. The lexer is available as the 'lexer' attribute. This 1051 was added to provide better lexing support for languages such as Fortran 1052 where certain types of tokens can't be conveniently expressed as regular 1053 expressions (and where the tokenizing function may want to perform a 1054 little backtracking). Suggested by Pearu Peterson. 1055 10566/20/01 : beazley 1057 Modified yacc() function so that an optional starting symbol can be specified. 1058 For example: 1059 1060 yacc.yacc(start="statement") 1061 1062 Normally yacc always treats the first production rule as the starting symbol. 1063 However, if you are debugging your grammar it may be useful to specify 1064 an alternative starting symbol. Idea suggested by Rich Salz. 1065 1066Version 1.0 June 18, 2001 1067-------------------------- 1068Initial public offering 1069 1070