loader.py revision 12882
112882Sspwilson2@wisc.edu# Copyright (c) 2017 Mark D. Hill and David A. Wood 212882Sspwilson2@wisc.edu# All rights reserved. 312882Sspwilson2@wisc.edu# 412882Sspwilson2@wisc.edu# Redistribution and use in source and binary forms, with or without 512882Sspwilson2@wisc.edu# modification, are permitted provided that the following conditions are 612882Sspwilson2@wisc.edu# met: redistributions of source code must retain the above copyright 712882Sspwilson2@wisc.edu# notice, this list of conditions and the following disclaimer; 812882Sspwilson2@wisc.edu# redistributions in binary form must reproduce the above copyright 912882Sspwilson2@wisc.edu# notice, this list of conditions and the following disclaimer in the 1012882Sspwilson2@wisc.edu# documentation and/or other materials provided with the distribution; 1112882Sspwilson2@wisc.edu# neither the name of the copyright holders nor the names of its 1212882Sspwilson2@wisc.edu# contributors may be used to endorse or promote products derived from 1312882Sspwilson2@wisc.edu# this software without specific prior written permission. 1412882Sspwilson2@wisc.edu# 1512882Sspwilson2@wisc.edu# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 1612882Sspwilson2@wisc.edu# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 1712882Sspwilson2@wisc.edu# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 1812882Sspwilson2@wisc.edu# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 1912882Sspwilson2@wisc.edu# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 2012882Sspwilson2@wisc.edu# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 2112882Sspwilson2@wisc.edu# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 2212882Sspwilson2@wisc.edu# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 2312882Sspwilson2@wisc.edu# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 2412882Sspwilson2@wisc.edu# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 2512882Sspwilson2@wisc.edu# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 2612882Sspwilson2@wisc.edu# 2712882Sspwilson2@wisc.edu# Authors: Sean Wilson 2812882Sspwilson2@wisc.edu 2912882Sspwilson2@wisc.edu''' 3012882Sspwilson2@wisc.eduContains the :class:`Loader` which is responsible for discovering and loading 3112882Sspwilson2@wisc.edutests. 3212882Sspwilson2@wisc.edu 3312882Sspwilson2@wisc.eduLoading typically follows the following stages. 3412882Sspwilson2@wisc.edu 3512882Sspwilson2@wisc.edu1. Recurse down a given directory looking for tests which match a given regex. 3612882Sspwilson2@wisc.edu 3712882Sspwilson2@wisc.edu The default regex used will match any python file (ending in .py) that has 3812882Sspwilson2@wisc.edu a name starting or ending in test(s). If there are any additional 3912882Sspwilson2@wisc.edu components of the name they must be connected with '-' or '_'. Lastly, 4012882Sspwilson2@wisc.edu file names that begin with '.' will be ignored. 4112882Sspwilson2@wisc.edu 4212882Sspwilson2@wisc.edu The following names would match: 4312882Sspwilson2@wisc.edu 4412882Sspwilson2@wisc.edu - `tests.py` 4512882Sspwilson2@wisc.edu - `test.py` 4612882Sspwilson2@wisc.edu - `test-this.py` 4712882Sspwilson2@wisc.edu - `tests-that.py` 4812882Sspwilson2@wisc.edu - `these-test.py` 4912882Sspwilson2@wisc.edu 5012882Sspwilson2@wisc.edu These would not match: 5112882Sspwilson2@wisc.edu 5212882Sspwilson2@wisc.edu - `.test.py` - 'hidden' files are ignored. 5312882Sspwilson2@wisc.edu - `test` - Must end in '.py' 5412882Sspwilson2@wisc.edu - `test-.py` - Needs a character after the hypen. 5512882Sspwilson2@wisc.edu - `testthis.py` - Needs a hypen or underscore to separate 'test' and 'this' 5612882Sspwilson2@wisc.edu 5712882Sspwilson2@wisc.edu 5812882Sspwilson2@wisc.edu2. With all files discovered execute each file gathering its test items we 5912882Sspwilson2@wisc.edu care about collecting. (`TestCase`, `TestSuite` and `Fixture` objects.) 6012882Sspwilson2@wisc.edu 6112882Sspwilson2@wisc.eduAs a final note, :class:`TestCase` instances which are not put into 6212882Sspwilson2@wisc.edua :class:`TestSuite` by the test writer will be placed into 6312882Sspwilson2@wisc.edua :class:`TestSuite` named after the module. 6412882Sspwilson2@wisc.edu 6512882Sspwilson2@wisc.edu.. seealso:: :func:`load_file` 6612882Sspwilson2@wisc.edu''' 6712882Sspwilson2@wisc.edu 6812882Sspwilson2@wisc.eduimport os 6912882Sspwilson2@wisc.eduimport re 7012882Sspwilson2@wisc.eduimport sys 7112882Sspwilson2@wisc.eduimport traceback 7212882Sspwilson2@wisc.edu 7312882Sspwilson2@wisc.eduimport config 7412882Sspwilson2@wisc.eduimport log 7512882Sspwilson2@wisc.eduimport suite as suite_mod 7612882Sspwilson2@wisc.eduimport test as test_mod 7712882Sspwilson2@wisc.eduimport fixture as fixture_mod 7812882Sspwilson2@wisc.eduimport wrappers 7912882Sspwilson2@wisc.eduimport uid 8012882Sspwilson2@wisc.edu 8112882Sspwilson2@wisc.educlass DuplicateTestItemException(Exception): 8212882Sspwilson2@wisc.edu ''' 8312882Sspwilson2@wisc.edu Exception indicates multiple test items with the same UID 8412882Sspwilson2@wisc.edu were discovered. 8512882Sspwilson2@wisc.edu ''' 8612882Sspwilson2@wisc.edu pass 8712882Sspwilson2@wisc.edu 8812882Sspwilson2@wisc.edu 8912882Sspwilson2@wisc.edu# Match filenames that either begin or end with 'test' or tests and use 9012882Sspwilson2@wisc.edu# - or _ to separate additional name components. 9112882Sspwilson2@wisc.edudefault_filepath_regex = re.compile( 9212882Sspwilson2@wisc.edu r'(((.+[_])?tests?)|(tests?([-_].+)?))\.py$') 9312882Sspwilson2@wisc.edu 9412882Sspwilson2@wisc.edudef default_filepath_filter(filepath): 9512882Sspwilson2@wisc.edu '''The default filter applied to filepaths to marks as test sources.''' 9612882Sspwilson2@wisc.edu filepath = os.path.basename(filepath) 9712882Sspwilson2@wisc.edu if default_filepath_regex.match(filepath): 9812882Sspwilson2@wisc.edu # Make sure doesn't start with . 9912882Sspwilson2@wisc.edu return not filepath.startswith('.') 10012882Sspwilson2@wisc.edu return False 10112882Sspwilson2@wisc.edu 10212882Sspwilson2@wisc.edudef path_as_modulename(filepath): 10312882Sspwilson2@wisc.edu '''Return the given filepath as a module name.''' 10412882Sspwilson2@wisc.edu # Remove the file extention (.py) 10512882Sspwilson2@wisc.edu return os.path.splitext(os.path.basename(filepath))[0] 10612882Sspwilson2@wisc.edu 10712882Sspwilson2@wisc.edudef path_as_suitename(filepath): 10812882Sspwilson2@wisc.edu return os.path.split(os.path.dirname(os.path.abspath((filepath))))[-1] 10912882Sspwilson2@wisc.edu 11012882Sspwilson2@wisc.edudef _assert_files_in_same_dir(files): 11112882Sspwilson2@wisc.edu if __debug__: 11212882Sspwilson2@wisc.edu if files: 11312882Sspwilson2@wisc.edu directory = os.path.dirname(files[0]) 11412882Sspwilson2@wisc.edu for f in files: 11512882Sspwilson2@wisc.edu assert os.path.dirname(f) == directory 11612882Sspwilson2@wisc.edu 11712882Sspwilson2@wisc.educlass Loader(object): 11812882Sspwilson2@wisc.edu ''' 11912882Sspwilson2@wisc.edu Class for discovering tests. 12012882Sspwilson2@wisc.edu 12112882Sspwilson2@wisc.edu Discovered :class:`TestCase` and :class:`TestSuite` objects are wrapped by 12212882Sspwilson2@wisc.edu :class:`LoadedTest` and :class:`LoadedSuite` objects respectively. 12312882Sspwilson2@wisc.edu These objects provided additional methods and metadata about the loaded 12412882Sspwilson2@wisc.edu objects and are the internal representation used by testlib. 12512882Sspwilson2@wisc.edu 12612882Sspwilson2@wisc.edu To simply discover and load all tests using the default filter create an 12712882Sspwilson2@wisc.edu instance and `load_root`. 12812882Sspwilson2@wisc.edu 12912882Sspwilson2@wisc.edu >>> import os 13012882Sspwilson2@wisc.edu >>> tl = Loader() 13112882Sspwilson2@wisc.edu >>> tl.load_root(os.getcwd()) 13212882Sspwilson2@wisc.edu 13312882Sspwilson2@wisc.edu .. note:: If tests are not contained in a TestSuite, they will 13412882Sspwilson2@wisc.edu automatically be placed into one for the module. 13512882Sspwilson2@wisc.edu 13612882Sspwilson2@wisc.edu .. warn:: This class is extremely thread-unsafe. 13712882Sspwilson2@wisc.edu It modifies the sys path and global config. 13812882Sspwilson2@wisc.edu Use with care. 13912882Sspwilson2@wisc.edu ''' 14012882Sspwilson2@wisc.edu def __init__(self): 14112882Sspwilson2@wisc.edu self.suites = [] 14212882Sspwilson2@wisc.edu self.suite_uids = {} 14312882Sspwilson2@wisc.edu self.filepath_filter = default_filepath_filter 14412882Sspwilson2@wisc.edu 14512882Sspwilson2@wisc.edu # filepath -> Successful | Failed to load 14612882Sspwilson2@wisc.edu self._files = {} 14712882Sspwilson2@wisc.edu 14812882Sspwilson2@wisc.edu @property 14912882Sspwilson2@wisc.edu def schedule(self): 15012882Sspwilson2@wisc.edu return wrappers.LoadedLibrary(self.suites, fixture_mod.global_fixtures) 15112882Sspwilson2@wisc.edu 15212882Sspwilson2@wisc.edu def load_schedule_for_suites(self, *uids): 15312882Sspwilson2@wisc.edu files = {uid.UID.uid_to_path(id_) for id_ in uids} 15412882Sspwilson2@wisc.edu for file_ in files: 15512882Sspwilson2@wisc.edu self.load_file(file_) 15612882Sspwilson2@wisc.edu 15712882Sspwilson2@wisc.edu return wrappers.LoadedLibrary( 15812882Sspwilson2@wisc.edu [self.suite_uids[id_] for id_ in uids], 15912882Sspwilson2@wisc.edu fixture_mod.global_fixtures) 16012882Sspwilson2@wisc.edu 16112882Sspwilson2@wisc.edu def _verify_no_duplicate_suites(self, new_suites): 16212882Sspwilson2@wisc.edu new_suite_uids = self.suite_uids.copy() 16312882Sspwilson2@wisc.edu for suite in new_suites: 16412882Sspwilson2@wisc.edu if suite.uid in new_suite_uids: 16512882Sspwilson2@wisc.edu raise DuplicateTestItemException( 16612882Sspwilson2@wisc.edu "More than one suite with UID '%s' was defined" %\ 16712882Sspwilson2@wisc.edu suite.uid) 16812882Sspwilson2@wisc.edu new_suite_uids[suite.uid] = suite 16912882Sspwilson2@wisc.edu 17012882Sspwilson2@wisc.edu def _verify_no_duplicate_tests_in_suites(self, new_suites): 17112882Sspwilson2@wisc.edu for suite in new_suites: 17212882Sspwilson2@wisc.edu test_uids = set() 17312882Sspwilson2@wisc.edu for test in suite: 17412882Sspwilson2@wisc.edu if test.uid in test_uids: 17512882Sspwilson2@wisc.edu raise DuplicateTestItemException( 17612882Sspwilson2@wisc.edu "More than one test with UID '%s' was defined" 17712882Sspwilson2@wisc.edu " in suite '%s'" 17812882Sspwilson2@wisc.edu % (test.uid, suite.uid)) 17912882Sspwilson2@wisc.edu test_uids.add(test.uid) 18012882Sspwilson2@wisc.edu 18112882Sspwilson2@wisc.edu def load_root(self, root): 18212882Sspwilson2@wisc.edu ''' 18312882Sspwilson2@wisc.edu Load files from the given root directory which match 18412882Sspwilson2@wisc.edu `self.filepath_filter`. 18512882Sspwilson2@wisc.edu ''' 18612882Sspwilson2@wisc.edu if __debug__: 18712882Sspwilson2@wisc.edu self._loaded_a_file = True 18812882Sspwilson2@wisc.edu 18912882Sspwilson2@wisc.edu for directory in self._discover_files(root): 19012882Sspwilson2@wisc.edu if directory: 19112882Sspwilson2@wisc.edu _assert_files_in_same_dir(directory) 19212882Sspwilson2@wisc.edu for f in directory: 19312882Sspwilson2@wisc.edu self.load_file(f) 19412882Sspwilson2@wisc.edu 19512882Sspwilson2@wisc.edu def load_dir(self, directory): 19612882Sspwilson2@wisc.edu for dir_ in self._discover_files(directory): 19712882Sspwilson2@wisc.edu _assert_files_in_same_dir(dir_) 19812882Sspwilson2@wisc.edu for f in dir_: 19912882Sspwilson2@wisc.edu self.load_file(f) 20012882Sspwilson2@wisc.edu 20112882Sspwilson2@wisc.edu def load_file(self, path): 20212882Sspwilson2@wisc.edu path = os.path.abspath(path) 20312882Sspwilson2@wisc.edu 20412882Sspwilson2@wisc.edu if path in self._files: 20512882Sspwilson2@wisc.edu if not self._files[path]: 20612882Sspwilson2@wisc.edu raise Exception('Attempted to load a file which already' 20712882Sspwilson2@wisc.edu ' failed to load') 20812882Sspwilson2@wisc.edu else: 20912882Sspwilson2@wisc.edu log.test_log.debug('Tried to reload: %s' % path) 21012882Sspwilson2@wisc.edu return 21112882Sspwilson2@wisc.edu 21212882Sspwilson2@wisc.edu # Create a custom dictionary for the loaded module. 21312882Sspwilson2@wisc.edu newdict = { 21412882Sspwilson2@wisc.edu '__builtins__':__builtins__, 21512882Sspwilson2@wisc.edu '__name__': path_as_modulename(path), 21612882Sspwilson2@wisc.edu '__file__': path, 21712882Sspwilson2@wisc.edu } 21812882Sspwilson2@wisc.edu 21912882Sspwilson2@wisc.edu # Add the file's containing directory to the system path. So it can do 22012882Sspwilson2@wisc.edu # relative imports naturally. 22112882Sspwilson2@wisc.edu old_path = sys.path[:] 22212882Sspwilson2@wisc.edu sys.path.insert(0, os.path.dirname(path)) 22312882Sspwilson2@wisc.edu cwd = os.getcwd() 22412882Sspwilson2@wisc.edu os.chdir(os.path.dirname(path)) 22512882Sspwilson2@wisc.edu config.config.file_under_load = path 22612882Sspwilson2@wisc.edu 22712882Sspwilson2@wisc.edu new_tests = test_mod.TestCase.collector.create() 22812882Sspwilson2@wisc.edu new_suites = suite_mod.TestSuite.collector.create() 22912882Sspwilson2@wisc.edu new_fixtures = fixture_mod.Fixture.collector.create() 23012882Sspwilson2@wisc.edu 23112882Sspwilson2@wisc.edu def cleanup(): 23212882Sspwilson2@wisc.edu config.config.file_under_load = None 23312882Sspwilson2@wisc.edu sys.path[:] = old_path 23412882Sspwilson2@wisc.edu os.chdir(cwd) 23512882Sspwilson2@wisc.edu test_mod.TestCase.collector.remove(new_tests) 23612882Sspwilson2@wisc.edu suite_mod.TestSuite.collector.remove(new_suites) 23712882Sspwilson2@wisc.edu fixture_mod.Fixture.collector.remove(new_fixtures) 23812882Sspwilson2@wisc.edu 23912882Sspwilson2@wisc.edu try: 24012882Sspwilson2@wisc.edu execfile(path, newdict, newdict) 24112882Sspwilson2@wisc.edu except Exception as e: 24212882Sspwilson2@wisc.edu log.test_log.debug(traceback.format_exc()) 24312882Sspwilson2@wisc.edu log.test_log.warn( 24412882Sspwilson2@wisc.edu 'Exception thrown while loading "%s"\n' 24512882Sspwilson2@wisc.edu 'Ignoring all tests in this file.' 24612882Sspwilson2@wisc.edu % (path)) 24712882Sspwilson2@wisc.edu cleanup() 24812882Sspwilson2@wisc.edu return 24912882Sspwilson2@wisc.edu 25012882Sspwilson2@wisc.edu # Create a module test suite for those not contained in a suite. 25112882Sspwilson2@wisc.edu orphan_tests = set(new_tests) 25212882Sspwilson2@wisc.edu for suite in new_suites: 25312882Sspwilson2@wisc.edu for test in suite: 25412882Sspwilson2@wisc.edu # Remove the test if it wasn't already removed. 25512882Sspwilson2@wisc.edu # (Suites may contain copies of tests.) 25612882Sspwilson2@wisc.edu if test in orphan_tests: 25712882Sspwilson2@wisc.edu orphan_tests.remove(test) 25812882Sspwilson2@wisc.edu if orphan_tests: 25912882Sspwilson2@wisc.edu orphan_tests = sorted(orphan_tests, key=new_tests.index) 26012882Sspwilson2@wisc.edu # FIXME Use the config based default to group all uncollected 26112882Sspwilson2@wisc.edu # tests. 26212882Sspwilson2@wisc.edu # NOTE: This is automatically collected (we still have the 26312882Sspwilson2@wisc.edu # collector active.) 26412882Sspwilson2@wisc.edu suite_mod.TestSuite(tests=orphan_tests, 26512882Sspwilson2@wisc.edu name=path_as_suitename(path)) 26612882Sspwilson2@wisc.edu 26712882Sspwilson2@wisc.edu try: 26812882Sspwilson2@wisc.edu loaded_suites = [wrappers.LoadedSuite(suite, path) 26912882Sspwilson2@wisc.edu for suite in new_suites] 27012882Sspwilson2@wisc.edu 27112882Sspwilson2@wisc.edu self._verify_no_duplicate_suites(loaded_suites) 27212882Sspwilson2@wisc.edu self._verify_no_duplicate_tests_in_suites(loaded_suites) 27312882Sspwilson2@wisc.edu except Exception as e: 27412882Sspwilson2@wisc.edu log.test_log.warn('%s\n' 27512882Sspwilson2@wisc.edu 'Exception thrown while loading "%s"\n' 27612882Sspwilson2@wisc.edu 'Ignoring all tests in this file.' 27712882Sspwilson2@wisc.edu % (traceback.format_exc(), path)) 27812882Sspwilson2@wisc.edu else: 27912882Sspwilson2@wisc.edu log.test_log.info('Discovered %d tests and %d suites in %s' 28012882Sspwilson2@wisc.edu '' % (len(new_tests), len(loaded_suites), path)) 28112882Sspwilson2@wisc.edu 28212882Sspwilson2@wisc.edu self.suites.extend(loaded_suites) 28312882Sspwilson2@wisc.edu self.suite_uids.update({suite.uid: suite 28412882Sspwilson2@wisc.edu for suite in loaded_suites}) 28512882Sspwilson2@wisc.edu cleanup() 28612882Sspwilson2@wisc.edu 28712882Sspwilson2@wisc.edu def _discover_files(self, root): 28812882Sspwilson2@wisc.edu ''' 28912882Sspwilson2@wisc.edu Recurse down from the given root directory returning a list of 29012882Sspwilson2@wisc.edu directories which contain a list of files matching 29112882Sspwilson2@wisc.edu `self.filepath_filter`. 29212882Sspwilson2@wisc.edu ''' 29312882Sspwilson2@wisc.edu # Will probably want to order this traversal. 29412882Sspwilson2@wisc.edu for root, dirnames, filenames in os.walk(root): 29512882Sspwilson2@wisc.edu dirnames.sort() 29612882Sspwilson2@wisc.edu if filenames: 29712882Sspwilson2@wisc.edu filenames.sort() 29812882Sspwilson2@wisc.edu filepaths = [os.path.join(root, filename) \ 29912882Sspwilson2@wisc.edu for filename in filenames] 30012882Sspwilson2@wisc.edu filepaths = filter(self.filepath_filter, filepaths) 30112882Sspwilson2@wisc.edu if filepaths: 30212882Sspwilson2@wisc.edu yield filepaths 303