1635204 - Fix flaky test_shared_presets[sample-suites]

Status: RESOLVED FIXED

(Developer Infrastructure :: Try, defect, P3)

Description

[Mitchell Hentges [:mhentges] 🦀]

When running ./mach python-test --python 2 tools/tryselect, one of the tests fails intermittently:

 0:05.05 =================================== FAILURES ===================================
 0:05.05 ______________________ test_shared_presets[sample-suites] ______________________
 0:05.05 
 0:05.05 run_mach = <function inner at 0x7f186520ced0>, shared_name = 'sample-suites'
 0:05.05 shared_preset = {'description': "Runs one chunk of every test suite plus all suites that aren't chunked. It is useful for testing infr...1$ !2$ !3$ !4$ !5$ !6$ !7$ !8$ !9$ !0$ !raptor !talos', "^test- 'raptor-speedometer | 'talos-g1"], 'selector': 'fuzzy'}
 0:05.05 
 0:05.05     def test_shared_presets(run_mach, shared_name, shared_preset):
 0:05.05         """This test makes sure that we don't break any of the in-tree presets when
 0:05.05         renaming/removing variables in any of the selectors.
 0:05.05         """
 0:05.05         assert 'description' in shared_preset
 0:05.05         assert 'selector' in shared_preset
 0:05.05     
 0:05.05         selector = shared_preset['selector']
 0:05.05         if selector == 'fuzzy':
 0:05.05             assert 'query' in shared_preset
 0:05.05             assert isinstance(shared_preset['query'], list)
 0:05.05     
 0:05.05         # Run the preset and assert there were no exceptions.
 0:05.05 >       assert run_mach(['try', '--no-push', '--preset', shared_name]) == 0
 0:05.05 E       AssertionError: assert 1 == 0
 0:05.05 E        +  where 1 = <function inner at 0x7f186520ced0>(['try', '--no-push', '--preset', 'sample-suites'])
 0:05.05 
 0:05.05 tools/tryselect/test/test_presets.py:56: AssertionError
 0:05.05 ----------------------------- Captured stdout call -----------------------------
 0:05.05 Error running mach:
 0:05.05 
 0:05.05     [u'try', u'--no-push', u'--preset', 'sample-suites']
 0:05.05 
 0:05.05 The error occurred in code that was called by the mach command. This is either
 0:05.05 a bug in the called code itself or in the way that mach is calling it.
 0:05.05 You can invoke |./mach busted| to check if this issue is already on file. If it
 0:05.05 isn't, please use |./mach busted file| to report it. If |./mach busted| is
 0:05.05 misbehaving, you can also inspect the dependencies of bug 1543241.
 0:05.05 
 0:05.05 If filing a bug, please include the full output of mach, including this error
 0:05.05 message.
 0:05.05 
 0:05.05 The details of the failure are as follows:
 0:05.05 
 0:05.05 IOError: [Errno 2] No such file or directory: u'/home/mitch/.mozbuild/srcdirs/firefox-bea071e6985e/cache/taskgraph/task_duration_history.json'
 0:05.05 
 0:05.05   File "/home/mitch/dev/firefox/tools/tryselect/mach_commands.py", line 206, in try_default
 0:05.05     'try', subcommand=sub, context=self._mach_context, argv=argv, **kwargs)
 0:05.05   File "/home/mitch/dev/firefox/python/mach/mach/registrar.py", line 152, in dispatch
 0:05.05     return self._run_command_handler(handler, context=context, **kwargs)
 0:05.05   File "/home/mitch/dev/firefox/python/mach/mach/registrar.py", line 109, in _run_command_handler
 0:05.05     result = fn(**kwargs)
 0:05.05   File "/home/mitch/dev/firefox/tools/tryselect/mach_commands.py", line 307, in try_fuzzy
 0:05.05     return self.run(**kwargs)
 0:05.05   File "/home/mitch/dev/firefox/tools/tryselect/mach_commands.py", line 177, in run
 0:05.05     return mod.run(**kwargs)
 0:05.05   File "/home/mitch/dev/firefox/tools/tryselect/selectors/fuzzy.py", line 390, in run
 0:05.05     closed_tree=closed_tree)
 0:05.05   File "/home/mitch/dev/firefox/tools/tryselect/push.py", line 146, in push_to_try
 0:05.05     display_push_estimates(try_task_config)
 0:05.05   File "/home/mitch/dev/firefox/tools/tryselect/push.py", line 124, in display_push_estimates
 0:05.05     dep_cache, task_labels_from_try_config(try_task_config), cache_dir)
 0:05.05   File "/home/mitch/dev/firefox/tools/tryselect/util/estimates.py", line 86, in duration_summary
 0:05.05     durations = task_duration_data(cache_dir)
 0:05.05   File "/home/mitch/dev/firefox/tools/tryselect/util/estimates.py", line 81, in task_duration_data
 0:05.05     with open(os.path.join(cache_dir, TASK_DURATION_CACHE)) as f:

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1635204: Mark flaky test, register with bug r?ahal

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

That patch ^ doesn't resolve the problem, but just annotates it with this bug.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/725ddeec3229
Mark flaky test, registers with bug r=ahal

Comment 4

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/725ddeec3229