1263361 - Exception in thread when hitting error ResourceNotFound 404

Status: REOPENED

(Testing :: mozregression, defect)

Description

[Loic]

STR:
Run "mozregression --bits=32 --good=2015-03-28 --bad=2015-03-29"

Result:

0:07.84 INFO: Got as far as we can go bisecting nightlies...
 0:07.84 INFO: Last good revision: ad587ca628cf (2015-03-28)
 0:07.84 INFO: First bad revision: 385840329d91 (2015-03-29)
 0:07.85 INFO: Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=ad587ca628cf&tocha
nge=385840329d91

 0:07.85 INFO: Switching bisection method to taskcluster
 0:07.85 INFO: Getting mozilla-central builds between ad587ca628cf and 385840329
d91
Exception in thread Thread-6:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 94, in
 __getitem__
    return self._future_build_infos[i].build_info
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 41, in
 build_info
    self._build_info = self._fetch()
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 35, in
 _fetch
    return self.build_info_fetcher.find_build_info(self.data)
  File "C:\Python27\lib\site-packages\mozregression\fetch_build_info.py", line 1
34, in find_build_info
    status = self.queue.status(task_id)['status']
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 481, in apiCa
ll
    return self._makeApiCall(e, *args, **kwargs)
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 248, in _make
ApiCall
    return self._makeHttpRequest(entry['method'], route, payload)
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 450, in _make
HttpRequest
    superExc=rerr
TaskclusterRestFailure: koqPZhJfR02QOS-hUxOWPA does not correspond to a task tha
t exists.
Are you sure this task exists?
----
errorCode:  ResourceNotFound
statusCode: 404
requestInfo:
  method:   status
  params:   {"taskId":"koqPZhJfR02QOS-hUxOWPA"}
  payload:  {}
  time:     2016-04-09T11:07:24.147Z
details:
{
  "taskId": "koqPZhJfR02QOS-hUxOWPA"
}

Exception in thread Thread-5:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 94, in
 __getitem__
    return self._future_build_infos[i].build_info
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 41, in
 build_info
    self._build_info = self._fetch()
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 35, in
 _fetch
    return self.build_info_fetcher.find_build_info(self.data)
  File "C:\Python27\lib\site-packages\mozregression\fetch_build_info.py", line 1
34, in find_build_info
    status = self.queue.status(task_id)['status']
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 481, in apiCa
ll
    return self._makeApiCall(e, *args, **kwargs)
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 248, in _make
ApiCall
    return self._makeHttpRequest(entry['method'], route, payload)
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 450, in _make
HttpRequest
    superExc=rerr
TaskclusterRestFailure: miy9naCORsC5N3HGUH8Q0w does not correspond to a task tha
t exists.
Are you sure this task exists?
----
errorCode:  ResourceNotFound
statusCode: 404
requestInfo:
  method:   status
  params:   {"taskId":"miy9naCORsC5N3HGUH8Q0w"}
  payload:  {}
  time:     2016-04-09T11:07:24.147Z
details:
{
  "taskId": "miy9naCORsC5N3HGUH8Q0w"
}

Exception in thread Thread-7:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 94, in
 __getitem__
    return self._future_build_infos[i].build_info
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 41, in
 build_info
    self._build_info = self._fetch()
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 35, in
 _fetch
    return self.build_info_fetcher.find_build_info(self.data)
  File "C:\Python27\lib\site-packages\mozregression\fetch_build_info.py", line 1
34, in find_build_info
    status = self.queue.status(task_id)['status']
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 481, in apiCa
ll
    return self._makeApiCall(e, *args, **kwargs)
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 248, in _make
ApiCall
    return self._makeHttpRequest(entry['method'], route, payload)
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 450, in _make
HttpRequest
    superExc=rerr
TaskclusterRestFailure: miy9naCORsC5N3HGUH8Q0w does not correspond to a task tha
t exists.
Are you sure this task exists?
----
errorCode:  ResourceNotFound
statusCode: 404
requestInfo:
  method:   status
  params:   {"taskId":"miy9naCORsC5N3HGUH8Q0w"}
  payload:  {}
  time:     2016-04-09T11:07:26.135Z
details:
{
  "taskId": "miy9naCORsC5N3HGUH8Q0w"
}

Exception in thread Thread-9:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 94, in
 __getitem__
    return self._future_build_infos[i].build_info
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 41, in
 build_info
    self._build_info = self._fetch()
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 35, in
 _fetch
    return self.build_info_fetcher.find_build_info(self.data)
  File "C:\Python27\lib\site-packages\mozregression\fetch_build_info.py", line 1
34, in find_build_info
    status = self.queue.status(task_id)['status']
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 481, in apiCa
ll
    return self._makeApiCall(e, *args, **kwargs)
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 248, in _make
ApiCall
    return self._makeHttpRequest(entry['method'], route, payload)
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 450, in _make
HttpRequest
    superExc=rerr
TaskclusterRestFailure: koqPZhJfR02QOS-hUxOWPA does not correspond to a task tha
t exists.
Are you sure this task exists?
----
errorCode:  ResourceNotFound
statusCode: 404
requestInfo:
  method:   status
  params:   {"taskId":"koqPZhJfR02QOS-hUxOWPA"}
  payload:  {}
  time:     2016-04-09T11:07:26.186Z
details:
{
  "taskId": "koqPZhJfR02QOS-hUxOWPA"
}

Exception in thread Thread-8:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 94, in
 __getitem__
    return self._future_build_infos[i].build_info
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 41, in
 build_info
    self._build_info = self._fetch()
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 35, in
 _fetch
    return self.build_info_fetcher.find_build_info(self.data)
  File "C:\Python27\lib\site-packages\mozregression\fetch_build_info.py", line 1
34, in find_build_info
    status = self.queue.status(task_id)['status']
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 481, in apiCa
ll
    return self._makeApiCall(e, *args, **kwargs)
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 248, in _make
ApiCall
    return self._makeHttpRequest(entry['method'], route, payload)
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 450, in _make
HttpRequest
    superExc=rerr
TaskclusterRestFailure: wLmukegfTPGjs81WmxajUQ does not correspond to a task tha
t exists.
Are you sure this task exists?
----
errorCode:  ResourceNotFound
statusCode: 404
requestInfo:
  method:   status
  params:   {"taskId":"wLmukegfTPGjs81WmxajUQ"}
  payload:  {}
  time:     2016-04-09T11:07:26.460Z
details:
{
  "taskId": "wLmukegfTPGjs81WmxajUQ"
}

Traceback (most recent call last):
  File "C:\Python27\Scripts\mozregression-script.py", line 9, in <module>
    load_entry_point('mozregression==2.3.3', 'console_scripts', 'mozregression')
()
  File "C:\Python27\lib\site-packages\mozregression\main.py", line 308, in main
    sys.exit(method())
  File "C:\Python27\lib\site-packages\mozregression\main.py", line 137, in bisec
t_nightlies
    expand=DEFAULT_EXPAND)
  File "C:\Python27\lib\site-packages\mozregression\main.py", line 161, in _bise
ct_inbounds
    result = self._do_bisect(handler, good_rev, bad_rev, expand=expand)
  File "C:\Python27\lib\site-packages\mozregression\main.py", line 202, in _do_b
isect
    return self.bisector.bisect(handler, good, bad, **kwargs)
  File "C:\Python27\lib\site-packages\mozregression\bisector.py", line 563, in b
isect
    return self._bisect(handler, build_range)
  File "C:\Python27\lib\site-packages\mozregression\bisector.py", line 579, in _
bisect
    result = bisection.init_handler(index)
  File "C:\Python27\lib\site-packages\mozregression\bisector.py", line 362, in i
nit_handler
    self.handler.initialize()
  File "C:\Python27\lib\site-packages\mozregression\bisector.py", line 72, in in
itialize
    repo = self.build_range[-1].repo_url
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 94, in
 __getitem__
    return self._future_build_infos[i].build_info
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 41, in
 build_info
    self._build_info = self._fetch()
  File "C:\Python27\lib\site-packages\mozregression\build_range.py", line 35, in
 _fetch
    return self.build_info_fetcher.find_build_info(self.data)
  File "C:\Python27\lib\site-packages\mozregression\fetch_build_info.py", line 1
34, in find_build_info
    status = self.queue.status(task_id)['status']
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 481, in apiCa
ll
    return self._makeApiCall(e, *args, **kwargs)
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 248, in _make
ApiCall
    return self._makeHttpRequest(entry['method'], route, payload)
  File "C:\Python27\lib\site-packages\taskcluster\client.py", line 450, in _make
HttpRequest
    superExc=rerr
taskcluster.exceptions.TaskclusterRestFailure: koqPZhJfR02QOS-hUxOWPA does not c
orrespond to a task that exists.
Are you sure this task exists?
----
errorCode:  ResourceNotFound
statusCode: 404
requestInfo:
  method:   status
  params:   {"taskId":"koqPZhJfR02QOS-hUxOWPA"}
  payload:  {}
  time:     2016-04-09T11:07:28.585Z
details:
{
  "taskId": "koqPZhJfR02QOS-hUxOWPA"
}

Comment 1

[Loic]

William, could you check this issue with Mozreg. I'm still seeing it with the latest version.

It happens every time when the range is "old" (before 2016) with no m-i builds.

Comment 2

[William Lachance (:wlach)]

Paul, would this be something you'd have time to look into? I can do some investigation sometime in the next few weeks if you're too busy.

Comment 3

[Paul Yang [:pyang] (away)]

Look like taskcluster client api changed.
I'll try to trace it this week; leave ni? as reminder.

Comment 4

[Paul Yang [:pyang] (away)]

https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-central.nightly.2015/gecko.v2.mozilla-central.nightly.2015
build fetcher looked up list from tc, but the earliest build starts from 2015/08/

I think exceptions raised are correct, however we can catch it and simply give short message.

Comment 5

[Paul Yang [:pyang] (away)]

Attachment: catch_tc_exception

Will,
I try to move tc query into try block so it can be covered.  However it will continuously look up not existing build and pop up warning message.

Comment 6

[Jens]

I'm tracing down another bug using "mozregression --good 2015-05-10 --bad 2015-05-11 --repo mozilla-aurora" which results in the same error. The problem is that this leaves me with a huge Pushlog (https://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=8316a66bfed3&tochange=e2ce1aac996e). 

Sorry, but I do not know where else to ask: 
What possibilities do I have now to further narrow down a bug? 
(there's no "help" or whatever on http://mozilla.github.io/mozregression/)

Comment 7

[William Lachance (:wlach)]

(In reply to Jens from comment #6)
> I'm tracing down another bug using "mozregression --good 2015-05-10 --bad
> 2015-05-11 --repo mozilla-aurora" which results in the same error. The
> problem is that this leaves me with a huge Pushlog
> (https://hg.mozilla.org/releases/mozilla-aurora/
> pushloghtml?fromchange=8316a66bfed3&tochange=e2ce1aac996e). 
> 
> Sorry, but I do not know where else to ask: 
> What possibilities do I have now to further narrow down a bug? 
> (there's no "help" or whatever on http://mozilla.github.io/mozregression/)

Hey Jens, I don't think bisecting on mozilla-aurora is going to be very helpful, especially for such an old push. 

That pushlog seems to correspond to a merge from nightly to aurora, which I imagine means the bug was introduced on nightly at some point? I'd try to bisect it down to one day on that branch (the default). That'll give you a much smaller range.

Probably something like "--good 2015-01-01 --bad 2015-05-10" should work (you could probably get away with a more recent ---good, just trying to be super safe).

Comment 8

[William Lachance (:wlach)]

Comment on attachment 8797533 [details] [review]
catch_tc_exception

Yeah, with this patch mozregression eventually just errors out with:

 3:02.55 INFO: There are no build artifacts on inbound for these changesets (they are probably too old).

That's probably the best we can do in this situation. I applied the patch, thanks Paul!

Comment 9

[Loic]

Not fixed for me with Mozregression 2.3.8.

Comment 10

[Paul Yang [:pyang] (away)]

Loic, did you try to cleanup mozregression, for instance create a new virtualenv and run the steps again?
I suspect we don't have proper requirements defined.

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

Clear a needinfo that is pending on an inactive user.

For more information, please visit auto_nag documentation.