Closed Bug 1448985 Opened 4 years ago Closed 4 years ago

Missing bug comments for intermittent failures

Categories

(Tree Management Graveyard :: OrangeFactor, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gbrown, Assigned: gbrown)

References

(Blocks 1 open bug)

Details

Some intermittent failure bugs are missing recent comments from the OF commenter (intermittent failures view for the bugs show recent failures, but there's no associated comment). Other bugs are okay.
Logs on brasstacks show errors beginning 03-25:

Submitting comment to bug 1447510 (1 occurrences)
Submitting comment to bug 1447698 (5 occurrences)
Submitting comment to bug 1377059 (10 occurrences)
HTTPError 401: {"documentation":"https://bugzilla.readthedocs.io/en/latest/api/","error":true,"code":410,"message":"You must log in 
before using this part of Bugzilla."}
Submitting comment to bug 1371906 (11 occurrences)
Submitting comment to bug 1371671 (1 occurrences)
HTTPError 401: {"documentation":"https://bugzilla.readthedocs.io/en/latest/api/","error":true,"code":410,"message":"You must log in 
before using this part of Bugzilla."}
Submitting comment to bug 1399348 (5 occurrences)
HTTPError 401: {"documentation":"https://bugzilla.readthedocs.io/en/latest/api/","error":true,"code":410,"message":"You must log in 
before using this part of Bugzilla."}
Submitting comment to bug 1447516 (5 occurrences)
Comment 1 was for this week's weekly comments.


The daily log is okay for 03-24, but failed on 03-25:

woo_commenter start at 2018-03-25 01:00:01.832244
Found 67 owner triaged components
Getting orange data for 2018-03-24 to 2018-03-24...
Getting bug data...
-> Fetching JSON from http://localhost/orangefactor/api/bybug?tree=all&startday=2018-03-24&endday=2018-03-24
Getting bug data...
-> Found in cache: /bybug?tree=all&startday=2018-03-24&endday=2018-03-24
Getting bug data...
-> Fetching JSON from http://localhost/orangefactor/api/bybug?tree=all&startday=2018-03-03&endday=2018-03-24
HTTPError 500 fetching http://localhost/orangefactor/api/bybug?tree=all&startday=2018-03-03&endday=2018-03-24: internal server error
Traceback (most recent call last):
  File "woo_commenter.py", line 268, in <module>
    main()
  File "woo_commenter.py", line 184, in main
    skip_bug_stats = skip_tb.stats_by_bug()
  File "/home/webtools/apps/orangefactor/src/orangefactor/woo_client.py", line 52, in stats_by_bug
    jdata = self.get_bybug()
  File "/home/webtools/apps/orangefactor/src/orangefactor/woo_client.py", line 114, in get_bybug
    return self.get_bugdata('/bybug?tree=%s&startday=%s&endday=%s' % (self.tree, self.startday, self.endday))
  File "/home/webtools/apps/orangefactor/src/orangefactor/woo_client.py", line 109, in get_bugdata
    jdata = self.get_json(url_path)
  File "/home/webtools/apps/orangefactor/src/orangefactor/woo_client.py", line 98, in get_json
    resp.raise_for_status()
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost/orangefactor/api/bybug?tree=all&sta
rtday=2018-03-03&endday=2018-03-24
The daily run is okay in --test mode, so I'm pretty sure comment 2 was a temporary failure.
The weekly run fails consistently, apparently on every bug comment.
The config file containing the api key has not been updated in the last year.
I'm guessing this has something to do with bug 1448919.

:mcote - Can you clarify / verify? Can I do anything to fix this on the brasstacks/woo_commenter side?
Flags: needinfo?(mcote)
dkl is looking into this.
Flags: needinfo?(mcote) → needinfo?(dkl)
(In reply to Geoff Brown [:gbrown] from comment #7)
> I'm guessing this has something to do with bug 1448919.
> 
> :mcote - Can you clarify / verify? Can I do anything to fix this on the
> brasstacks/woo_commenter side?

What is the hostname OF is trying to connect to get API data from BMO? BMO is now in an AWS cluster and certain hostnames may not work anymore.
Flags: needinfo?(gbrown)
OF uses https://bugzilla.mozilla.org/


I'm pretty sure we are failing when submitting a comment at:

https://hg.mozilla.org/automation/orangefactor/file/tip/woo_commenter.py#l86

and the response we're seeing is:

HTTPError 401: {"documentation":"https://bugzilla.readthedocs.io/en/latest/api/","error":true,"code":410,"message":"You must log in before using this part of Bugzilla."}

The session is using an api key:

https://hg.mozilla.org/automation/orangefactor/file/tip/woo_commenter.py#l167
Flags: needinfo?(gbrown)
(In reply to Geoff Brown [:gbrown] from comment #10)
> OF uses https://bugzilla.mozilla.org/
> 
> 
> I'm pretty sure we are failing when submitting a comment at:
> 
> https://hg.mozilla.org/automation/orangefactor/file/tip/woo_commenter.py#l86
> 
> and the response we're seeing is:
> 
> HTTPError 401:
> {"documentation":"https://bugzilla.readthedocs.io/en/latest/api/","error":
> true,"code":410,"message":"You must log in before using this part of
> Bugzilla."}
> 
> The session is using an api key:
> 
> https://hg.mozilla.org/automation/orangefactor/file/tip/woo_commenter.py#l167

Can you email the api key and I can see if that key still exists in the DB.

dkl@mozilla.com
Flags: needinfo?(dkl) → needinfo?(gbrown)
(In reply to David Lawrence [:dkl] from comment #11)
> Can you email the api key and I can see if that key still exists in the DB.

Done.
Flags: needinfo?(gbrown)
(In reply to Geoff Brown [:gbrown] from comment #12)
> (In reply to David Lawrence [:dkl] from comment #11)
> > Can you email the api key and I can see if that key still exists in the DB.
> 
> Done.

Ok. I have looked and the API key you sent me is still there and works for a simple test I have done. Now on March 24th around 12pm EST, BMO moved to AWS and there were quiet a few hiccups from that time til today (26th). So it could very well have been related to the move and us getting the bugs fixed as they came through. Are they working as of this afternoon? Is there any way I can get a sample call from brasstacks to bmo that is giving the login error? The links in comment 2 are all to localhost.

dkl
Flags: needinfo?(gbrown)
Still failing as of this morning.


I've verified now that we are failing at:

https://hg.mozilla.org/automation/orangefactor/file/tip/woo_commenter.py#l86

    try:
        r = bmo_session.put(BZ_API_URL % bug_id, json=params, timeout=30)
        r.raise_for_status()
    except (ConnectionError, Timeout) as e:
        print "%s: %s" % (e.__class__.__name__, str(e))
    except HTTPError:
        print "HTTPError %s: %s" % (r.status_code, r.text)
        
For example, with BZ_API_URL % bug_id == "https://bugzilla.mozilla.org/rest/bug/1391756" 

and params ==

{'comment': {'body': '2 failures in 822 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    \n\nRepository breakdown:\n* mozilla-inbound: 2\n\nPlatform breakdown:\n* windows10-64: 1\n* osx-10-10: 1\n\nFor more details, see:\nhttps://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1391756&startday=2018-03-20&endday=2018-03-26&tree=trunk'}}

we catch an HTTPError and print:

HTTPError 401: {"documentation":"https://bugzilla.readthedocs.io/en/latest/api/","error":true,"code":410,"message":"You must log in before using this part of Bugzilla."}
Flags: needinfo?(gbrown)
(In reply to David Lawrence [:dkl] from comment #13)
> The links in comment 2 are all to localhost.

Comment 2 was a temporary and separate problem: I do not see that problem any longer.
I was able to PUT a comment using the API key to production on a test bug without error.

Could you do the following from the same host as the script that is failing and report if it works? Please post output.

curl -v -X PUT -k -H 'x-bugzilla-api-key: <API_KEY>' -i 'https://bugzilla.mozilla.org/rest/bug/1449217' --data '{"comment":{"body":"Test comment"}}'

dkl
Flags: needinfo?(gbrown)
i've traced this down to an issue with how aws-bmo is handling the auth header.

gbrown: change the header in https://hg.mozilla.org/automation/orangefactor/file/tip/woo_commenter.py#l167

- bmo_session.headers['X_BUGZILLA_API_KEY'] = bugzilla_api_key
+ bmo_session.headers['X-BUGZILLA-API-KEY'] = bugzilla_api_key

we'll work on fixing the crux of the problem.
Thanks :glob! Thanks :dkl!

The change to X-BUGZILLA-API-KEY works great.

From my perspective, we can resolve this bug now; do you want to keep it open for anything?
Flags: needinfo?(gbrown)
We're tracking the proper fix over in bug 1449234.  I'll close this for you. :)
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Product: Tree Management → Tree Management Graveyard
You need to log in before you can comment on or make changes to this bug.