1069560 - Submit bug comments to Bugzilla directly rather than via TBPL's submitBugzillaComment.php

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P2)

Description

[Ed Morley [:emorley]]

Currently with treeherder, whenever someone classifies an intermittent failure with a bug number, we make a comment on the bug via TBPL's submitBugzillaComment.php

Longer term, I know we'd wanted to stop posting comments on bugs (apart from say weekly "this bug is in the top N failures this week" messages) - however this requires that OrangeFactor:
1) be more reliable (it currently doesn't support all job types & stats are sometimes wrong)
2) support the use cases that are currently supported by having these bug comments
-> which pretty much means we need to wait until OrangeFactor v2 which will use the treeherder API instead of ES.

Therefore similar to bug 1069502, if we presume that TBPL will want to be EOLed before OrangeFactor v2 is complete, we'll need to submit directly to Bugzilla here:
https://github.com/mozilla/treeherder-service/blob/e5967822541cb0a6fbdf954273bb79830d13370d/treeherder/etl/tbpl.py#L133

So we no longer depend on:
https://hg.mozilla.org/webtools/tbpl/file/default/php/submitBugzillaComment.php

Note:
The reason for doing server-side bug commenting is that we use a shared bugzilla account, the credentials for which need to be kept private. We'll therefore need to add these to the secret treeherder ENVs (I have the credentials, ping me for them).

Comment 1

[Ed Morley [:emorley]]

Attachment: Submit bug comments directly to Bugzilla rather than via TBPL

Not yet tested.

Submits a comment like that previously sent by https://hg.mozilla.org/webtools/tbpl/file/default/php/submitBugzillaComment.php

I've omitted the UTF-8 non-breaking space sanitisation that submitBugzillaComment.php does, since bug 569509 comment 2 (that added it) implies we should no longer hit this, since we construct the message text differently now.

Comment 2

[Legacy TBPL/Treeherder Robot]

test

Comment 3

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-12-21T12:56:33
log: http://local.treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-inbound&job_id=1817
repository: mozilla-inbound
who: emorley[at]mozilla[dot]com
machine: tst-linux64-spot-828
buildname: Ubuntu VM 12.04 x64 mozilla-inbound debug test jetpack
revision: c02c3b49ce95

TEST-UNEXPECTED-FAIL | addon-sdk/tests/test-content-events.test multiple tabs | TypeError: this.docShell is null
TypeError: this.docShell is null
TEST-UNEXPECTED-FAIL | addon-sdk/tests/test-content-events.test multiple tabs | Test timed out (after: START)
TEST-UNEXPECTED-FAIL | addon-sdk/tests/test-content-events.test multiple tabs | Should not be any unexpected tabs open
TypeError: tab.linkedBrowser.currentURI is null
TEST-UNEXPECTED-FAIL | addon-sdk/tests/test-content-events.test multiple tabs | Test output exceeded timeout (300s).
Exception: Test output exceeded timeout (300s).
TEST-UNEXPECTED-FAIL | layout-change/main.test compatibility | TypeError: window.getSelection is not a function
TypeError: window.getSelection is not a function
Assertion failure: mIsMainThread != MAIN_THREAD || !ipc::ParentProcessIsBlocked(), at /builds/slave/m-in-l64-d-0000000000000000000/build/src/xpcom/threads/nsThread.cpp:755
TEST-UNEXPECTED-FAIL | page-mod-debugger-post/main.testDebugger | Test timed out (after: the page-mod was attached to resource://test-page-mod-debugger-at-jetpack/page-mod-debugger-post/data/index.html)
TEST-UNEXPECTED-FAIL | page-mod-debugger-post/main.testDebugger | Should not be any unexpected tabs open
TEST-UNEXPECTED-FAIL | Jetpack startup | Test output exceeded timeout (300s).
Exception: Test output exceeded timeout (300s).

Comment 4

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-12-21T13:05:34
log: http://local.treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-inbound&job_id=1914
repository: mozilla-inbound
who: emorley[at]mozilla[dot]com
machine: tst-linux32-spot-718
buildname: Ubuntu VM 12.04 mozilla-inbound debug test jetpack
revision: c02c3b49ce95

TEST-UNEXPECTED-FAIL | addon-sdk/tests/test-content-events.test multiple tabs | TypeError: this.docShell is null
TypeError: this.docShell is null
TEST-UNEXPECTED-FAIL | addon-sdk/tests/test-content-events.test multiple tabs | Test timed out (after: START)
TEST-UNEXPECTED-FAIL | addon-sdk/tests/test-content-events.test multiple tabs | Should not be any unexpected tabs open
TypeError: tab.linkedBrowser.currentURI is null
TEST-UNEXPECTED-FAIL | addon-sdk/tests/test-content-events.test multiple tabs | Test output exceeded timeout (300s).
Exception: Test output exceeded timeout (300s).
TEST-UNEXPECTED-FAIL | layout-change/main.test compatibility | TypeError: window.getSelection is not a function
TypeError: window.getSelection is not a function
Assertion failure: mIsMainThread != MAIN_THREAD || !ipc::ParentProcessIsBlocked(), at /builds/slave/m-in-lx-d-00000000000000000000/build/src/xpcom/threads/nsThread.cpp:755
TEST-UNEXPECTED-FAIL | page-mod-debugger-post/main.testDebugger | Test timed out (after: the page-mod was attached to resource://test-page-mod-debugger-at-jetpack/page-mod-debugger-post/data/index.html)
TEST-UNEXPECTED-FAIL | page-mod-debugger-post/main.testDebugger | Should not be any unexpected tabs open
TEST-UNEXPECTED-FAIL | Jetpack startup | Test output exceeded timeout (300s).
Exception: Test output exceeded timeout (300s).

Comment 5

[Ed Morley [:emorley]]

Comment on attachment 8536890 [details] [review]
Submit bug comments directly to Bugzilla rather than via TBPL

PR updated, this is now ready for review, and works when tested locally.
Prior to merging this we'll obviously also need to update local.py on prod with the credentials for TBPLbot (I have them here).

Comment 6

[Ed Morley [:emorley]]

I added the Bugzilla credentials to dev's local.py, flipped the tbpl_transfer_bugs pref to true and this worked on dev after deploying the bugzilla-comment branch:

[2015-01-03 08:14:13,178: INFO/Worker-1] Sending data to https://bugzilla.mozilla.org/rest/bug/1114688/comment: {'comment': u'submit_timestamp: 2015-01-03T08:14:12\nlog: http://treeherder-dev.allizom.org/ui/logviewer.html#?repo=mozilla-central&job_id=64675\nrepository: mozilla-central\nwho: emorley[at]mozilla[dot]com\nmachine: tst-linux64-spot-283\nbuildname: Ubuntu VM 12.04 x64 mozilla-central opt test reftest\nrevision: 57e4e9c33bef\n\nREFTEST TEST-UNEXPECTED-FAIL | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/text-overflow/anonymous-block.html | image comparison (==), max difference: 138, number of differing pixels: 2\nREFTEST TEST-UNEXPECTED-PASS | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/forms/textarea/rtl.html | image comparison (!=), max difference: 110, number of differing pixels: 109\nREFTEST TEST-UNEXPECTED-FAIL | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/xul/tree-row-outline-1.xul | image comparison (==), max difference: 238, number of differing pixels: 142'}
[2015-01-03 08:14:13,180: INFO/Worker-1] Starting new HTTPS connection (1): bugzilla.mozilla.org
[2015-01-03 08:14:14,297: INFO/MainProcess] Task submit-bug-comment[547fa3cb-525e-4a1c-9889-d6ca3f32cc09] succeeded in 1.13779044151s: None

I'll undo the TBPL_BUGS_TRANSFER_ENABLED change and switch dev back to master for now - we'll want to update stage/prod with the credentials, after which I'll land this on master.

Comment 7

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/839091632ebe56032d12928f9a0b28dd13a64a42
Bug 1069560 - Submit comments directly to Bugzilla rather than via TBPL

Comment 8

[Ed Morley [:emorley]]

So I'm slightly confused here. I've deployed to stage, however attempting to classify a failure on stage, results in no comment to the bug.

Things I've checked:
1) TBPL_BUGS_TRANSFER_ENABLED isn't specified in stage's local.py, or in the environment defines specified by /etc/profile.d/treeherder.sh, so *should* be True - since that's the default set in base.py. (Whereas on dev it's set to False in local.py).
2) The TBPLbot credentials are in local.py for both the stage and prod directories on the admin node.
3) They are also in local.py on treeherder-rabbitmq1.stage
4) treeherder-rabbitmq1.stage has the changes made in the comment 7 commit in the working directory
5) newrelic doesn't report any errors on stage
6) Neither /var/log/celery/celery_worker_hp.log nor /var/log/run_celery_worker_hp-supervisor.log on treeherder-rabbitmq1.stage contain any errors, or any indication that the the tasks controlled by TBPL_BUGS_TRANSFER_ENABLED are actually being generated at all.
7) |curl https://bugzilla.mozilla.org/rest/| succeeds on rabbitmq1.stage, so it's not a flow issue

Presuming that:
a) treeherder-rabbitmq1.stage is the correct host for these tasks
b) the logs mentioned in #6 are the correct ones to be inspecting
c) the celery logging level is set the same as dev (where I saw activity in the log file for bug/ES tasks)

...then the most likely explanation is that none of the TBPL_BUGS_TRANSFER_ENABLED related tasks are being generated, since even if the commit here had broken bug commenting, the ElasticSearch ones should still be generating log entries.

Any ideas why TBPL_BUGS_TRANSFER_ENABLED isn't true? Or anything else I've missed?

Comment 9

[Mauro Doglio [:mdoglio]]

There's a missing netflow between the web nodes and treeherder-rabbitmq1.stage.private.scl3.mozilla.com, so the celery task will never be scheduled
:fubar can you please take care of it?

Comment 10

[Kendall Libby [:fubar] (he/him)]

submitted flow request. I see the flow listed in my docs, but I don't see it in any of the flow requests so it looks like I missed this one. :-(

Comment 11

[Ed Morley [:emorley]]

Thanks for spotting/sorting the missing flow :-)

Works now:

[2015-01-06 07:02:42,468: INFO/MainProcess] Task submit-star-comment[350e6417-2e26-4106-ba03-70ef938b5f99] succeeded in 0.253804732114s: None
[2015-01-06 07:02:42,506: INFO/Worker-1] Sending data to https://bugzilla.mozilla.org/rest/bug/1109123/comment: {'comment': u"submit_timestamp: 2015-01-06T07:02:41\nlog: https://treeherder.allizom.org/ui/logviewer.html#?repo=mozilla-central&job_id=94560\nrepository: mozilla-central\nwho: emorley[at]mozilla[dot]com\nmachine: tst-linux32-spot-609\nbuildname: Ubuntu VM 12.04 mozilla-central pgo test mochitest-e10s-1\nrevision: 2a193b7f395c\n\n1479 INFO TEST-UNEXPECTED-FAIL | dom/events/test/test_bug574663.html | Ctrl-scrolling should zoom while the user is touching the touchpad - didn't expect 1, but got it"}
[2015-01-06 07:02:44,064: INFO/MainProcess] Task submit-bug-comment[41c37c76-56b5-4b28-93ba-3361b4984008] succeeded in 1.59517880436s: None