1121205 - Bugscache population deletes all rows before re-populating, can lead to races/problems

Status: RESOLVED DUPLICATE of bug 1231361

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

>[14:16]	RyanVM	bug 1113038 isn't being suggested anymore
>[14:16]	firebot	https://bugzil.la/1113038 — ASSIGNED, [e-mail] — Intermittent browser_UITour3.js | Popup was opened -
>[14:17]	RyanVM	nor is bug 1109932
>[14:17]	firebot	https://bugzil.la/1109932 — NEW, [e-mail] — Intermittent We have not been able to establish a telnet connection with the emulator
>[14:17]	RyanVM	which is #1 on orangefactor

I also see bug 1115781 not being suggested for https://treeherder.mozilla.org/logviewer.html#?job_id=1157876&repo=b2g-inbound
Bug 1111137 is not being suggested for https://treeherder.mozilla.org/logviewer.html#?job_id=5422543&repo=mozilla-inbound but this might be because the version number has bumped from 37 to 38 on trunk.

It's not all failing, though. Several bugs are still being correctly suggested for some failures.

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Bug 1118749 is missing for https://treeherder.mozilla.org/logviewer.html#?job_id=5424688&repo=mozilla-inbound

Comment 2

[Ed Morley [:emorley]]

(In reply to Wes Kocher (:KWierso) from comment #0)
> I also see bug 1115781 not being suggested for
> https://treeherder.mozilla.org/logviewer.html#?job_id=1157876&repo=b2g-
> inbound

-> https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=f9625445803a&filter-searchStr=b2g_emulator_vm%20b2g-inbound%20opt%20test%20reftest-8

Grabbing the bug suggestion artefact URL from the web console:
https://treeherder.mozilla.org/api/project/b2g-inbound/artifact/?job_id=1157876&name=Bug+suggestions&type=json

	{
		"search": "TEST-UNEXPECTED-FAIL | http://10.0.2.2:8888/tests/dom/canvas/test/reftest/webgl-orientation-test.html?__&alpha&preserve | application timed out after 330 seconds with no output",
		"bugs": {
			"open_recent": [],
			"all_others": []
		}
	},

The search term used _should_ have been:
"webgl-orientation-test.html?__&alpha&preserve"

Which should have been queried using:
https://treeherder.mozilla.org/api/bugscache/?search=webgl-orientation-test.html?__&alpha&preserve

-> Which at least at the moment, returns bug suggestions. Strange.

{
	"open_recent": [{
		"crash_signature": "",
		"resolution": "",
		"summary": "Intermittent webgl-orientation-test.html?__&alpha&________ | application timed out after 330 seconds with no output",
		"relevance": 1.00,
		"keywords": "intermittent-failure",
		"os": "Gonk (Firefox OS)",
		"id": 1108506
	},
	{
		"crash_signature": "",
		"resolution": "",
		"summary": "Intermittent webgl-orientation-test.html?__&alpha&preserve | application timed out after 330 seconds with no output",
		"relevance": 1.00,
		"keywords": "intermittent-failure",
		"os": "Gonk (Firefox OS)",
		"id": 1105683
	}],
	"all_others": []
}

> Bug 1111137 is not being suggested for
> https://treeherder.mozilla.org/logviewer.html#?job_id=5422543&repo=mozilla-
> inbound

-> https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=e7da7bc52c94&filter-searchStr=Android%204.0%20armv7%20API%2010%2B%20mozilla-inbound%20debug%20test%20mochitest-8

The bug suggestions artefact:
https://treeherder.mozilla.org/api/project/mozilla-inbound/artifact/?job_id=5422543&name=Bug+suggestions&type=json

	{
		"search": "520 INFO TEST-UNEXPECTED-FAIL | netwerk/test/mochitests/test_user_agent_overrides.html | Navigator UA not overridden at step 1 - got Mozilla/5.0 (Android; Tablet; rv:38.0) Gecko/38.0 Firefox/38.0, expected DummyUserAgent",
		"bugs": {
			"open_recent": [{
				"crash_signature": "",
				"resolution": "",
				"summary": "Intermittent test_user_agent_overrides.html | uncaught exception - NS_ERROR_FAILURE:  at http://mochi.test:8888/tests/netwerk/test/mochitests/test_user_agent_overrides.html:38",
				"relevance": 1.00,
				"keywords": "intermittent-failure",
				"os": "Android",
				"id": 1065132
			}],
			"all_others": [{
				"crash_signature": "",
				"resolution": "WORKSFORME",
				"summary": "Intermittent netwerk/test/mochitests/test_user_agent_overrides.html | application timed out after 330 seconds with no output",
				"relevance": 1.00,
				"keywords": "intermittent-failure",
				"os": "Mac OS X",
				"id": 920514
			}]
		}
	},

The presumed search term:
"test_user_agent_overrides.html"

Bugs lookup:
https://treeherder.mozilla.org/api/bugscache/?search=test_user_agent_overrides.html

Result:

{
	"open_recent": [{
		"crash_signature": "",
		"resolution": "",
		"summary": "Intermittent test_user_agent_overrides.html | uncaught exception - NS_ERROR_FAILURE:  at http://mochi.test:8888/tests/netwerk/test/mochitests/test_user_agent_overrides.html:38",
		"relevance": 1.00,
		"keywords": "intermittent-failure",
		"os": "Android",
		"id": 1065132
	},
	{
		"crash_signature": "",
		"resolution": "",
		"summary": "Intermittent test_user_agent_overrides.html | Navigator UA not overridden at step 1 - got Mozilla/5.0 (Android; Mobile; rv:37.0) Gecko/37.0 Firefox/37.0, expected DummyUserAgent",
		"relevance": 1.00,
		"keywords": "intermittent-failure,leave-open",
		"os": "Android",
		"id": 1111137
	}],
	"all_others": [{
		"crash_signature": "",
		"resolution": "WORKSFORME",
		"summary": "Intermittent netwerk/test/mochitests/test_user_agent_overrides.html | application timed out after 330 seconds with no output",
		"relevance": 1.00,
		"keywords": "intermittent-failure",
		"os": "Mac OS X",
		"id": 920514
	}]
}

-> Which strangely includes the missing bug, at least now.

Comment 3

[Ed Morley [:emorley]]

Random thought: I've just noticed that when we update the bugscache, it first purges *all* of the old bugs, and then repopulates it:
https://github.com/mozilla/treeherder-service/blob/master/treeherder/model/derived/refdata.py#L1308

If the insert of the updated jobs fails, or if a lookup races with the insert, then I guess we'd miss bugs, right? Ideally we should diff the old and new bugs list, and just prune the ones that are missing, then update the others.

Mauro, can you think of any other explanation?

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

And now I see a suggestion for 1113038.

Comment 5

[Mauro Doglio [:mdoglio]]

I can't think of any other reason, your idea sounds correct to me.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

I can't recall the last time I've seen something like this play out. I'm inclined to WFM at this point.

Comment 7

[Ed Morley [:emorley]]

I think this is more luck than anything else, since the code as written isn't ideal. I'll leave this open to improve it, though the priority is lower if it's not occurring often :-)

Comment 8

[William Lachance (:wlach)]

This was magically fixed when I landed bug 1231361. \o/

(thanks to :emorley for pointing out what I did)