Closed Bug 1078450 Opened 10 years ago Closed 7 years ago

Combine/refactor the job artefacts for the failure summary and log viewer ('text_log_summary' and 'Bug suggestions')

Categories

(Tree Management :: Treeherder, defect, P4)

Product:
Tree Management
Component:
Treeherder
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: emorley, Unassigned)

References

(Blocks 1 open bug)

Details

Broken out from a recent email.

In order to reduce DB consumption from job artefacts, we should combine the artefacts for the log viewer and the failure summary, rather than duplicating them.
Blocks: 1062842
No longer blocks: 1062842
Priority: -- → P3
Examples of current artefacts from the same job:

# "Job Info"
{
	"job_details": [{
		"url": "https://hg.mozilla.org/build/mozharness/rev/39c4444434fd",
		"content_type": "link",
		"value": "https://hg.mozilla.org/build/mozharness/rev/39c4444434fd",
		"title": "script_revlink"
	},
	{
		"value": "purged clobber",
		"content_type": "raw_html"
	},
	{
		"url": "https://git.mozilla.org/?p=releases/gaia;a=tree;h=2a2b008f9ae957fe19ad540d233d86b5c0b6829e",
		"content_type": "link",
		"value": "https://git.mozilla.org/?p=releases/gaia;a=tree;h=2a2b008f9ae957fe19ad540d233d86b5c0b6829e",
		"title": "gaia_revlink"
	}],
	"logurl": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-flame-kk_eng-debug/1423830675/b2g_mozilla-inbound_flame-kk_eng-debug_dep-bm77-build1-build1.txt.gz"
}

# "Structured Log"
{
	"header": {
		"slave": "bld-linux64-spot-220",
		"buildid": "20150213043115",
		"builder": "b2g_mozilla-inbound_flame-kk_eng-debug_dep",
		"results": "success (0)",
		"starttime": "1423831631.23",
		"builduid": "9bdee7f7fa514bf6846b7f3c69234161",
		"revision": "848ee095492d"
	},
	"step_data": {
		"all_errors": [{
			"line": "04:50:26     INFO -  AttributeError: 'GitConfig' object has no attribute 'json'",
			"linenumber": 1135
		},
		{
			"line": "04:50:26    ERROR - Return code: 1",
			"linenumber": 1136
		}],
		"steps": [{
			"errors": [],
			"name": "set props: master",
			"started": "2015-02-13 04:47:11.232728",
			"started_linenumber": 8,
			"finished_linenumber": 10,
			"finished": "2015-02-13 04:47:11.233166",
			"result": "success",
			"duration": 0,
			"order": 0,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "set props: basedir",
			"started": "2015-02-13 04:47:11.233457",
			"started_linenumber": 12,
			"finished_linenumber": 42,
			"finished": "2015-02-13 04:47:13.440777",
			"result": "success",
			"duration": 2,
			"order": 1,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "downloading to buildprops.json",
			"started": "2015-02-13 04:47:13.441098",
			"started_linenumber": 44,
			"finished_linenumber": 45,
			"finished": "2015-02-13 04:47:13.509264",
			"result": "success",
			"duration": 0,
			"order": 2,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "'rm -rf ...'",
			"started": "2015-02-13 04:47:13.509559",
			"started_linenumber": 47,
			"finished_linenumber": 75,
			"finished": "2015-02-13 04:47:14.174642",
			"result": "success",
			"duration": 1,
			"order": 3,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "set props: script_repo_url",
			"started": "2015-02-13 04:47:14.174906",
			"started_linenumber": 77,
			"finished_linenumber": 79,
			"finished": "2015-02-13 04:47:14.175264",
			"result": "success",
			"duration": 0,
			"order": 4,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "'bash -c ...'",
			"started": "2015-02-13 04:47:14.175538",
			"started_linenumber": 81,
			"finished_linenumber": 120,
			"finished": "2015-02-13 04:47:14.572914",
			"result": "success",
			"duration": 0,
			"order": 5,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "set props: script_repo_revision script_repo_url",
			"started": "2015-02-13 04:47:14.573300",
			"started_linenumber": 122,
			"finished_linenumber": 154,
			"finished": "2015-02-13 04:47:15.824361",
			"result": "success",
			"duration": 1,
			"order": 6,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "'rm -rf ...'",
			"started": "2015-02-13 04:47:15.824679",
			"started_linenumber": 156,
			"finished_linenumber": 184,
			"finished": "2015-02-13 04:47:15.873069",
			"result": "success",
			"duration": 0,
			"order": 7,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "'hg clone ...'",
			"started": "2015-02-13 04:47:15.873390",
			"started_linenumber": 186,
			"finished_linenumber": 222,
			"finished": "2015-02-13 04:47:20.833096",
			"result": "success",
			"duration": 5,
			"order": 8,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "'hg update ...'",
			"started": "2015-02-13 04:47:20.833464",
			"started_linenumber": 224,
			"finished_linenumber": 253,
			"finished": "2015-02-13 04:47:21.403944",
			"result": "success",
			"duration": 1,
			"order": 9,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "set props: script_repo_revision",
			"started": "2015-02-13 04:47:21.404280",
			"started_linenumber": 255,
			"finished_linenumber": 285,
			"finished": "2015-02-13 04:47:21.754513",
			"result": "success",
			"duration": 0,
			"order": 10,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "downloading to oauth.txt",
			"started": "2015-02-13 04:47:21.754832",
			"started_linenumber": 287,
			"finished_linenumber": 288,
			"finished": "2015-02-13 04:47:21.770884",
			"result": "success",
			"duration": 0,
			"order": 11,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "tinderboxprint_script_revlink",
			"started": "2015-02-13 04:47:21.771184",
			"started_linenumber": 290,
			"finished_linenumber": 292,
			"finished": "2015-02-13 04:47:21.771564",
			"result": "success",
			"duration": 0,
			"order": 12,
			"error_count": 0
		},
		{
			"errors": [{
				"line": "04:50:26     INFO -  AttributeError: 'GitConfig' object has no attribute 'json'",
				"linenumber": 1135
			},
			{
				"line": "04:50:26    ERROR - Return code: 1",
				"linenumber": 1136
			}],
			"name": "'scripts/scripts/b2g_build.py --target ...'",
			"started": "2015-02-13 04:47:21.771842",
			"started_linenumber": 294,
			"finished_linenumber": 98370,
			"finished": "2015-02-13 06:27:49.147984",
			"result": "success",
			"duration": 6027,
			"order": 13,
			"error_count": 2
		},
		{
			"errors": [],
			"name": "set props: gecko_revision buildid compare_locales_revision tools_revision gaia_revision revision",
			"started": "2015-02-13 06:27:49.162923",
			"started_linenumber": 98372,
			"finished_linenumber": 98412,
			"finished": "2015-02-13 06:27:49.200629",
			"result": "success",
			"duration": 0,
			"order": 14,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "'rm -f ...'",
			"started": "2015-02-13 06:27:49.200917",
			"started_linenumber": 98414,
			"finished_linenumber": 98442,
			"finished": "2015-02-13 06:27:49.254240",
			"result": "success",
			"duration": 0,
			"order": 15,
			"error_count": 0
		},
		{
			"errors": [],
			"name": "reboot skipped",
			"started": "2015-02-13 06:27:49.254511",
			"started_linenumber": 98444,
			"finished_linenumber": 98445,
			"finished": "2015-02-13 06:27:49.254743",
			"result": "skipped",
			"duration": 0,
			"order": 16,
			"error_count": 0
		}]
	},
	"logurl": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-flame-kk_eng-debug/1423830675/b2g_mozilla-inbound_flame-kk_eng-debug_dep-bm77-build1-build1.txt.gz"
}

# "Bug suggestions"
[{
	"search": "AttributeError: 'GitConfig' object has no attribute 'json'",
	"bugs": {
		"open_recent": [],
		"all_others": []
	}
},
{
	"search": "Return code: 1",
	"bugs": {
		"open_recent": [],
		"all_others": []
	}
}]
We actually store the same failure lines not twice, but 3 times in the DB!

Twice in the artefact (unfortunately) named "Structured Log" - in 'all_errors' and also in the 'errors' property of each step, then another time in the artefact called "Bug Suggestions".
Instead of combining these artifacts into one, why don't we split the information in 3 artifacts:
1-Job steps
2-Job errors
3-bug suggestions
In the log viewer we will use 1 and 2, while in the bottom panel we will use 2 and 3.
Yeah totally agree - was just thinking the same.
We'll just need to make sure we can still correlate the step/line numbers, but that's easy enough. (Though some of the handling will probably change for the machine readable log viewer Cameron is working on).
Yeah the line number will be the link between 1,2 and 3.
I'm not sure Cam is going to use these artifacts for the new log viewer.
Priority: P3 → P4
From bug 1163144:

(In reply to Cameron Dawson [:camd] from 1163144 comment #0)
> A few issues here:
> 1. We have duplicated data in the different artifacts
> 2. text_log_summary has a "header" field that could be removed.  We should
> get that data from the job object itself.
> 
> The "all_errors" values in the blob of the ``text_log_summary`` are
> duplicated in 3 places:
> 
> 1. the "all_errors" field
> 2. the step where the errors came from (possibly multiple steps, but not
> usually)
> 3. the ``Bug suggestions`` artifact.
> 
> In the process of doing this, we may want to create some kind of "Status"
> for bug suggestions that indicate we need the artifact.  Currently we just
> save a "blank" one with an empty array, which indicates there are no error
> lines that needed them, but that we checked for them.
Summary: Combine the job artefacts for the failure summary and log viewer → Combine/refactor the job artefacts for the failure summary and log viewer ('text_log_summary' and 'Bug suggestions')
Depends on: 1175870
This has now been done elsewhere.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.