1331399 - The way we store job information on the front end is inefficient, wasting 10s of MB of memory (or more)

Status: NEW

(Tree Management :: Treeherder: Frontend, defect, P1)

Description

[William Lachance (:wlach)]

If you do a heap profile of treeherder in Chrome, you'll notice that a very large chunk of the data stored in Treeherder (especially on a branch with a lot of jobs like mozilla-inbound) in something called ThJobModel -- upwards of 64M in my most my recent testing. This is basically a literal translation of what we get from the jobs model, data structures like this:

{
    "build_architecture": "armv7",
    "build_os": "android",
    "build_platform": "android-4-0-armv7-api15",
    "build_platform_id": 153,
    "build_system_type": "buildbot",
    "end_timestamp": 0,
    "failure_classification_id": 1,
    "id": 69320706,
    "job_coalesced_to_guid": null,
    "job_group_description": "",
    "job_group_id": 2,
    "job_group_name": "unknown",
    "job_group_symbol": "?",
    "job_guid": "13669171854e32b8db0f5c8d8c38b4e73848c882",
    "job_type_description": "Build from sources",
    "job_type_id": 2,
    "job_type_name": "Build",
    "job_type_symbol": "B",
    "last_modified": "2017-01-16T16:01:06.712719",
    "machine_name": "unknown",
    "machine_platform_architecture": "armv7",
    "machine_platform_os": "android",
    "option_collection_hash": "102210fe594ee9b33d82058545b1ed14f4c8206e",
    "platform": "android-4-0-armv7-api15",
    "platform_option": "opt",
    "push_id": 160580,
    "reason": "unknown",
    "ref_data_name": "Android armv7 API 15+ try build",
    "result": "unknown",
    "result_set_id": 160580,
    "running_eta": 723,
    "signature": "b8d8be8202eb1c0461f60a309615dbea280889c4",
    "start_timestamp": 1484582367,
    "state": "running",
    "submit_timestamp": 1484582364,
    "tier": 1,
    "who": "unknown"
}

We could save a considerable amount of memory by factoring out some of the elements that are common between jobs into some kind of immutable data structure and then referencing that from the job objects themselves. E.g. in the above example, we could have a "build platform" object that looks like this and then just assign it to the buildPlatform:

{
    "architecture": "armv7",
    "os": "android",
    "platform": "android-4-0-armv7-api15",
    "id": 153
}

The same would be true for "machine_platform", "job group" and "job type"

To solve this bug, we'd probably want to create some kind of angular service/singleton to store the various permutations of machine/build/job type/job group information and then have a query function to fetch existing instances of this data (or store a new one) based on what we find in the jobs. Then when loading the jobs, we'd replace (remove) the job instance's string properties with direct references to this data structure.

Comment 1

[Cameron Dawson [:camd]]

This bug may be ancient and from the Angular days, but it's still relevant. I think this would help us quite a bit.

Comment 2

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/5237

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/5289

Comment 4

[Cameron Dawson [:camd]]

De-prioritizing Treeherder work in new reorg. So removing myself.