Open Bug 1331399 Opened 3 years ago Updated 5 months ago

The way we store job information on the front end is inefficient, wasting 10s of MB of memory (or more)

Categories

(Tree Management :: Treeherder: Frontend, defect, P1)

defect

Tracking

(Not tracked)

People

(Reporter: wlach, Assigned: camd)

References

Details

Attachments

(2 files)

If you do a heap profile of treeherder in Chrome, you'll notice that a very large chunk of the data stored in Treeherder (especially on a branch with a lot of jobs like mozilla-inbound) in something called ThJobModel -- upwards of 64M in my most my recent testing. This is basically a literal translation of what we get from the jobs model, data structures like this:

{
    "build_architecture": "armv7",
    "build_os": "android",
    "build_platform": "android-4-0-armv7-api15",
    "build_platform_id": 153,
    "build_system_type": "buildbot",
    "end_timestamp": 0,
    "failure_classification_id": 1,
    "id": 69320706,
    "job_coalesced_to_guid": null,
    "job_group_description": "",
    "job_group_id": 2,
    "job_group_name": "unknown",
    "job_group_symbol": "?",
    "job_guid": "13669171854e32b8db0f5c8d8c38b4e73848c882",
    "job_type_description": "Build from sources",
    "job_type_id": 2,
    "job_type_name": "Build",
    "job_type_symbol": "B",
    "last_modified": "2017-01-16T16:01:06.712719",
    "machine_name": "unknown",
    "machine_platform_architecture": "armv7",
    "machine_platform_os": "android",
    "option_collection_hash": "102210fe594ee9b33d82058545b1ed14f4c8206e",
    "platform": "android-4-0-armv7-api15",
    "platform_option": "opt",
    "push_id": 160580,
    "reason": "unknown",
    "ref_data_name": "Android armv7 API 15+ try build",
    "result": "unknown",
    "result_set_id": 160580,
    "running_eta": 723,
    "signature": "b8d8be8202eb1c0461f60a309615dbea280889c4",
    "start_timestamp": 1484582367,
    "state": "running",
    "submit_timestamp": 1484582364,
    "tier": 1,
    "who": "unknown"
}

We could save a considerable amount of memory by factoring out some of the elements that are common between jobs into some kind of immutable data structure and then referencing that from the job objects themselves. E.g. in the above example, we could have a "build platform" object that looks like this and then just assign it to the buildPlatform:

{
    "architecture": "armv7",
    "os": "android",
    "platform": "android-4-0-armv7-api15",
    "id": 153
}

The same would be true for "machine_platform", "job group" and "job type"

To solve this bug, we'd probably want to create some kind of angular service/singleton to store the various permutations of machine/build/job type/job group information and then have a query function to fetch existing instances of this data (or store a new one) based on what we find in the jobs. Then when loading the jobs, we'd replace (remove) the job instance's string properties with direct references to this data structure.
Blocks: 1067844
See Also: → 1084086
Component: Treeherder → Treeherder: Frontend

This bug may be ancient and from the Angular days, but it's still relevant. I think this would help us quite a bit.

Assignee: nobody → cdawson
Blocks: 1499551
Priority: -- → P2
Priority: P2 → P1
You need to log in before you can comment on or make changes to this bug.