1393670 - Optionally upload CC/GC log along with about:memory report for AWSY test

Status: NEW

(Testing :: AWSY, defect, P3)

Description

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

It will be really helpful to capture CC/GC log along with the memory measurement report while detecting regression on AWSY. CC/GC log + heapgraph scripts are the best tools we have to investigate memory leakage found in about:memory report.

It can be done in two different workflow in my imagination:

1) Add an option in try syntax. Engineer can manually turn on this option and submit try push to retrieve more information when a AWSY regression is detected.

2) Automatically trigger the extra test with CC/GC log captured when an perfherder alarm is fired. These information will be ready when a regression bug is filed so that developers can immediately use CC/GC log for debugging.

Comment 1

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

Attachment: save-cc-gc-log-in-awsy.patch

This is the patch I used for capturing CC/GC log while running awsy-test locally. Should be able to extend it to support saving CC/GC log on taskcluster.

Comment 2

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

Talked to @pyang this morning. He might have some idea on how to make it happen.

Comment 3

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

For mochitest we can manually set | browserEnv["MOZ_CC_LOG_DIRECTORY"] = os.environ["MOZ_UPLOAD_DIR"] | to upload CC/GC log as an artifact while running tests on try server. Not sure if this approach also works on awsy-test.

See https://developer.mozilla.org/en-US/docs/Mozilla/Performance/GC_and_CC_logs

Comment 4

[Paul Yang [:pyang] (away)]

(In reply to Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz) from comment #3)
> For mochitest we can manually set | browserEnv["MOZ_CC_LOG_DIRECTORY"] =
> os.environ["MOZ_UPLOAD_DIR"] | to upload CC/GC log as an artifact while
> running tests on try server. Not sure if this approach also works on
> awsy-test.
> 
> See
> https://developer.mozilla.org/en-US/docs/Mozilla/Performance/GC_and_CC_logs

It should work as expected if we re-use it.  In fact, https://dxr.mozilla.org/mozilla-central/source/testing/awsy/awsy/test_memory_usage.py#99
I put memory log into somewhere and then move them into MOZ_UPLOAD_DIR and it works just fine.

The other thing we should concern is should we add this config/option in try-syntax.

Joel, do you think it's feasible for adding those features?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

:pyang, thanks for working on this- I think this is useful to do.  Any concerns I have are related to storing large volumes of data which we never use- do you know what size of data we would store with each test run?

If we wanted to make this optional on try (so we don't store additional data by default), we would need some type of try syntax flag similar to the --geckoProfile which is parsed on the try syntax line and passed through taskcluster into mozharness and then in mozharness, passed to the test runner.  I know there is a lot of work to rethink this and make "custom actions" in taskcluster which support retriggering jobs or specifying jobs with certain attributes.

Comment 6

[Paul Yang [:pyang] (away)]

Agree extra storage is a major issue and adding option might be a quick approach.

I recall bug 1343327 which might have similar work. will check if we could perform same thing for this bug.