Closed
Bug 1431161
Opened 5 years ago
Closed 4 years ago
run windows 32 and 64 bit builds on windows10-64 hardware for talos performance tests
Categories
(Testing :: Talos, defect)
Testing
Talos
Tracking
(firefox60 fixed, firefox61 fixed)
RESOLVED
FIXED
mozilla61
People
(Reporter: jmaher, Assigned: rwood)
References
Details
(Whiteboard: [PI:March])
Attachments
(2 files)
Bug 1431161 - run windows 32 and 64 bit builds on windows10-64 hardware for talos performance tests;
59 bytes,
text/x-review-board-request
|
jmaher
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
pmoore
:
review+
|
Details |
as we migrate to new hardware we will not be installing windows 7 as an option on the new hardware. This means we will not have 32 bit coverage and there is still a need to ensure we don't have regressions. Once we have windows10 running on the new hardware and not in buildbot, we can easily turn on 32 bit binaries testing on the windows10-64 os/hardware. This is not needed for every push, so we will only run this on autoland. My thoughts are that we will do: * autoland only * only bisect/investigate if 5% regression and a reported regression is not seen on 64 bit windows10
Reporter | ||
Updated•5 years ago
|
Whiteboard: [PI:February]
Reporter | ||
Comment 1•4 years ago
|
||
lets make this bug track the work to make this official. here is a patch that I have been using to test: https://hg.mozilla.org/try/rev/a104cd781adbf49649662bdfeb73f417b221eb4c I suspect it won't be backwards compatible, we should: * fix windows7 xperf jobs to run properly (they run on a VM) * ensure mozharness config changes are in the right files and complete * consider splitting out reftest from that patch. :rwood, could you pick up this work in the short term?
Assignee | ||
Comment 2•4 years ago
|
||
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #1) > lets make this bug track the work to make this official. > > here is a patch that I have been using to test: > https://hg.mozilla.org/try/rev/a104cd781adbf49649662bdfeb73f417b221eb4c > > I suspect it won't be backwards compatible, we should: > * fix windows7 xperf jobs to run properly (they run on a VM) > * ensure mozharness config changes are in the right files and complete > * consider splitting out reftest from that patch. > > :rwood, could you pick up this work in the short term? Sure... I don't quite understand though - what mozharness config changes do you mean? I've never worked on xperf or refests, are they for that maybe? Thanks :) I'll file dependent bugs: - for the reftest part of your patch - for porting xperf win7 tests to run on 32 bit builds on win 10 - for mozharness config for ?
Flags: needinfo?(rwood) → needinfo?(jmaher)
Assignee | ||
Comment 3•4 years ago
|
||
Oh I see, you have mozharness configs already in testing/mozharness/configs/talos/windows_config.py I'll use this bug for that patch.
Flags: needinfo?(jmaher)
Reporter | ||
Comment 4•4 years ago
|
||
yeah, you figured out the mozharness bits- currently win7 xperf runs on VM, it should remain on VM when this patch is ready for review- no need for another bug. we have bug 1435844 for reftests- things are moving along!
Assignee | ||
Comment 5•4 years ago
|
||
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #4) > yeah, you figured out the mozharness bits- currently win7 xperf runs on VM, > it should remain on VM when this patch is ready for review- no need for > another bug. > > we have bug 1435844 for reftests- things are moving along! Ok thank you sir!
Assignee | ||
Comment 6•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b377ce9637ae127d37c6d74c2a11082de9c19f07
Assignee | ||
Comment 7•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c13027549fb17b1b7e6fdace2a9316b89a43762a
Assignee | ||
Updated•4 years ago
|
Assignee: nobody → rwood
Status: NEW → ASSIGNED
Assignee | ||
Comment 8•4 years ago
|
||
In my try run (coomment 7) I have the patch working except the known failure of xperf. It is running on the AWS VM.
Assignee | ||
Comment 9•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=3c370c529a50c695967db062ab1938614c95ba15
Assignee | ||
Comment 10•4 years ago
|
||
Update: The win32 tests (on Win 10 host) should run on: ['mozilla-beta', 'mozilla-central', 'mozilla-inbound', 'autoland', 'try'] Also note, talos g2 (damp) fails consistently on "complicated.netmonitor". There's an intermittent open for that however this seems consistent on the new h/w so I'm going to disable "complicated.netmonitor" from the damp test.
Comment 11•4 years ago
|
||
(In reply to Robert Wood [:rwood] from comment #10) > Update: The win32 tests (on Win 10 host) should run on: ['mozilla-beta', > 'mozilla-central', 'mozilla-inbound', 'autoland', 'try'] > > Also note, talos g2 (damp) fails consistently on "complicated.netmonitor". > There's an intermittent open for that however this seems consistent on the > new h/w so I'm going to disable "complicated.netmonitor" from the damp test. Could you do that only on windows? (Services.appinfo.OS == "WINNT")
Comment 12•4 years ago
|
||
Also, I fixed some races around netmonitor DAMP test in bug 1419327. It would be handy to check if it fixes this one?
Assignee | ||
Comment 13•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c3b5145e074c92a234249c6791ce20c9bfbccca5
Assignee | ||
Comment 14•4 years ago
|
||
(In reply to Alexandre Poirot [:ochameau] from comment #12) > Also, I fixed some races around netmonitor DAMP test in bug 1419327. > It would be handy to check if it fixes this one? Thanks Alexandre, here's a try run on the new talos windows hardware, with your damp test patch imported from bug 1419327: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9bd6e1fa2b3e408a36a855977c0c627e9a5f19e7
Assignee | ||
Comment 15•4 years ago
|
||
Xperf (win7-opt) runs on aws (i.e machine i-01579509c8d7db2cc) [1]: :27:43 INFO - u'xperf-e10s': {u'pagesets_name': u'tp5n.zip', 15:27:43 INFO - u'talos_options': [u'--xperf_path', 15:27:43 INFO - u'"c:/Program Files/Microsoft Windows Performance Toolkit/xperf.exe"'], 15:28:39 INFO - Calling ['Z:\\task_1518015759\\build\\venv\\Scripts\\python', 'Z:\\task_1518015759\\build\\tests\\talos\\talos\\run_tests.py', '--branchName', 'try', '--suite', 'xperf-e10s', '--executablePath', 'Z:\\task_1518015759\\build\\application\\firefox\\firefox', '--symbolsPath', 'https://queue.taskcluster.net/v1/task/Sq8JNtALRYqEf5n56ewXew/artifacts/public/build/target.crashreporter-symbols.zip', '--title', 'i-01579509c8d7db2cc', '--webServer', 'localhost', '--webServer', 'localhost', '--webServer', 'localhost', '--webServer', 'localhost', '--log-tbpl-level=debug', '--log-errorsummary=Z:\\task_1518015759\\build\\blobber_upload_dir\\xperf-e10s_errorsummary.log', '--log-raw=Z:\\task_1518015759\\build\\blobber_upload_dir\\xperf-e10s_raw.log'] with output_timeout 3600 15:28:40 INFO - ERROR: xperf.exe cannot be found at the path specified 15:28:40 ERROR - Return code: 1 [1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=c3b5145e074c92a234249c6791ce20c9bfbccca5&selectedJob=160860154
Assignee | ||
Comment 16•4 years ago
|
||
The xperf.exe path is the same on the existing xperf job on win7 aws [1] 14:46:14 INFO - u'xperf-e10s': {u'pagesets_name': u'tp5n.zip', 14:46:14 INFO - u'talos_options': [u'--xperf_path', 14:46:14 INFO - u'"c:/Program Files/Microsoft Windows Performance Toolkit/xperf.exe"'], I have no idea why it's failing on the taskcluster job (comment 15) [1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=talos&selectedJob=160852600
Assignee | ||
Comment 17•4 years ago
|
||
Ahh, 'c:/Program Files/Microsoft Windows Performance Toolkit' is missing from the path in the tc job, maybe that's it
Assignee | ||
Comment 18•4 years ago
|
||
(In reply to Robert Wood [:rwood] from comment #14) > (In reply to Alexandre Poirot [:ochameau] from comment #12) > > Also, I fixed some races around netmonitor DAMP test in bug 1419327. > > It would be handy to check if it fixes this one? > > Thanks Alexandre, here's a try run on the new talos windows hardware, with > your damp test patch imported from bug 1419327: > > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=9bd6e1fa2b3e408a36a855977c0c627e9a5f19e7 Still fails on 'complicated.netmonitor' unfortunately, so for now I'll disable that subtest inside DAMP on Windows only as Alexandre suggested in comment 11
Assignee | ||
Comment 19•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4ff90e81bf4be5792d7cefdf9a68e38befce7e0d
Assignee | ||
Comment 20•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa5147d0155c62829eee9087f7b94a83924dde0f
Assignee | ||
Comment 21•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ad8dd58883f9301b1310442042d5295cc512af2
Assignee | ||
Comment 22•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=68f797cd4a49cba390d7ba81c1a173bd5849e9e3
Assignee | ||
Comment 23•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=dcdaad27df0258e0f5f86ec6d5873fabc0785b9d
Assignee | ||
Comment 24•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8dcefa619b4c545e3810789861e73953ef7b11c7
Assignee | ||
Comment 25•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=940b95f59010e2087fba02b0dde88bac4cb29292
Assignee | ||
Comment 26•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ec1d2c72838b551ce0f7466c5757de7d898b3e75
Assignee | ||
Comment 27•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2743478bad063dae7cb0d8a26afa4fa1ff5ba4a6
Comment hidden (mozreview-request) |
Reporter | ||
Comment 29•4 years ago
|
||
mozreview-review |
Comment on attachment 8949562 [details] Bug 1431161 - run windows 32 and 64 bit builds on windows10-64 hardware for talos performance tests; https://reviewboard.mozilla.org/r/218916/#review224646 just need to sort out the win_worker_type_platform. ::: taskcluster/taskgraph/transforms/tests.py:928 (Diff revision 1) > test['worker-type'] = MACOSX_WORKER_TYPES['macosx64'] > elif test_platform.startswith('win'): > - win_worker_type_platform = WINDOWS_WORKER_TYPES[ > - test_platform.split('/')[0] > - ] > - if test.get('suite', '') == 'talos' and 'ccov' not in test['build-platform']: > + # for talos xperf we want win7vm; all else on win10 hw > + if test.get('suite', '') == 'talos' and "--suite=xperf-e10s" in \ > + test['mozharness']['extra-options']: > + win_worker_type_platform = WINDOWS_WORKER_TYPES['windows7-32'] I am concerned that all other win7 unittests will have problems as well- here this is for talos xperf only. Maybe: if suite == talos: wintype = win10 or something like: if test['virtualization'] == hardware: wintype = win10 ::: testing/mozharness/configs/talos/windows_vm_config.py:57 (Diff revision 1) > "win64": "python3_x64.manifest", > }, > "env": { > # python3 requires C runtime, found in firefox installation; see bug 1361732 > - "PATH": "%(PATH)s;c:\\slave\\test\\build\\application\\firefox;" > + "PATH": "%(PATH)s;c:\\slave\\test\\build\\application\\firefox;" \ > + "c:\\Program Files\\Microsoft Windows Performance Toolkit\\;" is this needed? ::: testing/talos/talos/tests/devtools/addon/content/damp.html:15 (Diff revision 1) > +// Bug 1400580 disable 'complicated.netmonitor' on Win > +ChromeUtils.import("resource://gre/modules/Services.jsm"); > +var run_complicated_netmonitor = true; > +if (Services.appinfo.OS == "WINNT") { > + run_complicated_netmonitor = false; > +} scope creep, but I am fine with it in here.
Attachment #8949562 -
Flags: review?(jmaher) → review-
Reporter | ||
Comment 30•4 years ago
|
||
:markco can you update us in this bug with the total number of available windows moonshot machines?
Flags: needinfo?(mcornmesser)
Comment 31•4 years ago
|
||
Right now there are 29 that are ready to pick up tasks. If it helps i can can get additional machines stood up Friday morning. Is there a specific number currently needed? I am planning Monday am at the latest to begin deploying the balance of the Windows nodes.
Flags: needinfo?(mcornmesser)
Assignee | ||
Comment 32•4 years ago
|
||
mozreview-review-reply |
Comment on attachment 8949562 [details] Bug 1431161 - run windows 32 and 64 bit builds on windows10-64 hardware for talos performance tests; https://reviewboard.mozilla.org/r/218916/#review224646 > I am concerned that all other win7 unittests will have problems as well- here this is for talos xperf only. Maybe: > if suite == talos: > wintype = win10 > > > or something like: > if test['virtualization'] == hardware: > wintype = win10 I'll try that thanks > is this needed? I thought it was but I'll take it out and try without thanks > scope creep, but I am fine with it in here. Yeah good point I'm going to file a separate bug for that and cc the test owner
Assignee | ||
Comment 33•4 years ago
|
||
I'm going to move the change to the DAMP test to it's own Bug 1437028
Assignee | ||
Comment 34•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=09b1a70b65a9ee21a20f4bbaef3a4a9735471e02
Comment hidden (mozreview-request) |
Reporter | ||
Comment 36•4 years ago
|
||
mozreview-review |
Comment on attachment 8949562 [details] Bug 1431161 - run windows 32 and 64 bit builds on windows10-64 hardware for talos performance tests; https://reviewboard.mozilla.org/r/218916/#review224804 looking much better, but still a concern in the transform. ::: taskcluster/taskgraph/transforms/tests.py:929 (Diff revisions 1 - 2) > - test['mozharness']['extra-options']: > - win_worker_type_platform = WINDOWS_WORKER_TYPES['windows7-32'] > - else: > win_worker_type_platform = WINDOWS_WORKER_TYPES['windows10-64'] > + else: > + win_worker_type_platform = WINDOWS_WORKER_TYPES['windows7-32'] this won't work as we are forcing win10-vm jobs to run on win7. This transform can change the machine type we choose for all jobs, in this case unittests and talos perf tests.
Attachment #8949562 -
Flags: review?(jmaher) → review-
Assignee | ||
Comment 37•4 years ago
|
||
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #36) > Comment on attachment 8949562 [details] > Bug 1431161 - run windows 32 and 64 bit builds on windows10-64 hardware for > talos performance tests; > > https://reviewboard.mozilla.org/r/218916/#review224804 > > looking much better, but still a concern in the transform. > > ::: taskcluster/taskgraph/transforms/tests.py:929 > (Diff revisions 1 - 2) > > - test['mozharness']['extra-options']: > > - win_worker_type_platform = WINDOWS_WORKER_TYPES['windows7-32'] > > - else: > > win_worker_type_platform = WINDOWS_WORKER_TYPES['windows10-64'] > > + else: > > + win_worker_type_platform = WINDOWS_WORKER_TYPES['windows7-32'] > > this won't work as we are forcing win10-vm jobs to run on win7. This > transform can change the machine type we choose for all jobs, in this case > unittests and talos perf tests. I didn't know there were any Win 10 vm jobs, I thought all win 10 was h/w. Alright scratch that one.
Reporter | ||
Comment 38•4 years ago
|
||
all mochitest, xpcshell, web-platform-tests, etc. run on vm- only talos and reftest run on hardware- reftest is due to issues with the win10 vm, ideally it would only be talos on hardware.
Assignee | ||
Comment 39•4 years ago
|
||
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #38) > all mochitest, xpcshell, web-platform-tests, etc. run on vm- only talos and > reftest run on hardware- reftest is due to issues with the win10 vm, ideally > it would only be talos on hardware. Ah, thanks, yeah I keep thinking this is only for talos and not *all* test jobs
Assignee | ||
Comment 40•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=bdd0b8188a47a933ec084b3b02c6481d751462c7
Assignee | ||
Comment 41•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f1f569596c14a3e56a9593d30cd670b0d089ae25
Assignee | ||
Comment 42•4 years ago
|
||
Thanks for the feedback, ok I *think* I have it correct now :)
Comment hidden (mozreview-request) |
Reporter | ||
Comment 44•4 years ago
|
||
mozreview-review |
Comment on attachment 8949562 [details] Bug 1431161 - run windows 32 and 64 bit builds on windows10-64 hardware for talos performance tests; https://reviewboard.mozilla.org/r/218916/#review224884 excellent, we need to wait until machines are available. Can you do a try run with: ./mach try -b do -p win32,win64 -u all -t all
Attachment #8949562 -
Flags: review?(jmaher) → review+
Assignee | ||
Comment 45•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=042ab2304e1d1d9af7e118f37a494a62396bae73
Assignee | ||
Comment 46•4 years ago
|
||
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #44) > Comment on attachment 8949562 [details] > Bug 1431161 - run windows 32 and 64 bit builds on windows10-64 hardware for > talos performance tests; > > https://reviewboard.mozilla.org/r/218916/#review224884 > > excellent, we need to wait until machines are available. Can you do a try > run with: > ./mach try -b do -p win32,win64 -u all -t all Thanks for the review, and right - we can't land it until the pool is ready. Good idea, thanks - landed '-u all -t all' on try (comment 45).
Comment hidden (mozreview-request) |
Assignee | ||
Comment 48•4 years ago
|
||
Rebased (and fixed conflicts) and landing on try again
Assignee | ||
Comment 49•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5a3953156915de4d37640dddb6cfb576693a91bf
Assignee | ||
Comment 50•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b11068c6e82e8193676ac590e6a3789981a5c93e
Comment hidden (mozreview-request) |
Comment 52•4 years ago
|
||
Pushed by rwood@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/a1711e96c622 run windows 32 and 64 bit builds on windows10-64 hardware for talos performance tests; r=jmaher
Comment 53•4 years ago
|
||
Backed out for talos performance test failures. backout: https://hg.mozilla.org/integration/autoland/rev/878d64506602a00a86f0c8e2eb970909a0276949 push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=a1711e96c6227cb839be7dee8e6c0d6ec9ae1750&selectedJob=163523588 failure log: https://tools.taskcluster.net/groups/ZYqwCbsUR5ejHuvQBaHUGQ/tasks/ejHxjg34ToKgAxr9uY3N0w/runs/0/logs/public%2Flogs%2Flive_backing.log [taskcluster 2018-02-21T19:37:47.587Z] TASK FAIL since the task payload is invalid. See errors: [taskcluster 2018-02-21T19:37:47.587Z] - supersederUrl: Additional property supersederUrl is not allowed [taskcluster 2018-02-21T19:37:47.588Z] Task not successful due to following exception(s): [taskcluster 2018-02-21T19:37:47.588Z] Exception 1) [taskcluster 2018-02-21T19:37:47.588Z] Validation of payload failed for task ejHxjg34ToKgAxr9uY3N0w [taskcluster 2018-02-21T19:37:47.588Z]
Flags: needinfo?(rwood)
Assignee | ||
Comment 54•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d7512e2243d5db463e1dad9444a33fc82354c6c2
Assignee | ||
Comment 55•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2fac654a176cdaec78c9232adaa8453485fa1ee0
Comment hidden (mozreview-request) |
Assignee | ||
Comment 57•4 years ago
|
||
Thanks Natalia, we have a potential fix under review
Flags: needinfo?(rwood)
Comment hidden (mozreview-request) |
Comment 59•4 years ago
|
||
mozreview-review |
Comment on attachment 8952844 [details] Bug 1431161 - Temporarily turn off coalescing on new win tc h/w; https://reviewboard.mozilla.org/r/222070/#review228352 Beautiful!
Attachment #8952844 -
Flags: review?(pmoore) → review+
Comment 60•4 years ago
|
||
Pushed by rwood@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/0160e724e111 run windows 32 and 64 bit builds on windows10-64 hardware for talos performance tests; r=jmaher https://hg.mozilla.org/integration/autoland/rev/213725db126c Temporarily turn off coalescing on new win tc h/w; r=pmoore
Comment 61•4 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/0160e724e111 https://hg.mozilla.org/mozilla-central/rev/213725db126c
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
status-firefox60:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
Reporter | ||
Comment 62•4 years ago
|
||
and here is the "massive" set of wins we get: https://treeherder.mozilla.org/perf.html#/alerts?id=11706
Comment 63•4 years ago
|
||
Backout by jmaher@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/5bc49a32f706 backout for win10 hardware failures. r=me
Comment 64•4 years ago
|
||
Backout by archaeopteryx@coole-files.de: https://hg.mozilla.org/mozilla-central/rev/c9ec0c37349c backout for win10 hardware failures. r=me a=backout CLOSED TREE
![]() |
||
Updated•4 years ago
|
Status: RESOLVED → REOPENED
status-firefox60:
fixed → ---
Flags: needinfo?(rwood)
Resolution: FIXED → ---
Target Milestone: mozilla60 → ---
Assignee | ||
Updated•4 years ago
|
Flags: needinfo?(rwood)
Comment 65•4 years ago
|
||
Looks like this backout also disabled the Windows QR talos test jobs from getting run at all. I had just turned those on recently in bug 1440968 and they're not showing up on TreeHerder any more.
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 68•4 years ago
|
||
Re-opened the review and rebased it, to be more prepared for when we try to land this again
Reporter | ||
Comment 69•4 years ago
|
||
:markco- any luck figuring out the disk space issues?
Flags: needinfo?(mcornmesser)
Reporter | ||
Updated•4 years ago
|
Whiteboard: [PI:February] → [PI:March]
Comment 70•4 years ago
|
||
The generic worker upgrade (Bug 1443589) will address this.
Depends on: 1443589
Flags: needinfo?(mcornmesser)
Comment 71•4 years ago
|
||
Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/4992808ab5a3 run windows 32 and 64 bit builds on windows10-64 hardware for talos performance tests. r=rwood
Comment 72•4 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/4992808ab5a3
Status: REOPENED → RESOLVED
Closed: 4 years ago → 4 years ago
status-firefox61:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
Reporter | ||
Updated•4 years ago
|
Whiteboard: [PI:March] → [PI:March][checkin-needed-beta]
Comment 73•4 years ago
|
||
bugherderuplift |
https://hg.mozilla.org/releases/mozilla-beta/rev/156e4b64363d
status-firefox60:
--- → fixed
Whiteboard: [PI:March][checkin-needed-beta] → [PI:March]
You need to log in
before you can comment on or make changes to this bug.
Description
•