Closed Bug 1659109 Opened 5 years ago Closed 5 years ago

Move FOG initialization off the main thread & provide pre-init buffer

Tracking

()

Status:

RESOLVED FIXED

Milestone:

83 Branch

Tracking Flags:

Tracking

Status

firefox83

---

fixed

People

(Reporter: janerik, Assigned: janerik)

References

Details

Attachments

(5 files, 2 obsolete files)

Bug 1659109 - Add a simple dispatch queue for FOG. r?chutten! 5 years ago Jan-Erik Rediger [:janerik] 47 bytes, text/x-phabricator-request		Details \| Review
Bug 1659109 - Use global dispatcher in Counter metric. r?chutten! 5 years ago Jan-Erik Rediger [:janerik] 47 bytes, text/x-phabricator-request		Details \| Review
Bug 1659109 - Add a dispatch queue for FOG. r?chutten! 5 years ago Jan-Erik Rediger [:janerik] 47 bytes, text/x-phabricator-request		Details \| Review
Bug 1659109 - Use global dispatcher in Counter metric. r?chutten! 5 years ago Jan-Erik Rediger [:janerik] 47 bytes, text/x-phabricator-request		Details \| Review
Bug 1659109 - Launch an uploader if a ping was submitted successfully. r?chutten! 5 years ago Jan-Erik Rediger [:janerik] 47 bytes, text/x-phabricator-request		Details \| Review
Bug 1659109 - Initiate shutdown of the dispatcher. r?chutten! 5 years ago Jan-Erik Rediger [:janerik] 47 bytes, text/x-phabricator-request		Details \| Review
Bug 1659109 - Block on the queue before getting data out of metrics. r?chutten! 5 years ago Jan-Erik Rediger [:janerik] 47 bytes, text/x-phabricator-request		Details \| Review

Jan-Erik Rediger [:janerik]

Assignee

Description

•

5 years ago

•

Edited

Currently glean-core is fully synchronous and so is FOG.
In other language implementations we move all initialization off the main thread.
Additionally all metric recording happens off the main thread and recordings prior to init are queued up in a "pre-init buffer"

We need to:

Replicate that behavior for FOG
Eventually guarantee that behavior in Glean proper.

Notes:

In Gecko we might want to use XPCom threads, not Rust's std::thread. See https://phabricator.services.mozilla.com/D57106#inline-357338
Maybe for now we're still fine with a single extra thread for the background work

Jan-Erik Rediger [:janerik]

Assignee

Comment 1

•

5 years ago

I have an early prototype of a dispatch queue in Rust as we have it on Kotlin/Swift.
Next steps:

I see if I can try them out inside FOG easily
I explore how/if that fits with xpcom
Eventually we write a proposal for threading in glean-core (bug 1615597)

Jan-Erik Rediger [:janerik]

Assignee

Comment 2

•

5 years ago

Attached file Bug 1659109 - Add a simple dispatch queue for FOG. r?chutten! (obsolete) — Details

Jan-Erik Rediger [:janerik]

Assignee

Comment 3

•

5 years ago

Attached file Bug 1659109 - Use global dispatcher in Counter metric. r?chutten! (obsolete) — Details

Depends on D87712

Jan-Erik Rediger [:janerik]

Assignee

Updated

•

5 years ago

Depends on: 1662121

Jan-Erik Rediger [:janerik]

Assignee

Updated

•

5 years ago

Depends on: 1662122

Jan-Erik Rediger [:janerik]

Assignee

Updated

•

5 years ago

Blocks: 1662121

No longer depends on: 1662121

Jan-Erik Rediger [:janerik]

Assignee

Updated

•

5 years ago

Blocks: 1662122

No longer depends on: 1662122

Phabricator Automation

Updated

•

5 years ago

Attachment #9171148 - Attachment is obsolete: true

Phabricator Automation

Updated

•

5 years ago

Attachment #9171147 - Attachment is obsolete: true

Jan-Erik Rediger [:janerik]

Assignee

Comment 4

•

5 years ago

Attached file Bug 1659109 - Add a dispatch queue for FOG. r?chutten! — Details

Jan-Erik Rediger [:janerik]

Assignee

Comment 5

•

5 years ago

Attached file Bug 1659109 - Use global dispatcher in Counter metric. r?chutten! — Details

Depends on D88837

Jan-Erik Rediger [:janerik]

Assignee

Comment 6

•

5 years ago

Attached file Bug 1659109 - Launch an uploader if a ping was submitted successfully. r?chutten! — Details

Depends on D88838

Jan-Erik Rediger [:janerik]

Assignee

Comment 7

•

5 years ago

Attached file Bug 1659109 - Initiate shutdown of the dispatcher. r?chutten! — Details

Depends on D88839

Phabricator Automation

Updated

•

5 years ago

Attachment #9173083 - Attachment description: Bug 1659109 - Add a simple dispatch queue for FOG. r?chutten! → Bug 1659109 - Add a dispatch queue for FOG. r?chutten!

Chris H-C :chutten

Updated

•

5 years ago

Blocks: 1664461

Pulsebot

Comment 8

•

5 years ago

Pushed by jrediger@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/c42a568625d0 Add a dispatch queue for FOG. r=chutten https://hg.mozilla.org/integration/autoland/rev/98e349acacfd Use global dispatcher in Counter metric. r=chutten https://hg.mozilla.org/integration/autoland/rev/1ef4a64d4e39 Launch an uploader if a ping was submitted successfully. r=chutten https://hg.mozilla.org/integration/autoland/rev/9a824e53d6f0 Initiate shutdown of the dispatcher. r=chutten

Bogdan Tara[:bogdan_tara | bogdant]

Comment 9

•

5 years ago

Backed out 4 changesets (bug 1659109) for dispatcher related bustage.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=build&fromchange=ac4c876a2e1ddfdbed714cd30d81976ff27fb747&tochange=4a924bf3a74daf85f816bfd97b6d2b1f94f5cdc2&selectedTaskRun=XYApIEk0Tii4nMCNQ_IVcw.0

Backout link: https://hg.mozilla.org/integration/autoland/rev/4a924bf3a74daf85f816bfd97b6d2b1f94f5cdc2

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=315968173&repo=autoland&lineNumber=14389

[task 2020-09-17T16:13:04.990Z] 16:13:04     INFO -       Running `/builds/worker/fetches/rustc/bin/rustdoc --edition=2018 --crate-type lib --test /builds/worker/checkouts/gecko/toolkit/components/glean/api/src/lib.rs --crate-name fog -L dependency=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps -L native=/builds/worker/workspace/obj-build/debug/build/libloading-469eacfa31c9fecf/out -L native=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/build/lmdb-rkv-sys-8c5b7a75ba61ac94/out -L dependency=/builds/worker/workspace/obj-build/debug/deps --extern bincode=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps/libbincode-24d9c011eb883f57.rlib --extern chrono=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps/libchrono-d50adc25e8a27af9.rlib --extern crossbeam_channel=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps/libcrossbeam_channel-ae9c4c6db6938659.rlib --extern fog=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps/libfog-5d0ca0bbb1944c9a.rlib --extern glean_core=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps/libglean_core-39e72257a05b5b79.rlib --extern log=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps/liblog-34812c6633e4a095.rlib --extern once_cell=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps/libonce_cell-334ddf84cd5499f5.rlib --extern serde=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps/libserde-196183453c31fde5.rlib --extern tempfile=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps/libtempfile-023393bb341c0f98.rlib --extern thiserror=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps/libthiserror-6ed05ba8f2c663c5.rlib --extern uuid=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps/libuuid-78ceca2978966c7a.rlib`
[task 2020-09-17T16:13:04.990Z] 16:13:04     INFO -  running 10 tests
[task 2020-09-17T16:13:04.991Z] 16:13:04     INFO -  test src/ipc.rs - ipc::TestNeedIpcRAII (line 80) ... ignored
[task 2020-09-17T16:13:04.991Z] 16:13:04     INFO -  test src/lib.rs -  (line 9) ... ignored
[task 2020-09-17T16:13:04.991Z] 16:13:04     INFO -  test src/ping_upload.rs - ping_upload (line 11) ... ignored
[task 2020-09-17T16:13:04.991Z] 16:13:04     INFO -  test src/private/labeled.rs - private::labeled::LabeledMetric (line 101) ... ignored
[task 2020-09-17T16:13:04.991Z] 16:13:04     INFO -  test src/private/labeled.rs - private::labeled::LabeledMetric (line 119) ... ignored
[task 2020-09-17T16:13:04.991Z] 16:13:04     INFO -  test src/private/string.rs - private::string::StringMetric (line 18) ... ignored
[task 2020-09-17T16:13:04.991Z] 16:13:04     INFO -  test src/private/string.rs - private::string::StringMetric (line 35) ... ignored
[task 2020-09-17T16:13:04.992Z] 16:13:04     INFO -  test src/dispatcher/mod.rs - dispatcher (line 38) ... FAILED
[task 2020-09-17T16:13:04.992Z] 16:13:04     INFO -  test src/dispatcher/mod.rs - dispatcher (line 9) ... FAILED
[task 2020-09-17T16:13:04.992Z] 16:13:04     INFO -  test src/dispatcher/mod.rs - dispatcher::Dispatcher (line 156) ... FAILED
[task 2020-09-17T16:13:04.992Z] 16:13:04     INFO -  failures:
[task 2020-09-17T16:13:04.992Z] 16:13:04     INFO -  ---- src/dispatcher/mod.rs - dispatcher (line 38) stdout ----
[task 2020-09-17T16:13:04.992Z] 16:13:04    ERROR -  error[E0433]: failed to resolve: use of undeclared type or module `dispatcher`
[task 2020-09-17T16:13:04.992Z] 16:13:04     INFO -   --> src/dispatcher/mod.rs:40:1
[task 2020-09-17T16:13:04.992Z] 16:13:04     INFO -    |
[task 2020-09-17T16:13:04.992Z] 16:13:04     INFO -  4 | dispatcher::flush_init();
[task 2020-09-17T16:13:04.993Z] 16:13:04     INFO -    | ^^^^^^^^^^ use of undeclared type or module `dispatcher`
[task 2020-09-17T16:13:04.993Z] 16:13:04    ERROR -  error[E0433]: failed to resolve: use of undeclared type or module `dispatcher`
[task 2020-09-17T16:13:04.993Z] 16:13:04     INFO -   --> src/dispatcher/mod.rs:42:1
[task 2020-09-17T16:13:04.993Z] 16:13:04     INFO -    |
[task 2020-09-17T16:13:04.993Z] 16:13:04     INFO -  6 | dispatcher::launch(|| {
[task 2020-09-17T16:13:04.993Z] 16:13:04     INFO -    | ^^^^^^^^^^ use of undeclared type or module `dispatcher`
[task 2020-09-17T16:13:04.993Z] 16:13:04    ERROR -  error[E0433]: failed to resolve: use of undeclared type or module `dispatcher`
[task 2020-09-17T16:13:04.993Z] 16:13:04     INFO -    --> src/dispatcher/mod.rs:47:1
[task 2020-09-17T16:13:04.994Z] 16:13:04     INFO -     |
[task 2020-09-17T16:13:04.994Z] 16:13:04     INFO -  11 | dispatcher::launch(|| {
[task 2020-09-17T16:13:04.994Z] 16:13:04     INFO -     | ^^^^^^^^^^ use of undeclared type or module `dispatcher`
[task 2020-09-17T16:13:04.994Z] 16:13:04     INFO -  error: aborting due to 3 previous errors
[task 2020-09-17T16:13:04.994Z] 16:13:04     INFO -  For more information about this error, try `rustc --explain E0433`.
[task 2020-09-17T16:13:04.994Z] 16:13:04     INFO -  Couldn't compile the test.
[task 2020-09-17T16:13:04.994Z] 16:13:04     INFO -  ---- src/dispatcher/mod.rs - dispatcher (line 9) stdout ----
[task 2020-09-17T16:13:04.994Z] 16:13:04    ERROR -  error[E0432]: unresolved import `dispatcher`
[task 2020-09-17T16:13:04.994Z] 16:13:04     INFO -   --> src/dispatcher/mod.rs:10:5
[task 2020-09-17T16:13:04.995Z] 16:13:04     INFO -    |
[task 2020-09-17T16:13:04.995Z] 16:13:04     INFO -  3 | use dispatcher::Dispatcher;
[task 2020-09-17T16:13:04.995Z] 16:13:04     INFO -    |     ^^^^^^^^^^ use of undeclared type or module `dispatcher`
[task 2020-09-17T16:13:04.995Z] 16:13:04    ERROR -  error: aborting due to previous error
[task 2020-09-17T16:13:04.995Z] 16:13:04     INFO -  For more information about this error, try `rustc --explain E0432`.
[task 2020-09-17T16:13:04.995Z] 16:13:04     INFO -  Couldn't compile the test.
[task 2020-09-17T16:13:04.995Z] 16:13:04     INFO -  ---- src/dispatcher/mod.rs - dispatcher::Dispatcher (line 156) stdout ----
[task 2020-09-17T16:13:04.995Z] 16:13:04    ERROR -  error[E0432]: unresolved import `dispatcher`
[task 2020-09-17T16:13:04.995Z] 16:13:04     INFO -   --> src/dispatcher/mod.rs:157:5
[task 2020-09-17T16:13:04.996Z] 16:13:04     INFO -    |
[task 2020-09-17T16:13:04.996Z] 16:13:04     INFO -  3 | use dispatcher::Dispatcher;
[task 2020-09-17T16:13:04.996Z] 16:13:04     INFO -    |     ^^^^^^^^^^ use of undeclared type or module `dispatcher`
[task 2020-09-17T16:13:04.996Z] 16:13:04    ERROR -  error: aborting due to previous error
[task 2020-09-17T16:13:04.996Z] 16:13:04     INFO -  For more information about this error, try `rustc --explain E0432`.
[task 2020-09-17T16:13:04.996Z] 16:13:04     INFO -  Couldn't compile the test.
[task 2020-09-17T16:13:04.996Z] 16:13:04     INFO -  failures:
[task 2020-09-17T16:13:04.996Z] 16:13:04     INFO -      src/dispatcher/mod.rs - dispatcher (line 38)
[task 2020-09-17T16:13:04.996Z] 16:13:04     INFO -      src/dispatcher/mod.rs - dispatcher (line 9)
[task 2020-09-17T16:13:04.997Z] 16:13:04     INFO -      src/dispatcher/mod.rs - dispatcher::Dispatcher (line 156)
[task 2020-09-17T16:13:04.997Z] 16:13:04     INFO -  test result: FAILED. 0 passed; 3 failed; 7 ignored; 0 measured; 0 filtered out
[task 2020-09-17T16:13:04.997Z] 16:13:04     INFO -     Doc-tests malloc_size_of_derive
[task 2020-09-17T16:13:04.998Z] 16:13:04     INFO -       Running `/builds/worker/fetches/rustc/bin/rustdoc --crate-type proc-macro --test /builds/worker/checkouts/gecko/third_party/rust/malloc_size_of_derive/lib.rs --crate-name malloc_size_of_derive -L dependency=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/deps -L native=/builds/worker/workspace/obj-build/debug/build/libloading-469eacfa31c9fecf/out -L native=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/debug/build/lmdb-rkv-sys-8c5b7a75ba61ac94/out -L dependency=/builds/worker/workspace/obj-build/debug/deps --extern malloc_size_of_derive=/builds/worker/workspace/obj-build/debug/deps/libmalloc_size_of_derive-968364b438410898.so --extern proc_macro2=/builds/worker/workspace/obj-build/debug/deps/libproc_macro2-6fc6519b18e55b42.rlib --extern syn=/builds/worker/workspace/obj-build/debug/deps/libsyn-39876c3a2d27bfae.rlib --extern synstructure=/builds/worker/workspace/obj-build/debug/deps/libsynstructure-2316346ce3d8b4de.rlib --extern proc_macro`
[task 2020-09-17T16:13:04.998Z] 16:13:04     INFO -  running 0 tests
[task 2020-09-17T16:13:04.998Z] 16:13:04     INFO -  test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
[task 2020-09-17T16:13:04.998Z] 16:13:04     INFO -  error: test failed, to rerun pass '--doc'
[task 2020-09-17T16:13:04.998Z] 16:13:04     INFO -  /builds/worker/checkouts/gecko/config/makefiles/rust.mk:342: recipe for target 'force-cargo-test-run' failed
[task 2020-09-17T16:13:04.998Z] 16:13:04    ERROR -  make[1]: *** [force-cargo-test-run] Error 101
[task 2020-09-17T16:13:04.998Z] 16:13:04     INFO -  make[1]: Leaving directory '/builds/worker/workspace/obj-build/toolkit/library/rust'
[task 2020-09-17T16:13:04.999Z] 16:13:04     INFO -  /builds/worker/checkouts/gecko/config/recurse.mk:99: recipe for target 'toolkit/library/rust/rusttests' failed
[task 2020-09-17T16:13:04.999Z] 16:13:04     INFO -  make: *** [toolkit/library/rust/rusttests] Error 2
[task 2020-09-17T16:13:04.999Z] 16:13:04     INFO -  11 compiler warnings present.
[task 2020-09-17T16:13:04.999Z] 16:13:04     INFO -  Hey! Builds initiated with `mach build $A_SPECIFIC_TARGET` may not always work, even if the code being built is correct. Consider doing a bare `mach build` instead.
[task 2020-09-17T16:13:05.039Z] 16:13:05    ERROR - Return code: 2
[task 2020-09-17T16:13:05.039Z] 16:13:05  WARNING - setting return code to 2
[task 2020-09-17T16:13:05.040Z] 16:13:05    FATAL - 'mach build -v pre-export export recurse_rusttests' did not run successfully. Please check log for errors.
[task 2020-09-17T16:13:05.040Z] 16:13:05    FATAL - Running post_fatal callback...
[task 2020-09-17T16:13:05.040Z] 16:13:05    FATAL - Exiting -1
[task 2020-09-17T16:13:05.040Z] 16:13:05     INFO - [mozharness: 2020-09-17 16:13:05.040111Z] Finished build step (failed)

Flags: needinfo?(jrediger)

Jan-Erik Rediger [:janerik]

Assignee

Comment 10

•

5 years ago

Attached file Bug 1659109 - Block on the queue before getting data out of metrics. r?chutten! — Details

In tests we don't want to run the tasks asynchronously, otherwise we run
into problems where metrics are set, but the data is not yet stored.
Just as in IPC we can ignore that part for mach rusttests.

If we eventually need to test from Rust that the dispatcher is working
as expected we can follow a similar pattern as for IPC where we use a
global static atomic variable we can toggle.

Jan-Erik Rediger [:janerik]

Assignee

Comment 11

•

5 years ago

There was indeed a bigger problem, not only in the doc tests (now removed), but also in the other tests.
Those were not run on try prior to landing. I fixed the test issues and added a try run for these explicitly. This will need another review and therefore will wait until after the soft-freeze most likely.

Flags: needinfo?(jrediger)

Phabricator Automation

Updated

•

5 years ago

Attachment #9176516 - Attachment description: Bug 1659109 - Make the dispatcher run tasks as blocking on the main thread. r?chutten! → Bug 1659109 - Block on the queue before getting data out of metrics. r?chutten!

Pulsebot

Comment 12

•

5 years ago

Pushed by jrediger@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/32c3d23de6ea Add a dispatch queue for FOG. r=chutten https://hg.mozilla.org/integration/autoland/rev/aa9157143036 Use global dispatcher in Counter metric. r=chutten https://hg.mozilla.org/integration/autoland/rev/0ffc40c1046a Launch an uploader if a ping was submitted successfully. r=chutten https://hg.mozilla.org/integration/autoland/rev/b3671a7196cf Initiate shutdown of the dispatcher. r=chutten https://hg.mozilla.org/integration/autoland/rev/fbe0ea62f4bb Block on the queue before getting data out of metrics. r=chutten

Natalia Csoregi [:nataliaCs]

Comment 13

•

5 years ago

Backed out for bustage on src/dispatcher/mod.rs

backout: https://hg.mozilla.org/integration/autoland/rev/998f8c709693ae189a05d5bd1ecf8c1387a48f1f

push: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=AjK3WGiHQXaxeSvPTPeyuA.0&revision=fbe0ea62f4bb50bfc5879a56667945697b2c90e7&searchStr=build

failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=316394606&repo=autoland&lineNumber=14033

[task 2020-09-22T19:19:39.499Z] 19:19:39 INFO - test src/dispatcher/mod.rs - dispatcher (line 9) ... FAILED
[task 2020-09-22T19:19:39.499Z] 19:19:39 INFO - failures:
[task 2020-09-22T19:19:39.499Z] 19:19:39 INFO - ---- src/dispatcher/mod.rs - dispatcher (line 38) stdout ----
[task 2020-09-22T19:19:39.499Z] 19:19:39 ERROR - error[E0433]: failed to resolve: use of undeclared type or module dispatcher
[task 2020-09-22T19:19:39.500Z] 19:19:39 INFO - --> src/dispatcher/mod.rs:40:1
[task 2020-09-22T19:19:39.500Z] 19:19:39 INFO - |
[task 2020-09-22T19:19:39.500Z] 19:19:39 INFO - 4 | dispatcher::flush_init();
[task 2020-09-22T19:19:39.500Z] 19:19:39 INFO - | ^^^^^^^^^^ use of undeclared type or module dispatcher
[task 2020-09-22T19:19:39.500Z] 19:19:39 ERROR - error[E0433]: failed to resolve: use of undeclared type or module dispatcher
[task 2020-09-22T19:19:39.500Z] 19:19:39 INFO - --> src/dispatcher/mod.rs:42:1
[task 2020-09-22T19:19:39.500Z] 19:19:39 INFO - |
[task 2020-09-22T19:19:39.500Z] 19:19:39 INFO - 6 | dispatcher::launch(|| {
[task 2020-09-22T19:19:39.501Z] 19:19:39 INFO - | ^^^^^^^^^^ use of undeclared type or module dispatcher
[task 2020-09-22T19:19:39.501Z] 19:19:39 ERROR - error[E0433]: failed to resolve: use of undeclared type or module dispatcher

Flags: needinfo?(jrediger)

Jan-Erik Rediger [:janerik]

Assignee

Comment 14

•

5 years ago

I failed to update the first patch in the stack. Fixed now and landing again (the try run used the updated patch)

Flags: needinfo?(jrediger)

Pulsebot

Comment 15

•

5 years ago

Pushed by jrediger@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/6aed6f240b16 Add a dispatch queue for FOG. r=chutten https://hg.mozilla.org/integration/autoland/rev/322053ce978f Use global dispatcher in Counter metric. r=chutten https://hg.mozilla.org/integration/autoland/rev/2b28eed21bad Launch an uploader if a ping was submitted successfully. r=chutten https://hg.mozilla.org/integration/autoland/rev/7deb04447157 Initiate shutdown of the dispatcher. r=chutten https://hg.mozilla.org/integration/autoland/rev/11d40183d5d0 Block on the queue before getting data out of metrics. r=chutten

Bogdan Tara[:bogdan_tara | bogdant]

Comment 16

•

5 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/6aed6f240b16
https://hg.mozilla.org/mozilla-central/rev/322053ce978f
https://hg.mozilla.org/mozilla-central/rev/2b28eed21bad
https://hg.mozilla.org/mozilla-central/rev/7deb04447157
https://hg.mozilla.org/mozilla-central/rev/11d40183d5d0

Status: ASSIGNED → RESOLVED

Closed: 5 years ago

status-firefox83: --- → fixed

Resolution: --- → FIXED

Target Milestone: --- → 83 Branch

You need to log in before you can comment on or make changes to this bug.