1839428 - Ensure dispatcher is stopped even if blocking on it times out

Status: NEW

(Data Platform and Tools :: Glean: SDK, task, P1)

Description

[Jan-Erik Rediger [:janerik]]

During shutdown we try to clear out any remaining tasks in the dispatcher, but if it takes too long we just bail out.
There's now the possibility that after the timeout the dispatcher continues to run, interfering with data, even though we expected it to be finished.

Comment 1

[Jan-Erik Rediger [:janerik]]

Applying this patch (after PR #2503 lands):

diff --git glean-core/rlb/examples/prototype.rs glean-core/rlb/examples/prototype.rs
index 630638cc5..0e3ecc390 100644
--- glean-core/rlb/examples/prototype.rs
+++ glean-core/rlb/examples/prototype.rs
@@ -57,8 +57,14 @@ fn main() {
     glean::initialize(cfg, client_info);
 
     glean_metrics::sample_boolean.set(true);
+    _ = glean_metrics::sample_boolean.test_get_value(None);
 
     PrototypePing.submit(None);
 
+    glean_core::dispatcher::launch(|| {
+        std::thread::sleep(std::time::Duration::from_secs(15));
+    });
+
     glean::shutdown();
+    std::thread::sleep(std::time::Duration::from_secs(10));
 }
diff --git glean-core/src/lib.rs glean-core/src/lib.rs
index 45198dd70..983dc1a16 100644
--- glean-core/src/lib.rs
+++ glean-core/src/lib.rs
@@ -5,7 +5,6 @@
 #![allow(clippy::significant_drop_in_scrutinee)]
 #![allow(clippy::uninlined_format_args)]
 #![deny(rustdoc::broken_intra_doc_links)]
-#![deny(missing_docs)]
 
 //! Glean is a modern approach for recording and sending Telemetry data.
 //!
@@ -37,7 +36,7 @@ mod core_metrics;
 mod coverage;
 mod database;
 mod debug;
-mod dispatcher;
+pub mod dispatcher;
 mod error;
 mod error_recording;
 mod event_database;

then running: RUST_LOG=debug cargo run --example prototype
will result in this log (trimmed to the last relevant lines):

[2023-06-20T14:52:58Z INFO  glean_core::upload] No more pings to upload! You are done.
[2023-06-20T14:53:08Z ERROR glean_core] Timeout while blocking on the dispatcher. No further shutdown cleanup will happen.
[2023-06-20T14:53:13Z INFO  glean_core::scheduler] Metrics Ping Scheduler cancelled. Exiting.

That is: the timeout is hit and shutdown bails out, then the dispatcher resumes and runs this task, doing more stuff on Glean (like clearing the dirty flag).

We should probably mark the dispatcher as finished so it stops working as soon as the current task that blocked finished.