Closed Bug 1712751 Opened 6 months ago Closed 1 month ago

[intermittent] Missing multiple gcno files

Categories

(Testing :: Code Coverage, defect)

defect

Tracking

(firefox95 fixed)

VERIFIED FIXED
95 Branch
Tracking Status
firefox95 --- fixed

People

(Reporter: jkratzer, Assigned: marco)

References

(Blocks 1 open bug, Regressed 1 open bug)

Details

Attachments

(1 file)

We are routinely noticing that key gcno files are missing from the linux64 --enable-fuzzing --enable-coverage builds. This appears to be happening intermittently. The most recent instance of this occurred between 5/22 (574a43c224bd) and 5/23 (9a3c2b7dfe84).

The following is a diff of x86_64-unknown-linux-gnu between those two builds.

@@ -3589,39 +3580,118 @@
 ./x86_64-unknown-linux-gnu/release/build/rayon-b9e429745da1a48b/out/probe0.gcno
 ./x86_64-unknown-linux-gnu/release/build/rayon-b9e429745da1a48b/out/probe1.gcno
 ./x86_64-unknown-linux-gnu/release/build/swgl-94319647fa97e54e/out/src/gl.gcno
+./x86_64-unknown-linux-gnu/release/deps/adler.gcno
+./x86_64-unknown-linux-gnu/release/deps/ahash.gcno
+./x86_64-unknown-linux-gnu/release/deps/aho_corasick.gcno
+./x86_64-unknown-linux-gnu/release/deps/anyhow.gcno
 ./x86_64-unknown-linux-gnu/release/deps/app_services_logger.gcno
 ./x86_64-unknown-linux-gnu/release/deps/app_units.gcno
+./x86_64-unknown-linux-gnu/release/deps/arrayref.gcno
 ./x86_64-unknown-linux-gnu/release/deps/arrayvec.gcno
-./x86_64-unknown-linux-gnu/release/deps/audioipc_client.gcno
-./x86_64-unknown-linux-gnu/release/deps/audioipc.gcno
-./x86_64-unknown-linux-gnu/release/deps/audioipc_server.gcno
+./x86_64-unknown-linux-gnu/release/deps/arrayvec.gcno
+./x86_64-unknown-linux-gnu/release/deps/ash.gcno
+./x86_64-unknown-linux-gnu/release/deps/atomic_refcell.gcno
+./x86_64-unknown-linux-gnu/release/deps/atty.gcno
+./x86_64-unknown-linux-gnu/release/deps/audio_thread_priority.gcno
+./x86_64-unknown-linux-gnu/release/deps/authenticator.gcno
+./x86_64-unknown-linux-gnu/release/deps/base16.gcno
+./x86_64-unknown-linux-gnu/release/deps/base64.gcno
+./x86_64-unknown-linux-gnu/release/deps/base64.gcno
+./x86_64-unknown-linux-gnu/release/deps/bench_collections_gtest.gcno
+./x86_64-unknown-linux-gnu/release/deps/binary_space_partition.gcno
 ./x86_64-unknown-linux-gnu/release/deps/bincode.gcno
+./x86_64-unknown-linux-gnu/release/deps/bitflags.gcno
+./x86_64-unknown-linux-gnu/release/deps/bitreader.gcno
+./x86_64-unknown-linux-gnu/release/deps/bit_set.gcno
+./x86_64-unknown-linux-gnu/release/deps/bit_vec.gcno
+./x86_64-unknown-linux-gnu/release/deps/block_buffer.gcno
+./x86_64-unknown-linux-gnu/release/deps/block_padding.gcno
 ./x86_64-unknown-linux-gnu/release/deps/bookmark_sync.gcno
+./x86_64-unknown-linux-gnu/release/deps/byteorder.gcno
+./x86_64-unknown-linux-gnu/release/deps/bytes.gcno
+./x86_64-unknown-linux-gnu/release/deps/bytes.gcno
+./x86_64-unknown-linux-gnu/release/deps/byte_tools.gcno
+./x86_64-unknown-linux-gnu/release/deps/c2_chacha.gcno
 ./x86_64-unknown-linux-gnu/release/deps/cascade_bloom_filter.gcno
+./x86_64-unknown-linux-gnu/release/deps/cast.gcno
 ./x86_64-unknown-linux-gnu/release/deps/cert_storage.gcno
 ./x86_64-unknown-linux-gnu/release/deps/cert_storage.gcno
+./x86_64-unknown-linux-gnu/release/deps/cfg_if.gcno
+./x86_64-unknown-linux-gnu/release/deps/cfg_if.gcno
+./x86_64-unknown-linux-gnu/release/deps/chardetng_c.gcno
+./x86_64-unknown-linux-gnu/release/deps/chardetng.gcno
 ./x86_64-unknown-linux-gnu/release/deps/chrono.gcno
-./x86_64-unknown-linux-gnu/release/deps/cssparser.gcno
+./x86_64-unknown-linux-gnu/release/deps/codespan_reporting.gcno
+./x86_64-unknown-linux-gnu/release/deps/copyless.gcno
+./x86_64-unknown-linux-gnu/release/deps/cosec.gcno
+./x86_64-unknown-linux-gnu/release/deps/cose.gcno
+./x86_64-unknown-linux-gnu/release/deps/cranelift_entity.gcno
+./x86_64-unknown-linux-gnu/release/deps/crc32fast.gcno
+./x86_64-unknown-linux-gnu/release/deps/crossbeam_channel.gcno
+./x86_64-unknown-linux-gnu/release/deps/crossbeam_channel.gcno
+./x86_64-unknown-linux-gnu/release/deps/crossbeam_deque.gcno
+./x86_64-unknown-linux-gnu/release/deps/crossbeam_epoch.gcno
+./x86_64-unknown-linux-gnu/release/deps/crossbeam_queue.gcno
+./x86_64-unknown-linux-gnu/release/deps/crossbeam_utils.gcno
+./x86_64-unknown-linux-gnu/release/deps/crossbeam_utils.gcno
+./x86_64-unknown-linux-gnu/release/deps/crossbeam_utils.gcno
+./x86_64-unknown-linux-gnu/release/deps/cubeb_backend.gcno
+./x86_64-unknown-linux-gnu/release/deps/cubeb_core.gcno
+./x86_64-unknown-linux-gnu/release/deps/cubeb.gcno
+./x86_64-unknown-linux-gnu/release/deps/cubeb_pulse.gcno
+./x86_64-unknown-linux-gnu/release/deps/cubeb_sys.gcno
+./x86_64-unknown-linux-gnu/release/deps/data_encoding.gcno
+./x86_64-unknown-linux-gnu/release/deps/dbus.gcno
+./x86_64-unknown-linux-gnu/release/deps/debug_unreachable.gcno
+./x86_64-unknown-linux-gnu/release/deps/digest.gcno
+./x86_64-unknown-linux-gnu/release/deps/dirs.gcno
+./x86_64-unknown-linux-gnu/release/deps/dirs_sys.gcno
+./x86_64-unknown-linux-gnu/release/deps/dns_parser.gcno
+./x86_64-unknown-linux-gnu/release/deps/dogear.gcno
+./x86_64-unknown-linux-gnu/release/deps/dtoa.gcno
+./x86_64-unknown-linux-gnu/release/deps/dtoa_short.gcno
 ./x86_64-unknown-linux-gnu/release/deps/ece.gcno
+./x86_64-unknown-linux-gnu/release/deps/either.gcno
+./x86_64-unknown-linux-gnu/release/deps/encoding_c.gcno
+./x86_64-unknown-linux-gnu/release/deps/encoding_c_mem.gcno
+./x86_64-unknown-linux-gnu/release/deps/encoding_rs.gcno
+./x86_64-unknown-linux-gnu/release/deps/env_logger.gcno
+./x86_64-unknown-linux-gnu/release/deps/error_chain.gcno
 ./x86_64-unknown-linux-gnu/release/deps/error_support.gcno
 ./x86_64-unknown-linux-gnu/release/deps/etagere.gcno
 ./x86_64-unknown-linux-gnu/release/deps/euclid.gcno
-./x86_64-unknown-linux-gnu/release/deps/failure.gcno
+./x86_64-unknown-linux-gnu/release/deps/fake_simd.gcno
+./x86_64-unknown-linux-gnu/release/deps/fallible_collections.gcno
 ./x86_64-unknown-linux-gnu/release/deps/fallible.gcno
+./x86_64-unknown-linux-gnu/release/deps/fallible_iterator.gcno
+./x86_64-unknown-linux-gnu/release/deps/fallible_streaming_iterator.gcno
+./x86_64-unknown-linux-gnu/release/deps/ffi_support.gcno
 ./x86_64-unknown-linux-gnu/release/deps/firefox_accounts_bridge.gcno
+./x86_64-unknown-linux-gnu/release/deps/fixedbitset.gcno
+./x86_64-unknown-linux-gnu/release/deps/flate2.gcno
 ./x86_64-unknown-linux-gnu/release/deps/fluent_bundle.gcno
-./x86_64-unknown-linux-gnu/release/deps/fluent_ffi.gcno
 ./x86_64-unknown-linux-gnu/release/deps/fluent.gcno
 ./x86_64-unknown-linux-gnu/release/deps/fluent_langneg_ffi.gcno
+./x86_64-unknown-linux-gnu/release/deps/fluent_langneg.gcno
+./x86_64-unknown-linux-gnu/release/deps/fluent_pseudo.gcno
+./x86_64-unknown-linux-gnu/release/deps/fluent_syntax.gcno
+./x86_64-unknown-linux-gnu/release/deps/fnv.gcno
 ./x86_64-unknown-linux-gnu/release/deps/fog_control.gcno
 ./x86_64-unknown-linux-gnu/release/deps/fog_control.gcno
 ./x86_64-unknown-linux-gnu/release/deps/fog.gcno
 ./x86_64-unknown-linux-gnu/release/deps/fog.gcno
 ./x86_64-unknown-linux-gnu/release/deps/fog_gtest.gcno
-./x86_64-unknown-linux-gnu/release/deps/fxa_client.gcno
+./x86_64-unknown-linux-gnu/release/deps/freetype.gcno
+./x86_64-unknown-linux-gnu/release/deps/futures_channel.gcno
+./x86_64-unknown-linux-gnu/release/deps/futures_core.gcno
+./x86_64-unknown-linux-gnu/release/deps/futures_cpupool.gcno
+./x86_64-unknown-linux-gnu/release/deps/futures.gcno
+./x86_64-unknown-linux-gnu/release/deps/futures_task.gcno
+./x86_64-unknown-linux-gnu/release/deps/fxhash.gcno
 ./x86_64-unknown-linux-gnu/release/deps/gecko_fuzz_targets.gcno
 ./x86_64-unknown-linux-gnu/release/deps/gecko_logger.gcno
-./x86_64-unknown-linux-gnu/release/deps/geckoservo.gcno
+./x86_64-unknown-linux-gnu/release/deps/generic_array.gcno
+./x86_64-unknown-linux-gnu/release/deps/getrandom.gcno
 ./x86_64-unknown-linux-gnu/release/deps/gfx_backend_empty.gcno
 ./x86_64-unknown-linux-gnu/release/deps/gfx_backend_vulkan.gcno
 ./x86_64-unknown-linux-gnu/release/deps/gfx_hal.gcno
@@ -3629,97 +3699,205 @@
 ./x86_64-unknown-linux-gnu/release/deps/gkrust_gtest.gcno
 ./x86_64-unknown-linux-gnu/release/deps/gkrust_shared.gcno
 ./x86_64-unknown-linux-gnu/release/deps/gkrust_shared.gcno
-./x86_64-unknown-linux-gnu/release/deps/glean_core.gcno
-./x86_64-unknown-linux-gnu/release/deps/glean_core.gcno
-./x86_64-unknown-linux-gnu/release/deps/glean.gcno
-./x86_64-unknown-linux-gnu/release/deps/glean.gcno
+./x86_64-unknown-linux-gnu/release/deps/gkrust_utils.gcno
+./x86_64-unknown-linux-gnu/release/deps/gleam.gcno
 ./x86_64-unknown-linux-gnu/release/deps/goblin.gcno
 ./x86_64-unknown-linux-gnu/release/deps/golden_gate.gcno
+./x86_64-unknown-linux-gnu/release/deps/gpu_alloc.gcno
+./x86_64-unknown-linux-gnu/release/deps/gpu_alloc_types.gcno
+./x86_64-unknown-linux-gnu/release/deps/gpu_descriptor.gcno
+./x86_64-unknown-linux-gnu/release/deps/gpu_descriptor_types.gcno
+./x86_64-unknown-linux-gnu/release/deps/hashbrown.gcno
+./x86_64-unknown-linux-gnu/release/deps/hashglobe.gcno
+./x86_64-unknown-linux-gnu/release/deps/hashlink.gcno
 ./x86_64-unknown-linux-gnu/release/deps/hawk.gcno
+./x86_64-unknown-linux-gnu/release/deps/hex.gcno
+./x86_64-unknown-linux-gnu/release/deps/http.gcno
 ./x86_64-unknown-linux-gnu/release/deps/http_sfv.gcno
+./x86_64-unknown-linux-gnu/release/deps/humantime.gcno
+./x86_64-unknown-linux-gnu/release/deps/id_arena.gcno
+./x86_64-unknown-linux-gnu/release/deps/idna.gcno
+./x86_64-unknown-linux-gnu/release/deps/indexmap.gcno
+./x86_64-unknown-linux-gnu/release/deps/inplace_it.gcno
+./x86_64-unknown-linux-gnu/release/deps/instant.gcno
+./x86_64-unknown-linux-gnu/release/deps/interrupt_support.gcno
+./x86_64-unknown-linux-gnu/release/deps/intl_memoizer.gcno
+./x86_64-unknown-linux-gnu/release/deps/intl_pluralrules.gcno
+./x86_64-unknown-linux-gnu/release/deps/iovec.gcno
+./x86_64-unknown-linux-gnu/release/deps/itertools.gcno
+./x86_64-unknown-linux-gnu/release/deps/itoa.gcno
+./x86_64-unknown-linux-gnu/release/deps/jsrust.gcno
+./x86_64-unknown-linux-gnu/release/deps/jsrust_shared.gcno
 ./x86_64-unknown-linux-gnu/release/deps/jsrust_shared.gcno
-./x86_64-unknown-linux-gnu/release/deps/jwcrypto.gcno
 ./x86_64-unknown-linux-gnu/release/deps/kvstore.gcno
 ./x86_64-unknown-linux-gnu/release/deps/kvstore.gcno
 ./x86_64-unknown-linux-gnu/release/deps/l10nregistry_ffi_gtest.gcno
-./x86_64-unknown-linux-gnu/release/deps/lucet_module_wasmsbx.gcno
-./x86_64-unknown-linux-gnu/release/deps/lucet_runtime.gcno
-./x86_64-unknown-linux-gnu/release/deps/lucet_runtime_internals_wasmsbx.gcno
-./x86_64-unknown-linux-gnu/release/deps/lucet_wasi.gcno
-./x86_64-unknown-linux-gnu/release/deps/malloc_size_of.gcno
+./x86_64-unknown-linux-gnu/release/deps/lazy_static.gcno
+./x86_64-unknown-linux-gnu/release/deps/leb128.gcno
+./x86_64-unknown-linux-gnu/release/deps/libc.gcno
+./x86_64-unknown-linux-gnu/release/deps/libdbus_sys.gcno
+./x86_64-unknown-linux-gnu/release/deps/libloading.gcno
+./x86_64-unknown-linux-gnu/release/deps/libsqlite3_sys.gcno
+./x86_64-unknown-linux-gnu/release/deps/libudev.gcno
+./x86_64-unknown-linux-gnu/release/deps/libudev_sys.gcno
+./x86_64-unknown-linux-gnu/release/deps/lmdb.gcno
+./x86_64-unknown-linux-gnu/release/deps/lmdb.gcno
+./x86_64-unknown-linux-gnu/release/deps/lock_api.gcno
+./x86_64-unknown-linux-gnu/release/deps/log.gcno
+./x86_64-unknown-linux-gnu/release/deps/matches.gcno
+./x86_64-unknown-linux-gnu/release/deps/maybe_uninit.gcno
+./x86_64-unknown-linux-gnu/release/deps/mdns_service.gcno
+./x86_64-unknown-linux-gnu/release/deps/memchr.gcno
+./x86_64-unknown-linux-gnu/release/deps/memmap.gcno
+./x86_64-unknown-linux-gnu/release/deps/memoffset.gcno
+./x86_64-unknown-linux-gnu/release/deps/miniz_oxide.gcno
+./x86_64-unknown-linux-gnu/release/deps/mio.gcno
+./x86_64-unknown-linux-gnu/release/deps/mio_uds.gcno
+./x86_64-unknown-linux-gnu/release/deps/moz_cbor.gcno
 ./x86_64-unknown-linux-gnu/release/deps/mozglue_static.gcno
+./x86_64-unknown-linux-gnu/release/deps/mozilla_central_workspace_hack.gcno
 ./x86_64-unknown-linux-gnu/release/deps/moz_task.gcno
 ./x86_64-unknown-linux-gnu/release/deps/moz_task_gtest.gcno
 ./x86_64-unknown-linux-gnu/release/deps/mozurl.gcno
+./x86_64-unknown-linux-gnu/release/deps/mp4parse_capi.gcno
+./x86_64-unknown-linux-gnu/release/deps/mp4parse.gcno
+./x86_64-unknown-linux-gnu/release/deps/mp4parse_gtest.gcno
+./x86_64-unknown-linux-gnu/release/deps/murmurhash3.gcno
 ./x86_64-unknown-linux-gnu/release/deps/naga.gcno
 ./x86_64-unknown-linux-gnu/release/deps/neqo_common.gcno
-./x86_64-unknown-linux-gnu/release/deps/neqo_crypto.gcno
 ./x86_64-unknown-linux-gnu/release/deps/neqo_glue.gcno
-./x86_64-unknown-linux-gnu/release/deps/neqo_http3.gcno
-./x86_64-unknown-linux-gnu/release/deps/neqo_qpack.gcno
-./x86_64-unknown-linux-gnu/release/deps/neqo_transport.gcno
-./x86_64-unknown-linux-gnu/release/deps/nss.gcno
+./x86_64-unknown-linux-gnu/release/deps/net2.gcno
+./x86_64-unknown-linux-gnu/release/deps/nix.gcno
+./x86_64-unknown-linux-gnu/release/deps/nodrop.gcno
+./x86_64-unknown-linux-gnu/release/deps/nss_sys.gcno
+./x86_64-unknown-linux-gnu/release/deps/nsstring.gcno
+./x86_64-unknown-linux-gnu/release/deps/nsstring_gtest.gcno
+./x86_64-unknown-linux-gnu/release/deps/num_cpus.gcno
+./x86_64-unknown-linux-gnu/release/deps/num_integer.gcno
+./x86_64-unknown-linux-gnu/release/deps/num_traits.gcno
 ./x86_64-unknown-linux-gnu/release/deps/object.gcno
-./x86_64-unknown-linux-gnu/release/deps/ouroboros.gcno
+./x86_64-unknown-linux-gnu/release/deps/once_cell.gcno
+./x86_64-unknown-linux-gnu/release/deps/opaque_debug.gcno
+./x86_64-unknown-linux-gnu/release/deps/ordered_float.gcno
+./x86_64-unknown-linux-gnu/release/deps/owning_ref.gcno
+./x86_64-unknown-linux-gnu/release/deps/packed_simd.gcno
+./x86_64-unknown-linux-gnu/release/deps/parity_wasm.gcno
 ./x86_64-unknown-linux-gnu/release/deps/parking_lot_core.gcno
 ./x86_64-unknown-linux-gnu/release/deps/parking_lot.gcno
-./x86_64-unknown-linux-gnu/release/deps/paste.gcno
-./x86_64-unknown-linux-gnu/release/deps/peek_poke.gcno
-./x86_64-unknown-linux-gnu/release/deps/phf.gcno
+./x86_64-unknown-linux-gnu/release/deps/percent_encoding.gcno
+./x86_64-unknown-linux-gnu/release/deps/petgraph.gcno
+./x86_64-unknown-linux-gnu/release/deps/phf_shared.gcno
+./x86_64-unknown-linux-gnu/release/deps/plain.gcno
 ./x86_64-unknown-linux-gnu/release/deps/plane_split.gcno
+./x86_64-unknown-linux-gnu/release/deps/ppv_lite86.gcno
+./x86_64-unknown-linux-gnu/release/deps/precomputed_hash.gcno
+./x86_64-unknown-linux-gnu/release/deps/prefs_parser.gcno
 ./x86_64-unknown-linux-gnu/release/deps/processtools.gcno
 ./x86_64-unknown-linux-gnu/release/deps/profiler_helper.gcno
-./x86_64-unknown-linux-gnu/release/deps/prost.gcno
-./x86_64-unknown-linux-gnu/release/deps/qlog.gcno
+./x86_64-unknown-linux-gnu/release/deps/profiling.gcno
+./x86_64-unknown-linux-gnu/release/deps/pulse_ffi.gcno
+./x86_64-unknown-linux-gnu/release/deps/pulse.gcno
+./x86_64-unknown-linux-gnu/release/deps/quick_error.gcno
+./x86_64-unknown-linux-gnu/release/deps/rand_chacha.gcno
+./x86_64-unknown-linux-gnu/release/deps/rand_core.gcno
+./x86_64-unknown-linux-gnu/release/deps/rand.gcno
+./x86_64-unknown-linux-gnu/release/deps/rand_pcg.gcno
 ./x86_64-unknown-linux-gnu/release/deps/rand_rccrypto.gcno
+./x86_64-unknown-linux-gnu/release/deps/raw_cpuid.gcno
+./x86_64-unknown-linux-gnu/release/deps/raw_window_handle.gcno
+./x86_64-unknown-linux-gnu/release/deps/rayon_core.gcno
+./x86_64-unknown-linux-gnu/release/deps/rayon.gcno
 ./x86_64-unknown-linux-gnu/release/deps/rc_crypto.gcno
+./x86_64-unknown-linux-gnu/release/deps/regex.gcno
+./x86_64-unknown-linux-gnu/release/deps/regex_syntax.gcno
 ./x86_64-unknown-linux-gnu/release/deps/remote.gcno
-./x86_64-unknown-linux-gnu/release/deps/rental.gcno
-./x86_64-unknown-linux-gnu/release/deps/rkv.gcno
-./x86_64-unknown-linux-gnu/release/deps/rkv.gcno
-./x86_64-unknown-linux-gnu/release/deps/rlbox_lucet_sandbox.gcno
+./x86_64-unknown-linux-gnu/release/deps/remove_dir_all.gcno
+./x86_64-unknown-linux-gnu/release/deps/ringbuf.gcno
 ./x86_64-unknown-linux-gnu/release/deps/ron.gcno
-./x86_64-unknown-linux-gnu/release/deps/rsdparsa_capi.gcno
+./x86_64-unknown-linux-gnu/release/deps/runloop.gcno
 ./x86_64-unknown-linux-gnu/release/deps/rusqlite.gcno
+./x86_64-unknown-linux-gnu/release/deps/rust_cascade.gcno
+./x86_64-unknown-linux-gnu/release/deps/rustc_demangle.gcno
 ./x86_64-unknown-linux-gnu/release/deps/rust_decimal.gcno
-./x86_64-unknown-linux-gnu/release/deps/scroll.gcno
-./x86_64-unknown-linux-gnu/release/deps/selectors.gcno
+./x86_64-unknown-linux-gnu/release/deps/ryu.gcno
+./x86_64-unknown-linux-gnu/release/deps/scopeguard.gcno
+./x86_64-unknown-linux-gnu/release/deps/semver.gcno
+./x86_64-unknown-linux-gnu/release/deps/semver_parser.gcno
 ./x86_64-unknown-linux-gnu/release/deps/serde_bytes.gcno
-./x86_64-unknown-linux-gnu/release/deps/serde.gcno
 ./x86_64-unknown-linux-gnu/release/deps/serde_json.gcno
-./x86_64-unknown-linux-gnu/release/deps/serde_with.gcno
 ./x86_64-unknown-linux-gnu/release/deps/sfv.gcno
+./x86_64-unknown-linux-gnu/release/deps/sha2.gcno
+./x86_64-unknown-linux-gnu/release/deps/shift_or_euc_c.gcno
+./x86_64-unknown-linux-gnu/release/deps/shift_or_euc.gcno
+./x86_64-unknown-linux-gnu/release/deps/siphasher.gcno
+./x86_64-unknown-linux-gnu/release/deps/slab.gcno
+./x86_64-unknown-linux-gnu/release/deps/smallbitvec.gcno
 ./x86_64-unknown-linux-gnu/release/deps/smallvec.gcno
+./x86_64-unknown-linux-gnu/release/deps/socket2.gcno
+./x86_64-unknown-linux-gnu/release/deps/spirv_headers.gcno
 ./x86_64-unknown-linux-gnu/release/deps/sql_support.gcno
+./x86_64-unknown-linux-gnu/release/deps/stable_deref_trait.gcno
+./x86_64-unknown-linux-gnu/release/deps/static_assertions.gcno
 ./x86_64-unknown-linux-gnu/release/deps/storage.gcno
 ./x86_64-unknown-linux-gnu/release/deps/storage_variant.gcno
-./x86_64-unknown-linux-gnu/release/deps/style.gcno
-./x86_64-unknown-linux-gnu/release/deps/style_traits.gcno
-./x86_64-unknown-linux-gnu/release/deps/swgl.gcno
-./x86_64-unknown-linux-gnu/release/deps/sync15.gcno
+./x86_64-unknown-linux-gnu/release/deps/svg_fmt.gcno
 ./x86_64-unknown-linux-gnu/release/deps/sync15_traits.gcno
 ./x86_64-unknown-linux-gnu/release/deps/sync_guid.gcno
-./x86_64-unknown-linux-gnu/release/deps/thiserror.gcno
-./x86_64-unknown-linux-gnu/release/deps/to_shmem.gcno
+./x86_64-unknown-linux-gnu/release/deps/target_lexicon.gcno
+./x86_64-unknown-linux-gnu/release/deps/tempfile.gcno
+./x86_64-unknown-linux-gnu/release/deps/termcolor.gcno
+./x86_64-unknown-linux-gnu/release/deps/thin_slice.gcno
+./x86_64-unknown-linux-gnu/release/deps/thin_vec.gcno
+./x86_64-unknown-linux-gnu/release/deps/threadbound.gcno
+./x86_64-unknown-linux-gnu/release/deps/thread_local.gcno
+./x86_64-unknown-linux-gnu/release/deps/time.gcno
+./x86_64-unknown-linux-gnu/release/deps/tinystr.gcno
+./x86_64-unknown-linux-gnu/release/deps/tokio_codec.gcno
+./x86_64-unknown-linux-gnu/release/deps/tokio_current_thread.gcno
+./x86_64-unknown-linux-gnu/release/deps/tokio_executor.gcno
+./x86_64-unknown-linux-gnu/release/deps/tokio_fs.gcno
+./x86_64-unknown-linux-gnu/release/deps/tokio.gcno
+./x86_64-unknown-linux-gnu/release/deps/tokio_io.gcno
+./x86_64-unknown-linux-gnu/release/deps/tokio_reactor.gcno
+./x86_64-unknown-linux-gnu/release/deps/tokio_tcp.gcno
+./x86_64-unknown-linux-gnu/release/deps/tokio_threadpool.gcno
+./x86_64-unknown-linux-gnu/release/deps/tokio_timer.gcno
+./x86_64-unknown-linux-gnu/release/deps/tokio_udp.gcno
+./x86_64-unknown-linux-gnu/release/deps/tokio_uds.gcno
+./x86_64-unknown-linux-gnu/release/deps/tracy_rs.gcno
+./x86_64-unknown-linux-gnu/release/deps/type_map.gcno
+./x86_64-unknown-linux-gnu/release/deps/typenum.gcno
 ./x86_64-unknown-linux-gnu/release/deps/uluru.gcno
 ./x86_64-unknown-linux-gnu/release/deps/unic_langid_ffi.gcno
+./x86_64-unknown-linux-gnu/release/deps/unic_langid.gcno
+./x86_64-unknown-linux-gnu/release/deps/unic_langid_impl.gcno
+./x86_64-unknown-linux-gnu/release/deps/unicode_bidi.gcno
+./x86_64-unknown-linux-gnu/release/deps/unicode_normalization.gcno
+./x86_64-unknown-linux-gnu/release/deps/unicode_segmentation.gcno
+./x86_64-unknown-linux-gnu/release/deps/unicode_width.gcno
+./x86_64-unknown-linux-gnu/release/deps/unreachable.gcno
 ./x86_64-unknown-linux-gnu/release/deps/url.gcno
-./x86_64-unknown-linux-gnu/release/deps/viaduct.gcno
+./x86_64-unknown-linux-gnu/release/deps/uuid.gcno
+./x86_64-unknown-linux-gnu/release/deps/void.gcno
+./x86_64-unknown-linux-gnu/release/deps/wasmparser.gcno
+./x86_64-unknown-linux-gnu/release/deps/wast.gcno
+./x86_64-unknown-linux-gnu/release/deps/wat.gcno
 ./x86_64-unknown-linux-gnu/release/deps/webext_storage_bridge.gcno
-./x86_64-unknown-linux-gnu/release/deps/webext_storage.gcno
-./x86_64-unknown-linux-gnu/release/deps/webrender_api.gcno
 ./x86_64-unknown-linux-gnu/release/deps/webrender_bindings.gcno
 ./x86_64-unknown-linux-gnu/release/deps/webrender_build.gcno
 ./x86_64-unknown-linux-gnu/release/deps/webrender.gcno
 ./x86_64-unknown-linux-gnu/release/deps/webrtc_sdp.gcno
-./x86_64-unknown-linux-gnu/release/deps/wgpu_bindings.gcno
 ./x86_64-unknown-linux-gnu/release/deps/wgpu_core.gcno
 ./x86_64-unknown-linux-gnu/release/deps/wgpu_types.gcno
+./x86_64-unknown-linux-gnu/release/deps/whatsys.gcno
 ./x86_64-unknown-linux-gnu/release/deps/wr_malloc_size_of.gcno
+./x86_64-unknown-linux-gnu/release/deps/xfailure.gcno
+./x86_64-unknown-linux-gnu/release/deps/xmldecl.gcno
 ./x86_64-unknown-linux-gnu/release/deps/xpcom.gcno
 ./x86_64-unknown-linux-gnu/release/deps/xpcom_gtest.gcno
 ./x86_64-unknown-linux-gnu/release/deps/xulstore.gcno
 ./x86_64-unknown-linux-gnu/release/deps/xulstore.gcno
+./x86_64-unknown-linux-gnu/release/deps/zeitstempel.gcno

Marco, can you look into this or find someone to work with Jason and Jesse? This is causing issues in our fuzzing coverage reports, in particular the /servo directory is missing from some coverage reports, causing major fluctuations in our final report.

Flags: needinfo?(mcastelluccio)

Me and Calixte talked about this. The intermittency of it is weird, and we came to the conclusion that it might be something related to caches or similar.
We then thought a likely culprit could be sccache, and by searching on Bugzilla I found bug 1334402 and remembered that we then re-enabled sccache for ccov builds in bug 1334940 after making sccache cache gcno files too.

The patch which added support for gcno caching in sccache (https://github.com/mozilla/sccache/pull/260) only added it to C/C++ compilers though, as at the time Rust coverage didn't exist yet.

So, at this point I'm confident that this might actually be it, and that we just need to add support for gcno caching to sccache for Rust compilation. I guess we need to something similar to https://github.com/mozilla/sccache/pull/260/files in https://github.com/mozilla/sccache/blob/master/src/compiler/rust.rs.

Glandium, is this something you could help with?

Flags: needinfo?(mcastelluccio) → needinfo?(mh+mozilla)

Note: probably the underlying cause of bug 1712327 is the same as this one.

Blocks: 1720345
Blocks: domino
Blocks: 1712327

There's a PR to fix https://github.com/mozilla/sccache/issues/1021, https://github.com/mozilla/sccache/pull/1052, but Glandium won't be able to review it shortly.

Let's try just disabling sccache for now.

Flags: needinfo?(mh+mozilla)
Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
Attachment #9247041 - Attachment description: Bug 1712751 - Disable sccache on ccov builds. r=glandium → Bug 1712751 - Disable sccache on ccov builds. r=#firefox-build-system-reviewers
See Also: → 1737006
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/04330041da3f
Disable sccache on ccov builds. r=firefox-build-system-reviewers,andi DONTBUILD
Status: ASSIGNED → RESOLVED
Closed: 1 month ago
Resolution: --- → FIXED
Target Milestone: --- → 95 Branch

Could you verify this is fixed?

Flags: needinfo?(jkratzer)
Regressions: 1737285
No longer blocks: domino
Flags: needinfo?(jkratzer)
Flags: needinfo?(jkratzer)

Verified that this appears to be fixed in m-c rev 33d6ea7b0aad.

Flags: needinfo?(jkratzer)
Status: RESOLVED → VERIFIED
Regressions: 1742652
You need to log in before you can comment on or make changes to this bug.