Aaron, looks like the conditions in the [lambda function](https://searchfox.org/mozilla-central/source/testing/talos/talos/xtalos/etlparser.py#381-385) cover only the events with no `DiskWriteCount` and `DiskWriteBytes`: ``` [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\favicons.sqlite', '6732 (main)', 'startup') {'DiskReadBytes': 196808, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 8} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('Z:\\task_1586329312\\build\\application\\firefox\\defaults\\pref\\channel-prefs.js', '6732 (main)', 'startup') {'DiskReadBytes': 866, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\permissions.sqlite', '6732 (main)', 'startup') {'DiskReadBytes': 4392, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 12} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('Z:\\task_1586329312\\AppData\\Roaming\\Mozilla\\firefox\\Crash Reports\\InstallTime20200408054848', '6732 (main)', 'startup') {'DiskReadBytes': 20, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\key4.db', '6732 (main)', 'startup') {'DiskReadBytes': 196936, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 16} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\favicons.sqlite-shm', '6732 (main)', 'startup') {'DiskReadBytes': 65536, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('Z:\\task_1586329312\\build\\application\\firefox\\dependentlibs.list', '6732 (main)', 'startup') {'DiskReadBytes': 16384, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 4} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\prefs.js', '6732 (main)', 'startup') {'DiskReadBytes': 28970, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\cookies.sqlite', '6732 (main)', 'startup') {'DiskReadBytes': 131272, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 6} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Windows\\System32\\InputHost.dll', '6732 (main)', 'startup') {'DiskReadBytes': 32768, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\user.js', '6732 (main)', 'startup') {'DiskReadBytes': 18288, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Windows\\System32\\spool\\drivers\\color\\sRGB Color Space Profile.icm', '6732 (main)', 'startup') {'DiskReadBytes': 8192, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\cert9.db', '6732 (main)', 'startup') {'DiskReadBytes': 131336, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 10} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Windows\\System32\\setupapi.dll', '6732 (main)', 'startup') {'DiskReadBytes': 88064, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 8} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\xulstore\\data.mdb', '6732 (main)', 'startup') {'DiskReadBytes': 1216, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 8} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\places.sqlite-shm', '6732 (main)', 'startup') {'DiskReadBytes': 65536, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:16.992Z] 07:30:16 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\places.sqlite', '6732 (main)', 'startup') {'DiskReadBytes': 262344, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 10} [task 2020-04-08T07:30:16.992Z] 07:30:16 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\pkcs11.txt', '6732 (main)', 'startup') {'DiskReadBytes': 16384, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 4} [task 2020-04-08T07:30:16.993Z] 07:30:16 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\cookies.sqlite-shm', '6732 (main)', 'startup') {'DiskReadBytes': 65536, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:16.993Z] 07:30:16 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\compatibility.ini', '6732 (main)', 'startup') {'DiskReadBytes': 446, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} ``` We probably need to revisit that logic in order to get rid of the zero-values.
Bug 1614805 Comment 16 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Aaron, looks like the conditions in the [lambda function](https://searchfox.org/mozilla-central/source/testing/talos/talos/xtalos/etlparser.py#381-385) cover only the events with no `DiskWriteCount` and `DiskWriteBytes`: ``` [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\favicons.sqlite', '6732 (main)', 'startup') {'DiskReadBytes': 196808, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 8} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('Z:\\task_1586329312\\build\\application\\firefox\\defaults\\pref\\channel-prefs.js', '6732 (main)', 'startup') {'DiskReadBytes': 866, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\permissions.sqlite', '6732 (main)', 'startup') {'DiskReadBytes': 4392, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 12} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('Z:\\task_1586329312\\AppData\\Roaming\\Mozilla\\firefox\\Crash Reports\\InstallTime20200408054848', '6732 (main)', 'startup') {'DiskReadBytes': 20, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\key4.db', '6732 (main)', 'startup') {'DiskReadBytes': 196936, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 16} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\favicons.sqlite-shm', '6732 (main)', 'startup') {'DiskReadBytes': 65536, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('Z:\\task_1586329312\\build\\application\\firefox\\dependentlibs.list', '6732 (main)', 'startup') {'DiskReadBytes': 16384, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 4} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\prefs.js', '6732 (main)', 'startup') {'DiskReadBytes': 28970, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.003Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\cookies.sqlite', '6732 (main)', 'startup') {'DiskReadBytes': 131272, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 6} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Windows\\System32\\InputHost.dll', '6732 (main)', 'startup') {'DiskReadBytes': 32768, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\user.js', '6732 (main)', 'startup') {'DiskReadBytes': 18288, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Windows\\System32\\spool\\drivers\\color\\sRGB Color Space Profile.icm', '6732 (main)', 'startup') {'DiskReadBytes': 8192, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\cert9.db', '6732 (main)', 'startup') {'DiskReadBytes': 131336, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 10} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Windows\\System32\\setupapi.dll', '6732 (main)', 'startup') {'DiskReadBytes': 88064, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 8} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\xulstore\\data.mdb', '6732 (main)', 'startup') {'DiskReadBytes': 1216, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 8} [task 2020-04-08T07:30:15.004Z] 07:30:15 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\places.sqlite-shm', '6732 (main)', 'startup') {'DiskReadBytes': 65536, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:16.992Z] 07:30:16 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\places.sqlite', '6732 (main)', 'startup') {'DiskReadBytes': 262344, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 10} [task 2020-04-08T07:30:16.992Z] 07:30:16 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\pkcs11.txt', '6732 (main)', 'startup') {'DiskReadBytes': 16384, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 4} [task 2020-04-08T07:30:16.993Z] 07:30:16 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\cookies.sqlite-shm', '6732 (main)', 'startup') {'DiskReadBytes': 65536, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} [task 2020-04-08T07:30:16.993Z] 07:30:16 INFO - ('C:\\Users\\task_1586329312\\AppData\\Local\\Temp\\tmpmpvoz_\\profile\\compatibility.ini', '6732 (main)', 'startup') {'DiskReadBytes': 446, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 2} ``` We probably need to revisit that logic in order to get rid of the zero-values. There's this [lambda function](https://searchfox.org/mozilla-central/source/testing/talos/talos/xtalos/etlparser.py#381-385) that filters those values and looks like this: ``` filekeys = filter(lambda x: (all_stages or x[2] == stages[0]) and (all_threads or x[1].endswith("(main)")) and (all_stages and x[2] != stages[0] or not checkWhitelist(x[0], whitelist)), files.iterkeys()) ``` If we take `favicons.sqlite` and look at the unfiltered values from `etl_output.csv` it looks like this: ``` ('C:\\Users\\task_1586182132\\AppData\\Local\\Temp\\tmpajjk25\\profile\\favicons.sqlite', '9912', 'all'): {'DiskReadBytes': 458952, 'DiskWriteCount': 94, 'DiskWriteBytes': 4915200, 'DiskReadCount': 16}, ('C:\\Users\\task_1586182132\\AppData\\Local\\Temp\\tmpajjk25\\profile\\favicons.sqlite', '11180 (main)', 'all'): {'DiskReadBytes': 196808, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 8}, ('C:\\Users\\task_1586182132\\AppData\\Local\\Temp\\tmpajjk25\\profile\\favicons.sqlite', '9912', 'normal'): {'DiskReadBytes': 458952, 'DiskWriteCount': 94, 'DiskWriteBytes': 4915200, 'DiskReadCount': 16}, ('C:\\Users\\task_1586182132\\AppData\\Local\\Temp\\tmpajjk25\\profile\\favicons.sqlite', '11180 (main)', 'startup'): {'DiskReadBytes': 196808, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 8}, ('C:\\Users\\task_1586182132\\AppData\\Local\\Temp\\tmpajjk25\\profile\\favicons.sqlite', '9912', 'startup'): {'DiskReadBytes': 262344, 'DiskWriteCount': 0, 'DiskWriteBytes': 0, 'DiskReadCount': 10}, ``` `DiskWriteCount` and `DiskWriteBytes` are greater than 0 for `all` and `normal` stages. Looking at the code for debug mode: ``` if debug: # in debug, we want stages = [startup+normal] and all threads, not just (main) # we will use this data to upload fileIO info to blobber only for debug mode outputData = filter(lambda x: (all_stages or x[2] in [stages[0], stages[1]]) and (all_stages and x[2] not in [stages[0], stages[1]] or not checkWhitelist(x[0], whitelist)), files.iterkeys()) ``` we can see that it includes `stages[1]` in `x[2]`, which is exactly the `normal` stage. I don't have thorough enough knowledge about this code to tell if this is the right fix, but is the closest fix I found so far (inclusing ` stages[1]` in the expression against `x[2]`). Thoughts?