1288415 - Investigate BBB double scheduling

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

We have seen this twice, in 48.0b6 and 48.0b9, when BBB scheduled a task twice, what caused some pain. It would be great to figure out what is going on here...

Comment 1

[Rail Aliiev [:rail]]

Attachment: Add some logging

Comment 2

[Rail Aliiev [:rail]]

Attachment: PR

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 8782632 [details] [review]
PR

With multiple instances running this may not help.

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 8782632 [details] [review]
PR

The patch looks simpler with `git diff -b`

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8782632 [details] [review]
PR

r+ed in the PR

remote:   https://hg.mozilla.org/build/puppet/rev/facb4d3bd6c7
remote:   https://hg.mozilla.org/build/puppet/rev/2992f71e8556

Comment 6

[Rail Aliiev [:rail]]

The patch worked fine yesterday in Firefox 49.0b8. I'll keep an eye on this for another couple of weeks to be sure.

Comment 7

[Rail Aliiev [:rail]]

Yesterday BBB ignored 51 dupes! The release builds were flawless because of that. \o/

Copying here for posterity.

Sep 05 16:21:48 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task eU-DAZT5Som01-ptmPIfVw run 0 brid 123344523: ignoring duplicated message
Sep 05 16:21:52 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task bqCCOuFkQDGkkN9nT7MpYg run 0 brid 123344531: ignoring duplicated message
Sep 05 16:21:54 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task G8Ma2ouTTJ-D-RjOYw3QOw run 0 brid 123344548: ignoring duplicated message
Sep 05 16:21:58 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task BcEAIBaNQCyiO6O3JUw7Hw run 0 brid 123344544: ignoring duplicated message
Sep 05 16:22:00 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task qfelFPwIRKqB36Kt0G2QAw run 0 brid 123344528: ignoring duplicated message
Sep 05 16:22:04 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task TEDvas8vRh22IlkIbxyzrg run 0 brid 123344526: ignoring duplicated message
Sep 05 16:22:06 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task 8OkNYYY3TySKQ4x_shzfHA run 0 brid 123344536: ignoring duplicated message
Sep 05 16:22:06 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task uHlu26wjRY6fYsoEmY--Jg run 0 brid 123344527: ignoring duplicated message
Sep 05 16:22:08 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task QxmQ9PZFTbSwfg0jKtijhw run 0 brid 123344543: ignoring duplicated message
Sep 05 16:22:12 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task 6VhzGdmPRsOR_vnkUTTBpw run 0 brid 123344561: ignoring duplicated message
Sep 05 16:22:14 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task 5r2kRbBJQLaKwQ7TpI9i3A run 0 brid 123344545: ignoring duplicated message
Sep 05 16:22:18 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task KsB8G9FHR0Gdv0X2otMe9A run 0 brid 123344566: ignoring duplicated message
Sep 05 16:22:20 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task Y92IPKU8T--9qgiRIfOU-Q run 0 brid 123344570: ignoring duplicated message
Sep 05 16:22:20 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task _GHZY5OgQBOvhwr1KkwVMw run 0 brid 123344556: ignoring duplicated message
Sep 05 16:22:22 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task nFE7H_1rTk2dsiFH4iq0Qg run 0 brid 123344569: ignoring duplicated message
Sep 05 16:22:26 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task MnLBiVReT-a9Ev3YqMWNNA run 0 brid 123344571: ignoring duplicated message
Sep 05 16:22:27 buildbot-master82.bb.releng.scl3.mozilla.com python: buildbot_bridge_tclistener bbb.services - task iSsj_iN3T6WPOtZbqG-pHA run 0 brid 123344572: ignoring duplicated message
Sep 05 16:22:31 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task YnYP43fgTYOo__JDyTGUQw run 0 brid 123344522: ignoring duplicated message
Sep 05 16:22:32 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task Yt3br839RCe_wEXuw4DHeg run 0 brid 123344542: ignoring duplicated message
Sep 05 16:22:36 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task N0nkZvxDRhGKOYPFnNdj_g run 0 brid 123344533: ignoring duplicated message
Sep 05 16:22:37 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task 59n_ggxoSGi-in0INpyX-w run 0 brid 123344530: ignoring duplicated message
Sep 05 16:22:38 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task wBQxu7HMQs6h9wrJHFiVMA run 0 brid 123344535: ignoring duplicated message
Sep 05 16:22:39 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task aQ7--urkQiCyI9dzj-ZQtA run 0 brid 123344537: ignoring duplicated message
Sep 05 16:22:40 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task JzkvgW3KSA-XM9YfP4PC-Q run 0 brid 123344555: ignoring duplicated message
Sep 05 16:22:40 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task atm1UwmgQnO1kaOJJyekbA run 0 brid 123344525: ignoring duplicated message
Sep 05 16:22:40 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task DBJKYBDGTxOySMiGuUxqrw run 0 brid 123344540: ignoring duplicated message
Sep 05 16:22:41 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task KiqihTUjS8KwlCBJbvEKZA run 0 brid 123344558: ignoring duplicated message
Sep 05 16:22:42 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task z6FWDHyVTjG0R9QdwwcjrA run 0 brid 123344529: ignoring duplicated message
Sep 05 16:22:42 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task -dyO1J-JSkmxvpKsgfR7-A run 0 brid 123344541: ignoring duplicated message
Sep 05 16:22:43 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task uWixZCCqSwGfcnqQouQNPA run 0 brid 123344524: ignoring duplicated message
Sep 05 16:22:43 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task zxDVly0VRRCSuq6IYa3v4Q run 0 brid 123344563: ignoring duplicated message
Sep 05 16:22:44 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task KzJKrRd6ShuESxTVYtQS9g run 0 brid 123344568: ignoring duplicated message
Sep 05 16:22:44 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task BjxP_PnPR6aBp2zzZa_KgA run 0 brid 123344546: ignoring duplicated message
Sep 05 16:22:45 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task XBoTqlfVQ6m1uIVnRx33IA run 0 brid 123344567: ignoring duplicated message
Sep 05 16:22:46 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task DMUnAekYT8K15Rjebzc_bA run 0 brid 123344550: ignoring duplicated message
Sep 05 16:22:46 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task w79-83_2RriGQgeCSjx1tg run 0 brid 123344554: ignoring duplicated message
Sep 05 16:22:46 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task Xj3k6gncQ1iW_FKDN_KDGA run 0 brid 123344549: ignoring duplicated message
Sep 05 16:22:47 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task iI0eguwVTHmrlwBhOygR5Q run 0 brid 123344534: ignoring duplicated message
Sep 05 16:22:47 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task 3XOGWfwWRvSx3d5d6ceChw run 0 brid 123344547: ignoring duplicated message
Sep 05 16:22:48 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task KbXfWN2WSO-B0Bn9wO2bbg run 0 brid 123344553: ignoring duplicated message
Sep 05 16:22:48 buildbot-master72.bb.releng.usw2.mozilla.com python: buildbot_bridge_tclistener bbb.services - task 8Z679qm0QxWleEZN_ejOdw run 0 brid 123344573: ignoring duplicated message
Sep 05 16:22:49 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task zcll8acUS26d3DdAlVdmEw run 0 brid 123344532: ignoring duplicated message
Sep 05 16:22:51 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task i7wx0XmmRD2fmc925MsEnw run 0 brid 123344557: ignoring duplicated message
Sep 05 16:22:52 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task vTHnSYFURTm7BtyCa0z-AQ run 0 brid 123344559: ignoring duplicated message
Sep 05 16:22:53 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task 6fxsd-epSfSXYzq4hh-n_A run 0 brid 123344538: ignoring duplicated message
Sep 05 16:22:55 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task VlvSr15bRIG-zVl82GyP7g run 0 brid 123344562: ignoring duplicated message
Sep 05 16:22:56 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task X9G_-6CWTyqOsQRrpwNbvQ run 0 brid 123344565: ignoring duplicated message
Sep 05 16:22:57 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task S4JwIv_6SbaEPAu0HnsUqA run 0 brid 123344564: ignoring duplicated message
Sep 05 16:22:59 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task Urwq4498TyKiku8Dbn7NKQ run 0 brid 123344574: ignoring duplicated message
Sep 05 16:23:00 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task pOn2-hLCRKuBXA5OgdrnAQ run 0 brid 123344551: ignoring duplicated message
Sep 05 16:23:01 buildbot-master70.bb.releng.use1.mozilla.com python: buildbot_bridge_tclistener bbb.services - task gY8YXUdIRTWEmuQrkhj6HQ run 0 brid 123344575: ignoring duplicated message