812709 - Not all B2G emulator logs are getting uploaded/processed

Status: RESOLVED INVALID

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

Some logs for B2G emulator tests are going missing.  The results of the tests (red/green/orange) show up in the self-server build api, but they don't appear in the build folder, and thus they don't show up in TBPL.

Case in point:

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&noignore=1&rev=a94288026ea5 shows only 1 emulator test (R4), but they are all shown here:

https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-inbound/rev/a94288026ea5

Comment 1

[Phil Ringnalda (:philor)]

Welcome to my hellish world, the world of job coalescing.

If slaves had become available to run tests on a94288026ea5 before any builds finished on later pushes and scheduled tests, then they would have been run there, and tbpl would show them there. But because the b2g emulator tests run on the linux32 test slaves, which were already our worst or second-worst platform, there was only that one slave available, the one that did the R4.

Once the next push finished and scheduled tests, then any that were still pending on the first push were instead run against the next push's build (though only two ran there, the rest didn't get caught until something like five pushes later), but they are still shown by self-serve against the first push, so that if there winds up being some question which of the two were at fault, I can retrigger them on the first push, at which point they'll actually run against it, and I'll be able to tell whether to back out the first push or the second push for breaking them.

(Of course that's oversimplified, and usually it'll be more like "well, linux64 coalesced across six pushes, linux32 coalesced across nine pushes, but it fails five out of seven times on linux32 and only three out of eleven times on linux64, and we still have 400 pending linux32 jobs but only 95 pending linux64 jobs, so I'll retrigger linux64 on these pushes, and linux32 on these, and not actually go to bed tonight.")

So, no logs are missing, because only one run happened against that push, self-serve is lying to you, but it's lying for a good reason, to give you a place to retrigger if you need to.

Comment 2

[Jonathan Griffin (:jgriffin)]

Ah, I see, thanks for the explanation.