1480025 - Upload screenshots as TaskCluster artifacts on failed MozMill tests

Status: RESOLVED FIXED

(Thunderbird :: Testing Infrastructure, defect)

Description

[Geoff Lankow (:darktrojan)]

I wanted to see what was happening on the test machines to figure out why they were failing. It turns out that MozMill already makes screenshots of failing tests, so all I had to do was find them and get them into the right place to be uploaded.

I think this should be turned off by default as there isn't much point if nobody's using the results. You can see it working on this Try run (at least until it expires): https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=b54ed9ef250e78fa85770f908d7c4278242b8e5f

Comment 1

[Geoff Lankow (:darktrojan)]

Attachment: 1480025-mozmill-screenshots-1.diff

Comment 2

[Jorg K (CEST = GMT+2)]

Comment on attachment 8996639 [details] [diff] [review]
1480025-mozmill-screenshots-1.diff

How did you come up with Joshua as a reviewer? You could get review at Christmas ... 2020 :-(

Either ask a build person (Tom, Philipp, Rob) or MozMill man Aceman.

Comment 3

[Geoff Lankow (:darktrojan)]

I thought he might be feeling nostalgic and want to see his old code again…

Besides, I don't really care when this happens or who does it. It's not exactly important.

Comment 4

[:aceman]

So what is this doing?
In the past we had the feature to see a screenshot of the last state of the TB window when a test failed. That helped to see the problem tremendously. Somehow that got lost.

Is this patch restoring that? Can it be on by default? We would use it. Is the any large perf or storage penalty?

Comment 5

[Geoff Lankow (:darktrojan)]

(In reply to :aceman from comment #4)
> Is this patch restoring that?

Essentially, yes.

> Can it be on by default? We would use it. Is the any large perf or storage penalty?

It could be, but the reason I haven't done that is because I don't know the answer to the second question. If we don't have a problem with using more storage, then we can turn it on by default.

Comment 6

[:aceman]

Comment on attachment 8996639 [details] [diff] [review]
1480025-mozmill-screenshots-1.diff

Review of attachment 8996639 [details] [diff] [review]:
-----------------------------------------------------------------

Can you please document here how we can access the screenshots when they are created?

So we can try landing it default off for now.
Can anybody push a patch to try server just switching "UPLOAD_SCREENSHOTS = True" and he will get the screenshots in that run only?

Comment 7

[Geoff Lankow (:darktrojan)]

That is correct. Switch the False to True and any screenshots will appear in the list of artifacts from the test on TreeHerder. Click on the test (e.g. Z1) and then on Job Details.

Here's one I did earlier, although this link won't work forever:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=7a816d66ca1b377c4b050742cc3a850fdf85c35f&group_state=expanded&selectedJob=194461099

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/1cd9c94cada2
Upload screenshots as TaskCluster artifacts on failed MozMill tests. r=aceman DONTBUILD

Comment 9

[Jorg K (CEST = GMT+2)]

Nice - https://taskcluster-artifacts.net/NrZq0c5HQ0KEPbi3SjG4uQ/0/public/test_info//test-general-content-policy.js__test_insertImageIntoReply-TIQhlv.png

Aceman can now fix bug 1474219 ;-)