Closed Bug 1227637 Opened 4 years ago Closed 4 years ago

Install the correct mesa library for the desktop-test image

Categories

(Taskcluster Graveyard :: Docker Images, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: armenzg, Assigned: armenzg)

References

Details

Attachments

(3 files, 1 obsolete file)

Spin off from bug 1223123

I appended this to /etc/apt/sources.list
deb http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng/ precise main

Is this the right way of installing it? [1] I specified more packages as from looking at a test machine [2]
Should I generate an image based on this for now?
Should I open a separate bug for this mesa work and make less noise in here?

You mentioned that we would need to go about using this by putting this into tooltool.
I inspected the file referenced in here [3]
I see a bunch of .deb file and some metadata files and dist/ dir.

How do I go about generating something like it?
> reply from dustin:
> There's an update.sh in that tarball which you can adapt to your purposes.
> Bascially you put the .deb's you want into a subdirectory, run ./update.sh
> to generate the metadatas, and then tar up the whole thing.  If the image
> works out for you, I can build a repo with just the files you mentioned.

[1]
apt-get install -y --force-yes libgl1-mesa-dev:i386=8.0.4-0ubuntu0.6mozilla1 libgl1-mesa-dri=8.0.4-0ubuntu0.6mozilla1 libgl1-mesa-dri:i386=8.0.4-0ubuntu0.6mozilla1 libgl1-mesa-glx=8.0.4-0ubuntu0.6mozilla1 libgl1-mesa-glx:i386=8.0.4-0ubuntu0.6mozilla1 libglapi-mesa=8.0.4-0ubuntu0.6mozilla1 libglapi-mesa:i386=8.0.4-0ubuntu0.6mozilla1 libglu1-mesa=8.0.4-0ubuntu0.6mozilla1 libglu1-mesa:i386=8.0.4-0ubuntu0.6mozilla1 libxatracker1=8.0.4-0ubuntu0.6mozilla1 mesa-common-dev:i386=8.0.4-0ubuntu0.6mozilla1
[2]
[cltbld@tst-linux64-ec2-armenzg.test.releng.use1.mozilla.com ~]$ dpkg -l | grep 6mozilla1 | sort
ii  libgl1-mesa-dev:i386                   8.0.4-0ubuntu0.6mozilla1                free implementation of the OpenGL API -- GLX development files
ii  libgl1-mesa-dri                        8.0.4-0ubuntu0.6mozilla1                free implementation of the OpenGL API -- DRI modules
ii  libgl1-mesa-dri:i386                   8.0.4-0ubuntu0.6mozilla1                free implementation of the OpenGL API -- DRI modules
ii  libgl1-mesa-glx                        8.0.4-0ubuntu0.6mozilla1                free implementation of the OpenGL API -- GLX runtime
ii  libgl1-mesa-glx:i386                   8.0.4-0ubuntu0.6mozilla1                free implementation of the OpenGL API -- GLX runtime
ii  libglapi-mesa                          8.0.4-0ubuntu0.6mozilla1                free implementation of the GL API -- shared library
ii  libglapi-mesa:i386                     8.0.4-0ubuntu0.6mozilla1                free implementation of the GL API -- shared library
ii  libglu1-mesa                           8.0.4-0ubuntu0.6mozilla1                Mesa OpenGL utility library (GLU)
ii  libglu1-mesa:i386                      8.0.4-0ubuntu0.6mozilla1                Mesa OpenGL utility library (GLU)
ii  libxatracker1                          8.0.4-0ubuntu0.6mozilla1                X acceleration library -- runtime
ii  mesa-common-dev:i386                   8.0.4-0ubuntu0.6mozilla1                Developer documentation for Mesa

[3] https://dxr.mozilla.org/mozilla-central/source/testing/docker/ubuntu1204-test/system-setup.sh#181
[4] https://mozilla-releng-usw2-tooltool.s3-us-west-2.amazonaws.com/sha512/45005c7e1fdb4839f3bb96bfdb5e448672e7e40fa3cfc22ef646e2996a541f151e88c61b8a568cc00b7fcf5cb5e98e00c4e603acb0b73f85125582fa00aae76e?Signature=tTOHHkXRtWGRPZeogMcCi8l2FSM%3D&Expires=1448321058&AWSAccessKeyId=AKIAILEHYUYSBPB5N3DQ
Hi dustin, if you could help with the packaging that would be great.
I can keep on moving forward.
The installation seemed to go well.
Assignee: armenzg → dustin
Armen: do you want 8.0.4 or the 9.2.1 from bug 1220658?
Flags: needinfo?(armenzg)
We would eventually need 9.2.1; let's do it :)

Thanks Dustin
Flags: needinfo?(armenzg)
Hah, I had the 8.0.4 tarball ready to upload to tooltool when we got DDOS'd -- so I never uploaded it :)

Based on the repo generated in bug 1220658, I've built a tarball with

./pool/precise/libgl1-mesa-glx-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libgl1-mesa-dri-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libgles1-mesa-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libegl1-mesa-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libgbm-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libgles2-mesa-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libgbm-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libgles1-mesa-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libxatracker1-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libglapi-mesa-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libegl1-mesa-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libxatracker-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libgles2-mesa-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/mesa-common-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libgles2-mesa-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libopenvg1-mesa-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libopenvg1-mesa-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libgl1-mesa-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libgles2-mesa-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libgl1-mesa-dri-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libopenvg1-mesa-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libgl1-mesa-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libegl1-mesa-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libgles1-mesa-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libxatracker1-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libgles1-mesa-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libegl1-mesa-drivers-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/mesa-common-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libgl1-mesa-glx-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libegl1-mesa-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libegl1-mesa-drivers-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libopenvg1-mesa-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libxatracker-dev-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libglapi-mesa-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./pool/precise/libgbm1-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_i386.deb
./pool/precise/libgbm1-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1_amd64.deb
./update.sh
./dists/precise/all/binary-i386/Packages
./dists/precise/all/binary-i386/Packages.bz2
./dists/precise/all/binary-amd64/Packages
./dists/precise/all/binary-amd64/Packages.bz2

I left out the package tarball and diffs, and uploaded them to tooltool separately, for future reference:

[
{
"size": 59446, 
"visibility": "public", 
"digest": "920b094e3ec2a0d14a84ed48299ecb836b90d19e839ad2583bdb9ca3df90426ca1f89acb3e33d6bf8440bf256bc5e6cceb3c2ee8b88c7ed1e2190ec51b44108f", 
"algorithm": "sha512", 
"filename": "mesa-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1.diff.gz"
}, 
{
"size": 3834, 
"visibility": "public", 
"digest": "d31f8795d0fd52d448a76531debd17e17d7e8c62ecb20cdaeb746f9c541b22e4754b4e35d1b2a36583a9b35c0e7db77dd2bf5153183ab6c97174236231b44def", 
"algorithm": "sha512", 
"filename": "mesa-lts-saucy_9.2.1-1ubuntu3~precise1mozilla1.dsc"
}, 
{
"size": 8850801, 
"visibility": "public", 
"digest": "1bb80a4d2788d4fc2c63819e71a3afb70ed5657c15780dcbed1488c73e9a3ba8771d6e3b0fc719e2c55b316064e1ce865d1360ce566055fcdbe69e5d0aa97709", 
"algorithm": "sha512", 
"filename": "mesa-lts-saucy_9.2.1.orig.tar.gz"
}, 
{
"size": 11619521, 
"visibility": "public", 
"digest": "2ec1dbb735d06a5ae8c1f31cd0a9256339a50b0221c84fc1f6e8ada3b3c2757c16df6e2ace4a66f68159950d46ddce563ecd71dd5c2d2d421b085168a86a52fd", 
"algorithm": "sha512", 
"filename": "mesa-repo-9.2.1-1ubuntu3~precise1mozilla1.tar.gz"
}
]
I've already uploaded the images as taskcluster/desktop-test:0.4.5, if you want to test it out.
Bug 1227637/1220658: install mesa 9.2.1-1ubuntu3~precise1mozilla1 in desktop-test; r?armenzg
Attachment #8693930 - Flags: review?(armenzg)
Comment on attachment 8693930 [details]
MozReview Request: Bug 1227637: install mesa 9.2.1-1ubuntu3~precise1mozilla2 in desktop-test; r?armenzg

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/26619/diff/1-2/
Comment on attachment 8693930 [details]
MozReview Request: Bug 1227637: install mesa 9.2.1-1ubuntu3~precise1mozilla2 in desktop-test; r?armenzg

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/26619/diff/2-3/
(In reply to Dustin J. Mitchell [:dustin] from comment #6)
> I've already uploaded the images as taskcluster/desktop-test:0.4.5, if you
> want to test it out.

Which is to say, I overwrote the existing image being used in production :( :(

(I was based on an old copy of inbound)

I rebuilt 0.4.5 from the tip of mozilla-central, re-pushed it, and then pushed my changes as 0.4.6.  Sorry about that!
Duplicate of this bug: 1229297
Attachment #8693930 - Flags: review?(armenzg) → review+
Comment on attachment 8693930 [details]
MozReview Request: Bug 1227637: install mesa 9.2.1-1ubuntu3~precise1mozilla2 in desktop-test; r?armenzg

https://reviewboard.mozilla.org/r/26619/#review24147

Beautiful! Just a small nit. Thanks dustin!

::: testing/docker/ubuntu1204-test/system-setup.sh:225
(Diff revision 3)
> +    install libgl1-mesa-dev-lts-saucy:i386 \

Could you please move the word install above?
^^ that's unrelated, dunno why the try push did that
<armenzg> dustin, in case you need to get the syntax for desktop test changes - https://treeherder.mozilla.org/#/jobs?repo=try&revision=f358404125a0 and https://treeherder.mozilla.org/#/jobs?repo=try&revision=420f3350bcb3

(the images are still building right now)
As I mentioned on IRC, we're seeing unexpected test passing for the gl job.

jrmuizel: would you expect these tests to fail if we upgrade to the newer mesa?

19:11:19     INFO -  96 INFO TEST-UNEXPECTED-PASS | dom/canvas/test/webgl-conformance/_wrappers/test_conformance__extensions__oes-texture-float.html | fail-if condition in manifest - We expected at least one failure
19:45:52     INFO -  685 INFO TEST-UNEXPECTED-PASS | dom/canvas/test/webgl-conformance/_wrappers/test_conformance__textures__texture-mips.html | Expected failures: conformance/textures/texture-mips.html - expected FAIL
19:46:21     INFO -  792 INFO TEST-UNEXPECTED-PASS | dom/canvas/test/webgl-conformance/_wrappers/test_conformance__textures__texture-size-cube-maps.html | Expected failures: conformance/textures/texture-size-cube-maps.html - expected FAIL
Flags: needinfo?(jmuizelaar)
(In reply to Armen Zambrano Gasparnian [:armenzg] from comment #20)
> As I mentioned on IRC, we're seeing unexpected test passing for the gl job.
> 
> jrmuizel: would you expect these tests to fail if we upgrade to the newer
> mesa?
> 
> 19:11:19     INFO -  96 INFO TEST-UNEXPECTED-PASS |
> dom/canvas/test/webgl-conformance/_wrappers/
> test_conformance__extensions__oes-texture-float.html | fail-if condition in
> manifest - We expected at least one failure
> 19:45:52     INFO -  685 INFO TEST-UNEXPECTED-PASS |
> dom/canvas/test/webgl-conformance/_wrappers/
> test_conformance__textures__texture-mips.html | Expected failures:
> conformance/textures/texture-mips.html - expected FAIL
> 19:46:21     INFO -  792 INFO TEST-UNEXPECTED-PASS |
> dom/canvas/test/webgl-conformance/_wrappers/
> test_conformance__textures__texture-size-cube-maps.html | Expected failures:
> conformance/textures/texture-size-cube-maps.html - expected FAIL

Nope, presumably these are failing because of the old version. Them passing is an added benefit of upgrading mesa.
Flags: needinfo?(jmuizelaar)
This is one of those cases when the terms "pass" and "fail" are ambiguous, but from the sound of it, the patch tested out OK.

Armen, am I clear to land?
Can you please disable the GL tier2 job as part of this? [1]
I don't want it running orange on Treeherder.

I will have to figure out how to change the tests *only* when running on a docker image (TaskCluster) or wait until bug 1220658 also lands for the Buildbot test machines.

Thanks dustin.

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=13022e404bfd&filter-searchStr=Linux%2064%20debug%20gl
yeah, we will need to coordinate fixing the manifests for when the buildbot ec2 images are updated.

one idea is to disable these few tests in all cases, then re-enabled them as expected-pass; that would imply that we are within a week or two of getting the mesa libraries updated for the ec2 instances.
Comment on attachment 8693930 [details]
MozReview Request: Bug 1227637: install mesa 9.2.1-1ubuntu3~precise1mozilla2 in desktop-test; r?armenzg

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/26619/diff/3-4/
Attachment #8693930 - Attachment description: MozReview Request: Bug 1227637/1220658: install mesa 9.2.1-1ubuntu3~precise1mozilla1 in desktop-test; r?armenzg → MozReview Request: Bug 1227637/1220658: install mesa 9.2.1-1ubuntu3~precise1mozilla1 in desktop-test; r=armenzg
Bug 1227637: disable mochitest-webgl on TC for the moment; r?armenzg
Attachment #8694774 - Flags: review?(armenzg)
Comment on attachment 8694774 [details]
MozReview Request: Bug 1227637: disable mochitest-webgl on TC for the moment; r?armenzg

https://reviewboard.mozilla.org/r/26869/#review24295

lgtm
Attachment #8694774 - Flags: review?(armenzg) → review+
https://hg.mozilla.org/mozilla-central/rev/4478329ac0f9
https://hg.mozilla.org/mozilla-central/rev/5648bce2ec8d
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
We would like to revert this change.

Is this the right command to remove the packages? [1]
apt-get -q -y --force-yes remove libgl1-mesa-dev-lts-saucy:i386 libgl1-mesa-dri-lts-saucy libgl1-mesa-dri-lts-saucy:i386 libgl1-mesa-glx-lts-saucy libgl1-mesa-glx-lts-saucy:i386 libglapi-mesa-lts-saucy libglapi-mesa-lts-saucy:i386 libxatracker1-lts-saucy  mesa-common-dev-lts-saucy:i386

And I would have to let the packages get installed naturally by reverting this? [2]

[1] https://hg.mozilla.org/mozilla-central/rev/4478329ac0f9#l6.106
[2] https://hg.mozilla.org/mozilla-central/rev/4478329ac0f9#l6.12
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment on attachment 8704333 [details]
MozReview Request: Bug 1227637 - Revert mesa

Does this do the right trick?

I'm building the images now to see if there are any issues and will push to try tomorrow (or perhaps tonight).
Attachment #8704333 - Flags: feedback?(dustin)
Attachment #8704333 - Flags: feedback?(dustin)
Wrt to wpt8:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=050d6b6dd77d,034d7071c2f2&filter-searchStr=W(8)

gl stays the same:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=050d6b6dd77d,034d7071c2f2&filter-searchStr=gl&selectedJob=15125679

Without this change [1]:
> 16:04:30     INFO - TEST-UNEXPECTED-PASS | /webgl/bufferSubData.html | bufferSubData - expected FAIL

With this change [2]:
> 19:03:12     INFO - TEST-FAIL | /webgl/bufferSubData.html | bufferSubData - assert_true: Should be able to get a context. expected true got false

[1] https://public-artifacts.taskcluster.net/WBThygXMQeSWCUN15yh-Kw/0/public/logs/live_backing.log
[2] https://public-artifacts.taskcluster.net/NBG_bYnQQS6as72UHgTVqA/0/public/logs/live_backing.log
dustin: could you please let me know when the new images are uploaded so I can land the change? Thanks!
Docker is not being nice to me:

Selecting previously unselected package xfonts-scalable.
Unpacking xfonts-scalable (from .../xfonts-scalable_1%3a1.0.3-1_all.deb) ...
Selecting previously unselected package xul-ext-ubufox.
Unpacking xul-ext-ubufox (from .../xul-ext-ubufox_3.2-0ubuntu0.12.04.1_all.deb) ...
Segmentation fault (core dumped)
dpkg: error processing /var/cache/apt/archives/xul-ext-ubufox_3.2-0ubuntu0.12.04.1_all.deb (--unpack):
 subprocess new pre-installation script returned error exit status 139
Segmentation fault (core dumped)
dpkg: error while cleaning up:
 subprocess new post-removal script returned error exit status 139
Selecting previously unselected package xvfb.
Unpacking xvfb (from .../xvfb_2%3a1.11.4-0ubuntu10.17_amd64.deb) ...
Segmentation fault (core dumped)

..or maybe it's the EC2 instance I'm using, I don't know.  I'll keep trying!  Unless you've already generated images and I can just pull and re-push them?
OK, pushed.  I must have an old kernel or something.. worked fine with AUFS.
I don't see it.

armenzg@armenzg-thinkpad:~$ docker pull taskcluster/desktop-test:0.5.3
Pulling repository docker.io/taskcluster/desktop-test
Tag 0.5.3 not found in repository docker.io/taskcluster/desktop-test
I wonder why that failed the first time :(

https://hub.docker.com/r/taskcluster/desktop-test/tags/ -- it's up there now
https://hg.mozilla.org/integration/mozilla-inbound/rev/514090cdb74ff1009d96b2a4b64d7197ad86ad86
Bug 1227637 - Revert mesa upgrade for desktop-test jobs (TaskCluster Linux64 debug tier2 jobs). r=dustin
https://hg.mozilla.org/mozilla-central/rev/514090cdb74f
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
It seems we need to deploy this again.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
it looks like mesa is coming back.  I am working on greening up the tests- we need desktop-test to have the latest drivers, this looks like the code that backed it out:
https://reviewboard.mozilla.org/r/29619/diff/1#index_header
Flags: needinfo?(dustin)
So let's figure out how to time that, but we'll need to rebuild and `docker push` the base images again, with new version numbers.  Other than that wrinkle, this is basically a re-landing of the backed-out patch.
Flags: needinfo?(dustin)
I will grab this as it is visible on Treeherder.
Assignee: dustin → armenzg
Attached patch mesa.diffSplinter Review
Missing the tooltool info.
It also needs the ubuntu test images being created.

Wrt to version I only changed to 0.1.8 and changed 11 to 18 (for 18th of Feburary)
Attachment #8720946 - Flags: feedback?(dustin)
In a perfectly expected way, since rather than doing anything like comment 24 we just left them permaorange on buildbot for two days, and then finally greened them up by changing the manifest to make them green on buildbot and permaorange on taskcluster until it gets an updated mesa. I just hid the taskcluster mochitest-gl, rather than looking at any more permaorange.
Comment on attachment 8720946 [details] [diff] [review]
mesa.diff

This looks a lot like the patch I have in my working copy, where I'm building the images now :)
Attachment #8720946 - Flags: feedback?(dustin) → feedback+
So, the repository I copied into tooltool is, I think the repository from
  https://bugzilla.mozilla.org/show_bug.cgi?id=1220658#c92
with everything *mozilla1* removed from it.  In particular, it includes the cherry-picked stuff (libdrm-* and libllvm3.3*).  It's not clear that's necessary on a more up-to-date precise image such as we're using.  So if this turns out to not work, that will be the place to look.
docker images are pushed based on this patch..
Comment on attachment 8693930 [details]
MozReview Request: Bug 1227637: install mesa 9.2.1-1ubuntu3~precise1mozilla2 in desktop-test; r?armenzg

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/26619/diff/4-5/
Attachment #8693930 - Attachment description: MozReview Request: Bug 1227637/1220658: install mesa 9.2.1-1ubuntu3~precise1mozilla1 in desktop-test; r=armenzg → MozReview Request: Bug 1227637: install mesa 9.2.1-1ubuntu3~precise1mozilla2 in desktop-test; r?armenzg
Attachment #8694774 - Attachment is obsolete: true
It seems my push last night did not pass:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ead4e8e3b50f&filter-searchStr=gl&exclusion_profile=false&selectedJob=16928015
The reason being that I forgot about the change in test manifets [1]. I will rebase and push again.

[1]
https://hg.mozilla.org/integration/mozilla-inbound/rev/bd0cafabc32c
I've autolanded this. We should see a notification.
https://hg.mozilla.org/mozilla-central/rev/52d0db384a03
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
Product: Taskcluster → Taskcluster Graveyard
You need to log in before you can comment on or make changes to this bug.