1754723 - Updatebot fails when updating libjxl due to symlinks

Status: RESOLVED DUPLICATE of bug 1760886

(Developer Infrastructure :: Mach Vendor & Updatebot, defect, P1)

Description

[June Wilde (she/her) [:jewilde]]

Even though libjxl's moz.yaml file excludes the tools directory symlinks in tools/benchmark/metrics it seems as if they aren't being deleted on import

These symlinks have existed in upstream for more than a year, and the exclude rule has been in the moz.yaml file for more than four months. Issue was seen when updating to commit 4322679b1c418addc2284c5ea84fc2c3935b4a75 from 9a74bd70b7932750deb78a8aebd6e041ce7f8b01 with mozilla-central 78719225247cfbe9c17c7a65deeeb29b4d931c15 and Updatebot d000053c55f90487038de1df8986d1fb5f6a79ad

I can't seem to reproduce the failure locally running the exact commands Updatebot ran.

The cron job log in treeherder shows the symlinks being removed prior to running ./mach try auto where we end up throwing an error due to them being included in the patch.

[task 2022-02-07T21:08:55.370Z] remote: adding changesets
[task 2022-02-07T21:08:55.370Z] remote: adding manifests
[task 2022-02-07T21:08:55.370Z] remote: adding file changes
[task 2022-02-07T21:08:55.370Z] remote: 
[task 2022-02-07T21:08:55.370Z] remote: ****************************** ERROR *******************************
[task 2022-02-07T21:08:55.370Z] remote: 37cd60ccaefc adds or modifies the following symlinks:
[task 2022-02-07T21:08:55.370Z] remote: 
[task 2022-02-07T21:08:55.370Z] remote:   third_party/jpeg-xl/tools/benchmark/metrics/fsim-y.sh
[task 2022-02-07T21:08:55.370Z] remote:   third_party/jpeg-xl/tools/benchmark/metrics/lpips-rgb.sh
[task 2022-02-07T21:08:55.370Z] remote:   third_party/jpeg-xl/tools/benchmark/metrics/msssim-rgb.sh
[task 2022-02-07T21:08:55.370Z] remote:   third_party/jpeg-xl/tools/benchmark/metrics/msssim-y.sh
[task 2022-02-07T21:08:55.370Z] remote:   third_party/jpeg-xl/tools/benchmark/metrics/nlpd-y.sh
[task 2022-02-07T21:08:55.370Z] remote: 
[task 2022-02-07T21:08:55.370Z] remote: Symlinks aren't allowed in this repo. Convert these paths to regular
[task 2022-02-07T21:08:55.370Z] remote: files and try your push again.
[task 2022-02-07T21:08:55.370Z] remote: ********************************************************************
[task 2022-02-07T21:08:55.370Z] remote: 
[task 2022-02-07T21:08:55.370Z] remote: transaction abort!

https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=updatebot&revision=78719225247cfbe9c17c7a65deeeb29b4d931c15

I think looking into changes that landed in mach vendor and updatebot around February 2nd might be a good start to figuring out what happened here, but right now I'm not exactly certain what it could be.

Comment 1

[Tom Ritter [:tjr] (ooto until 9/1)]

My initial thought was "I have slightly different python versions on treeherder and locally - I bet it's that." But we use mozfile.remove(). It hasn't changed.

I think to figure this out you're going to need to reproduce it in holly. After you've done that, you can really start debugging it.

mozfile will not complain if the thing doesn't exist; it might be worth adding an if not os.path.exists(path): log() here.

You can also add a subprocess.run("ls -R " + tmpextractdir, shell=True) (don't forget import) before here

Comment 3

[June Wilde (she/her) [:jewilde]]

Latest occurrence is in Bug 1760719
https://treeherder.mozilla.org/logviewer?job_id=371828914&repo=mozilla-central&lineNumber=6977
https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=updatebot&revision=c15abadbc3c7008549ecb562326483749698ce46

Comment 4

[Tom Ritter [:tjr] (ooto until 9/1)]

The problem is that os.path.exsists returns False for broken symlinks; and therefore mozfile.remove does not try to remove the file.

Comment 5

[Tom Ritter [:tjr] (ooto until 9/1)]

This should be fixed with Bug 1760886