1833507 - docker image task expired its docker image before the task expired

Status: NEW

(Firefox Build System :: Task Configuration, defect, P2)

Description

[[e-mail] (:bhearsum)]

This causes us to insert it into graphs as a cached task, but the tasks that depended on it couldn't pull the image.

Task in question: https://firefox-ci-tc.services.mozilla.com/tasks/J1D9qCm9SHmjbXjs8EH6sA

It expires on 2024-02-14T05:27:42.688Z while the docker image artifact was 2023-05-15T05:27:42.688Z.

I don't know if this is widespread or not, but it could be. Possibly related to https://bugzilla.mozilla.org/show_bug.cgi?id=1809580 ?

Comment 1

[Mike Hommey [:glandium]]

Considering the task you're linking is defined with:

  ...
  "expires": "2024-02-14T05:27:42.688Z",
  ...
  "artifacts": {
      "public/image.tar.zst": {
        "path": "/workspace/image.tar.zst",
        "type": "file",
        "expires": "2023-05-15T05:27:42.688Z"
      }
    },

... pretty sure it's the same underlying issue.

Comment 2

[Julien Cristau [:jcristau]]

A couple ideas we've discussed:

• adding a verification to ensure that cached tasks' artifacts don't expire until the task itself does
• having tasks declare the artifacts they need, and make index-search take that into consideration when deciding whether to reuse a task

Comment 3

[Mike Hommey [:glandium]]

having tasks declare the artifacts they need, and make index-search take that into consideration when deciding whether to reuse a task

For tasks that use fetch-content (which should be most of them except tests), that's in the MOZ_FETCHES environment variable in the task definition.

Comment 4

[Mike Hommey [:glandium]]

3 months later, this happened again (because that's the expiry time for the docker images).

Comment 5

[Mike Hommey [:glandium]]

... which says bug 1809580 was not enough... (look around) so, in fact bug 1809580 seems to have been enough, but when it landed, some images weren't retriggered (or all of them?), so we still had the soon to expire docker images from before bug 1809580. Heitor triggered the Rebuild Cached Tasks action, which created new images that expire in a year.

Comment 6

[Andrew Halberstadt [:ahal]]

I wrote up a verification for this in bug 1860081, but it actually didn't find any problematic tasks. To test, I modified some cached tasks to expire their artifacts after a day and the verification detected those no problem, so I think it's working properly.

So I guess someone must have fixed cached task expiry at some point, and the errors were seeing now are just the tasks that were created before the fix? If that's the case, we could rebuild cached tasks if we wanted to avoid seeing this again.

Comment 7

[[e-mail] (:bhearsum)]

It looks like we hit this again on esr115: https://treeherder.mozilla.org/jobs?repo=mozilla-esr115&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&revision=baa3abffc5a68e9f07dfb834ee443908f7df6221

There's a number of tasks in there that depend on https://firefox-ci-tc.services.mozilla.com/tasks/I-5otmLQS4K4cOY3CcSdqg, which expired its toolchain artifacts a day before the failing tasks were created.