1482786 - 500 Server Error: Internal Server Error with references to different trees than the one that holds the failures

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Natalia Csoregi [:nataliaCs]]

There have been some build bustages on some pushes on m-i and try

E.g. https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=f4bbda249beed3143907a74c0683c87c9a9a692b&group_state=expanded&selectedJob=193582086


HTTPError: 500 Server Error: Internal Server Error for url: https://queue.taskcluster.net/v1/task/A17yqz7qRKuodUonoSnJAA/artifacts/public/chainOfTrust.json.asc

Checking the above link https://taskcluster-artifacts.net/A17yqz7qRKuodUonoSnJAA/0/public/chainOfTrust.json.asc , noticed some references to some maple revisions, e.g. : 
 "metadata": {
      "owner": "[e-mail]",
      "source": "https://hg.mozilla.org/projects/maple/file/87998708e1e56ac53293bea05205658378c65dc5/taskcluster/ci/toolchain",
      "description": "Node repack toolchain build ([Treeherder push](https://treeherder.mozilla.org/#/jobs?repo=maple&revision=87998708e1e56ac53293bea05205658378c65dc5))",
      "name": "toolchain-linux64-node"
    },
There are cases where the m-i tree opens when clicking on the link from the similar links to the one above e.g. https://treeherder.mozilla.org/#/jobs?repo=maple\u0026revision=6fc7c3a7b74aa0804fe868d450c222fc1e8d4cc6 but the revision is nowhere to be found, or another case where it points to this push https://treeherder.mozilla.org/#/jobs?repo=maple&tochange=6fc7c3a7b74aa0804fe868d450c222fc1e8d4cc6&fromchange=b66612c01771e4ad690233a903e3c2f23fe318bb

:bhearsum is this expected behavior, for tc links to point to different trees than the ones where the failure occurs?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Urgh, A17yqz7qRKuodUonoSnJAA is indeed a maple toolchain-linux64-node task, which sets the routes:
index.gecko.cache.level-3.toolchains.v2.linux64-node.hash.7be6c6ffac14d08058f0b606b0c91dbc017a565406a898dd639dcea7c80567ef
index.gecko.cache.level-3.toolchains.v2.linux64-node.latest
index.gecko.cache.level-3.toolchains.v2.linux64-node.pushdate.2018.08.09.20180809130554

The middle one is probably how a m-i push picked it up. We'll need to configure maple more carefully, somehow.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Hit again:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=cd69094e843a3fec0e963b62a107c5c1519d9385&selectedJob=193658212 

Log: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=819b923159847ebf0227a910e966490f4f9dc177 

Which mentions:
    "metadata": {
      "owner": "[e-mail]",
      "source": "https://hg.mozilla.org/integration/mozilla-inbound/file/819b923159847ebf0227a910e966490f4f9dc177/taskcluster/ci/toolchain",
      "description": "Clang 6 toolchain build ([Treeherder push](https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=819b923159847ebf0227a910e966490f4f9dc177))",
      "name": "toolchain-linux64-clang-6"
    },

Comment 3

[Tom Prince [:tomprince]]

Note that the 503 error is unrelated cross-tree tasks. The cross tree tasks are expected, though they shouldn't point to maple.

Comment 4

[[e-mail] (:bhearsum)]

The quick fix here is probably to force a rebuild of toolchain-linux64-node job on inbound - and maybe back something out of maple, or possibly merge central to it. I'm not familiar with the node jobs, and I don't think we're doing anything with them on maple.

Long term, the fact that maple is a level 3 repo is very problematic. We use this repo for staging release testing - which means we can't guarantee that it stays in a working state. Being level 3 means it shares toolchains and some other things with other level 3 repos (as I understand it).

Comment 5

[Tom Prince [:tomprince]]

The original issue is fixed. I filed Bug 1491491 to track making maple not be level 3.