1148941 - gecko, tc-vcs: Decision tasks should fail if tc-vcs fails to pull hg.m.o

Status: RESOLVED INVALID

(Taskcluster :: General, defect)

Description

[Jonas Finnemann Jensen (:jonasfj)]

See:
http://docs.taskcluster.net/tools/task-inspector/#7yZHS0D3RH2WXsHNyP7YCw/0

Which basically ended up trying to run the wrong revision.
That is pretty bad, we only discovered it here because running
the wrong revision caused a test to fail due to configuration issues.

We have the following issues here:
 1) tc-vcs **must** exit non-zero, if it doesn't do the job
 2) decision task **must** fail if tc-vcs fails
 3) decision task should validate that it has the right checkout
 4) tc-vcs should retry with exponential back-off

Issue (3) is a nice to have. But a simple sanity check that we got the right
revision would add some extra robustness, we could do it right here:
https://dxr.mozilla.org/mozilla-central/source/testing/docker/decision/bin/entrypoint#23
Just: if [$(tc-vcs revision $DEST) != $GECKO_HEAD_REV ]; then exit 1; fi

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Any news on this?

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

Did we fix this... I can't remember if you fixed this.

Looking at entrypoint for decision task, it seems we're still missing the robustness check in bash.
Assuming we want it.

Comment 3

[James Lal [:lightsofapollo]]

https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/gjcb6sLtQvai4VKArr2ZfQ/0/public/logs/live_backing.log <- still happens

Comment 4

[James Lal [:lightsofapollo]]

Attachment: MozReview Request: Bug 1148941 - Use set in decision task vcs pull r=auswerk

Bug 1148941 - Use set in decision task vcs pull r=auswerk

Comment 5

[James Lal [:lightsofapollo]]

https://reviewboard.mozilla.org/r/11481/#review9905

::: testing/docker/builder/bin/decision.sh:1
(Diff revision 1)
> +set -ex

This is the important part

Comment 6

[Aus Lacroix [:aus]]

https://reviewboard.mozilla.org/r/11481/#review9907

lgtm! :)

Comment 7

[Pulsebot]

https://hg.mozilla.org/integration/b2g-inbound/rev/47ef6f3abc9f

Comment 8

[Aus Lacroix [:aus]]

Comment on attachment 8623313 [details]
MozReview Request: Bug 1148941 - Use set in decision task vcs pull r=auswerk

https://reviewboard.mozilla.org/r/11483/#review9909

Ship It!

Comment 9

[Pulsebot]

Backout:
https://hg.mozilla.org/integration/b2g-inbound/rev/6c11f1f4dd5d

Comment 10

[Armen [:armenzg]]

Motif of the backout:

23:03:01     INFO - Copy/paste: hgtool.py -r RELEASE_AUTOMATION https://hg.mozilla.org/build/compare-locales /home/worker/workspace/B2G/compare-locales
23:03:01     INFO - Using env: {'PATH': '/home/worker/workspace/gecko/testing/taskcluster/scripts/builder:/tools/gcc-4.7.3-0moz1/bin:/tools/tools/buildfarm/utils:/tools/python27-mercurial/bin:/tools/python27/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/worker/bin/'}
23:03:01     INFO -  Traceback (most recent call last):
23:03:01     INFO -    File "/tools/tools/buildfarm/utils/hgtool.py", line 14, in <module>
23:03:01     INFO -      from util.hg import mercurial, out, remove_path
23:03:01     INFO -    File "/tools/python27/lib/python2.7/site-packages/buildtools-1.0.6-py2.7.egg/util/hg.py", line 11, in <module>
23:03:01     INFO -      from util.retry import retry, retrier
23:03:01     INFO -    File "/tools/python27/lib/python2.7/site-packages/buildtools-1.0.6-py2.7.egg/util/retry.py", line 6, in <module>
23:03:01     INFO -      from redo import retry, retriable, retrying, retrier
23:03:01     INFO -  ImportError: No module named redo
23:03:01    ERROR - Return code: 1

https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/Pj1RXMXLSkCNv_qJm4BczA/0/public/logs/live_backing.log
https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=47ef6f3abc9f

Comment 11

[Greg Arndt [:garndt]]

Resurrecting this bug to see if either it needs to be worked on or if this issue has been resolved.  Looking at comment 10 , it's about a failed build where this patch does not touch any build tasks.  This was only done for the decision task that extends the task graph.

Armen, this is old, and I apologize for bugging you with it, but do you recall why this patch was backed out related to tasks that this patch didn't touch?

Comment 12

[Armen [:armenzg]]

garndt: I would try to push to try and see what happens.
If we could output the revision of /tools we would not the real difference.

From looking at the treeherder push we can see all builds are orange:
https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=47ef6f3abc9f&filter-searchStr=taskcluster

Perhaps the image of taskcluster/builder:0.6.0 has a newer tools repo which has the new redo dependency? [1]

For the record, we can now call hgtool.py from external_tools/ inside of mozharness instead of the one on the tools repo.
Hence making the tools repo unnecessary.



(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #0)
>  3) decision task should validate that it has the right checkout
> 
> Issue (3) is a nice to have. But a simple sanity check that we got the right
> revision would add some extra robustness, we could do it right here:
> https://dxr.mozilla.org/mozilla-central/source/testing/docker/decision/bin/
> entrypoint#23
> Just: if [$(tc-vcs revision $DEST) != $GECKO_HEAD_REV ]; then exit 1; fi

Should this validation also be done by tc-vcs and exit as non 0? (instead of the decision task)
We do this validation in hgtool.py and gittool.py.

>  4) tc-vcs should retry with exponential back-off

What does back-off mean in here?

[1]
22:39:23     INFO - Copy/paste: hgtool.py -r RELEASE_AUTOMATION https://hg.mozilla.org/build/compare-locales /home/worker/workspace/B2G/compare-locales
22:39:23     INFO - Using env: {'PATH': '/home/worker/workspace/gecko/testing/taskcluster/scripts/builder:/tools/gcc-4.7.3-0moz1/bin:/tools/tools/buildfarm/utils:/tools/python27-mercurial/bin:/tools/python27/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/worker/bin/'}
22:39:23     INFO -  Traceback (most recent call last):
22:39:23     INFO -    File "/tools/tools/buildfarm/utils/hgtool.py", line 14, in <module>
22:39:23     INFO -      from util.hg import mercurial, out, remove_path
22:39:23     INFO -    File "/tools/python27/lib/python2.7/site-packages/buildtools-1.0.6-py2.7.egg/util/hg.py", line 11, in <module>
22:39:23     INFO -      from util.retry import retry, retrier
22:39:23     INFO -    File "/tools/python27/lib/python2.7/site-packages/buildtools-1.0.6-py2.7.egg/util/retry.py", line 6, in <module>
22:39:23     INFO -      from redo import retry, retriable, retrying, retrier
22:39:23     INFO -  ImportError: No module named redo
22:39:23    ERROR - Return code: 1

Comment 13

[Armen [:armenzg]]

In order to take advantage of hgtool.py living in the tree bug 1201171 would also need to be fixed.

Unlike the hgtool.py from tools/ where it has various dependencies, this is a single file which does the same:
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/external_tools/hgtool.py

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Well, we're using `hg robustcheckout` now instead of tc-vcs or hgtool :)