`GitRepository.get_outgoing_files()` returns the whole shallow history on CI git checkouts, so `mach static-analysis check --outgoing` times out on git reviewbot pushes
Categories
(Firefox Build System :: General, defect)
Tracking
(Not tracked)
People
(Reporter: nhadji, Assigned: ahochheiden)
References
(Blocks 1 open bug)
Details
Attachments
(3 external files, 2 obsolete files)
While testing the code-review bot on git (Bug 2048352) against github.com/mozilla-releng/staging-firefox, source-test-clang-tidy fails on every reviewbot push by hitting its 1h30 max-run-time. The pushed stack contains no C++ changes (taskcluster yml + python only), yet the task analyzes unrelated files (toolkit/mozapps/update/updater/updater.cpp, mozglue/misc/TimeStamp_posix.cpp, dom/media/platforms/ffmpeg/...).
The task runs mach static-analysis check --outgoing, which calls GitRepository.get_outgoing_files() (python/mozversioncontrol/mozversioncontrol/repo/git.py). That runs git log --name-only ... HEAD --not --remotes (get_mozilla_remote_args() falls back to bare --remotes when no official mozilla remote is configured). On the checkout produced by run-task with --gecko-shallow-clone, that subtraction does not match the push base, so it returns files from the whole shallow history instead of the pushed commits. base_ref uses the same --not --remotes pattern and likely degrades the same way (compare the taskgraph-diff null-base-revision failure on git pushes).
Examples on staging-firefox: task V9YZaoxqTiKoal6qyDmkyw (push f830aecdaedaf6ddd3ae14880fa0085ceeb40706, D4188 on phabricator-dev) and task exOE70W5Truu8gH5N5NFvA (push 524bcb903526774fea47dfdb7b8851bf4d02f616), both aborted at exactly 1h30. Treeherder: https://treeherder.mozilla.org/jobs?repo=staging-firefox&revision=f830aecdaedaf6ddd3ae14880fa0085ceeb40706
Impact: on git-backed pushes every reviewbot analysis reports "The analysis task source-test-clang-tidy failed", so clean revisions can never get a pass verdict - this blocks the code-review bot hg to git migration (Bug 2048352). Other --outgoing consumers (e.g. mach lint --outgoing) presumably compute the same wrong file set on CI git checkouts.
Possible fix: make CI git checkouts expose the push base (proper remote-tracking refs, or honor the base revision the decision task already knows) so get_outgoing_files() / base_ref subtract correctly.
| Assignee | ||
Updated•2 months ago
|
| Assignee | ||
Comment 1•2 months ago
|
||
GitHub reports event.before as the base of a push. For a new branch that
is the null revision and for a force push it is the orphaned old tip. In
both cases files_changed covers the whole tree, so every decision made
from it is wrong too.
The decision task now finds where the pushed history forks off the default
branch and uses that as base_rev:
- A push to the default branch keeps
event.before. - A fast forward push keeps it too. A shallow probe fetch first looks for
event.beforein the last 50 commits. If that misses, the same
ancestry check runs again after the full branch history is fetched
below, so the probe only saves a fetch and cannot change the answer. - A head that is the tip of the default branch is its own base.
- Otherwise
git fetch --shallow-exclude=<default branch>fetches the
pushed commits and nothing else. The lowest one becomes the graft root,
and the parent recorded in its raw commit object is the fork point. It
is fetched at depth 1 so the tree diff has both sides.
Anything that cannot be resolved this way, such as a shallow boundary with
more than one parent, logs a warning and keeps the reported base.
| Assignee | ||
Comment 2•2 months ago
|
||
run-task fetches the commit named by GECKO_BASE_REV at depth 1 before it
checks out the head. That gives a task the tree it needs to diff the push
against its base on a shallow clone.
fetch-base-rev turns the export on per job and is accepted by every job
type used by source-test. The source-test kind sets it for all of its
tasks, since each of them analyzes what a push changed. The null revision
is skipped because there is nothing to fetch, and a base equal to the head
because the head fetch already brings it.
| Assignee | ||
Comment 3•2 months ago
|
||
get_outgoing_files finds the files a stack touched by walking history
from HEAD back to the upstream with git log. A shallow clone has no
history to walk: git marks the fetched commits as graft roots and does not
follow their parents. When HEAD is such a root, git log treats it as the
first commit in the repository and reports every file as added. Every CI
checkout is a depth 1 clone, so this is the normal case there.
get_outgoing_files now checks whether the outgoing range contains a
graft root:
- If it does not, on full clones and on shallow clones whose outgoing
history is complete, the history walk runs unchanged. - If it does,
git diff <upstream> HEADcompares the two trees, which
needs only the two commits. The upstream must then be the exact base,
and a missing upstream raises MissingUpstreamRepo instead of guessing.
A CI checkout knows its fork point: run-task exports it as GECKO_BASE_REV
next to GECKO_HEAD_REV. When no upstream is given, get_outgoing_files
and base_ref use it ahead of looking at the remotes, provided HEAD is
that GECKO_HEAD_REV, so another repository opened in the same task does
not inherit it. Looking at the remotes first would not do: on a push to
the default branch the clone's remote branch is HEAD itself, so nothing
would look outgoing. base_ref also no longer strips the first character
of a commit when the rev list has no boundary entry.
Callers that run where no base exists catch the error instead of listing
every file: the mach artifact install hint, mach doctor, the sentry
check on mach startup, mach try coverage and the files changed lookup
behind mach taskgraph --force-local-files-changed.
| Assignee | ||
Comment 4•2 months ago
|
||
--outgoing now takes an optional upstream, the same shape as mach lint --outgoing. The clang tidy tasks pass $GECKO_BASE_REV. The variable is
only set on git checkouts, so the shell expansion leaves hg tasks with a
bare --outgoing and they keep selecting draft commits. Passing it on the
command line keeps the task's dependency on the base visible, even though
get_outgoing_files would pick the variable up itself.
The codereview target method selects these tasks by their code-review
attribute and never consulted run-on-repo-type, so [hg, git] records
what already happens.
Comment 5•1 month ago
|
||
The severity field is not set for this bug.
:ahochheiden, could you have a look please?
For more information, please visit BugBot documentation.
| Assignee | ||
Updated•1 month ago
|
Comment 6•10 days ago
|
||
(via GitHub webhook, authenticated as shared-secret)
Updated•8 days ago
|
Updated•8 days ago
|
Updated•4 days ago
|
Updated•4 days ago
|
Description
•