Open Bug 2072920 Opened 2 months ago Updated 4 days ago

`GitRepository.get_outgoing_files()` returns the whole shallow history on CI git checkouts, so `mach static-analysis check --outgoing` times out on git reviewbot pushes

Categories

(Firefox Build System :: General, defect)

defect

Tracking

(Not tracked)

People

(Reporter: nhadji, Assigned: ahochheiden)

References

(Blocks 1 open bug)

Details

Attachments

(3 external files, 2 obsolete files)

While testing the code-review bot on git (Bug 2048352) against github.com/mozilla-releng/staging-firefox, source-test-clang-tidy fails on every reviewbot push by hitting its 1h30 max-run-time. The pushed stack contains no C++ changes (taskcluster yml + python only), yet the task analyzes unrelated files (toolkit/mozapps/update/updater/updater.cpp, mozglue/misc/TimeStamp_posix.cpp, dom/media/platforms/ffmpeg/...).

The task runs mach static-analysis check --outgoing, which calls GitRepository.get_outgoing_files() (python/mozversioncontrol/mozversioncontrol/repo/git.py). That runs git log --name-only ... HEAD --not --remotes (get_mozilla_remote_args() falls back to bare --remotes when no official mozilla remote is configured). On the checkout produced by run-task with --gecko-shallow-clone, that subtraction does not match the push base, so it returns files from the whole shallow history instead of the pushed commits. base_ref uses the same --not --remotes pattern and likely degrades the same way (compare the taskgraph-diff null-base-revision failure on git pushes).

Examples on staging-firefox: task V9YZaoxqTiKoal6qyDmkyw (push f830aecdaedaf6ddd3ae14880fa0085ceeb40706, D4188 on phabricator-dev) and task exOE70W5Truu8gH5N5NFvA (push 524bcb903526774fea47dfdb7b8851bf4d02f616), both aborted at exactly 1h30. Treeherder: https://treeherder.mozilla.org/jobs?repo=staging-firefox&revision=f830aecdaedaf6ddd3ae14880fa0085ceeb40706

Impact: on git-backed pushes every reviewbot analysis reports "The analysis task source-test-clang-tidy failed", so clean revisions can never get a pass verdict - this blocks the code-review bot hg to git migration (Bug 2048352). Other --outgoing consumers (e.g. mach lint --outgoing) presumably compute the same wrong file set on CI git checkouts.

Possible fix: make CI git checkouts expose the push base (proper remote-tracking refs, or honor the base revision the decision task already knows) so get_outgoing_files() / base_ref subtract correctly.

Assignee: nobody → ahochheiden

GitHub reports event.before as the base of a push. For a new branch that
is the null revision and for a force push it is the orphaned old tip. In
both cases files_changed covers the whole tree, so every decision made
from it is wrong too.

The decision task now finds where the pushed history forks off the default
branch and uses that as base_rev:

  • A push to the default branch keeps event.before.
  • A fast forward push keeps it too. A shallow probe fetch first looks for
    event.before in the last 50 commits. If that misses, the same
    ancestry check runs again after the full branch history is fetched
    below, so the probe only saves a fetch and cannot change the answer.
  • A head that is the tip of the default branch is its own base.
  • Otherwise git fetch --shallow-exclude=<default branch> fetches the
    pushed commits and nothing else. The lowest one becomes the graft root,
    and the parent recorded in its raw commit object is the fork point. It
    is fetched at depth 1 so the tree diff has both sides.

Anything that cannot be resolved this way, such as a shallow boundary with
more than one parent, logs a warning and keeps the reported base.

run-task fetches the commit named by GECKO_BASE_REV at depth 1 before it
checks out the head. That gives a task the tree it needs to diff the push
against its base on a shallow clone.

fetch-base-rev turns the export on per job and is accepted by every job
type used by source-test. The source-test kind sets it for all of its
tasks, since each of them analyzes what a push changed. The null revision
is skipped because there is nothing to fetch, and a base equal to the head
because the head fetch already brings it.

get_outgoing_files finds the files a stack touched by walking history
from HEAD back to the upstream with git log. A shallow clone has no
history to walk: git marks the fetched commits as graft roots and does not
follow their parents. When HEAD is such a root, git log treats it as the
first commit in the repository and reports every file as added. Every CI
checkout is a depth 1 clone, so this is the normal case there.

get_outgoing_files now checks whether the outgoing range contains a
graft root:

  • If it does not, on full clones and on shallow clones whose outgoing
    history is complete, the history walk runs unchanged.
  • If it does, git diff <upstream> HEAD compares the two trees, which
    needs only the two commits. The upstream must then be the exact base,
    and a missing upstream raises MissingUpstreamRepo instead of guessing.

A CI checkout knows its fork point: run-task exports it as GECKO_BASE_REV
next to GECKO_HEAD_REV. When no upstream is given, get_outgoing_files
and base_ref use it ahead of looking at the remotes, provided HEAD is
that GECKO_HEAD_REV, so another repository opened in the same task does
not inherit it. Looking at the remotes first would not do: on a push to
the default branch the clone's remote branch is HEAD itself, so nothing
would look outgoing. base_ref also no longer strips the first character
of a commit when the rev list has no boundary entry.

Callers that run where no base exists catch the error instead of listing
every file: the mach artifact install hint, mach doctor, the sentry
check on mach startup, mach try coverage and the files changed lookup
behind mach taskgraph --force-local-files-changed.

--outgoing now takes an optional upstream, the same shape as mach lint --outgoing. The clang tidy tasks pass $GECKO_BASE_REV. The variable is
only set on git checkouts, so the shell expansion leaves hg tasks with a
bare --outgoing and they keep selecting draft commits. Passing it on the
command line keeps the task's dependency on the base visible, even though
get_outgoing_files would pick the variable up itself.

The codereview target method selects these tasks by their code-review
attribute and never consulted run-on-repo-type, so [hg, git] records
what already happens.

See Also: → 2073031

The severity field is not set for this bug.
:ahochheiden, could you have a look please?

For more information, please visit BugBot documentation.

Flags: needinfo?(ahochheiden)
Severity: -- → S4
Flags: needinfo?(ahochheiden)
Attachment #9645332 - Attachment is obsolete: true
Attachment #9645333 - Attachment is obsolete: true
Attachment #9645331 - Attachment description: Bug 2072920 - Use the fork point off the default branch as the base of GitHub pushes r?ahal → Bug 2072920 - Pass the base revision run-task derives to the GitHub decision task r?ahal
Attachment #9645334 - Attachment description: Bug 2072920 - Pass the base revision to `mach static-analysis check --outgoing` r?ahal → Bug 2072920 - Run clang-tidy on the files the decision task found changed r?ahal
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: