1676130 - Cannot merge beta to release: `/app/workdir/src/build/defines.sh does not contain EARLY_BETA_OR_EARLIER=1`

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Johan Lorenzo [:jlorenzo]]

Regressed by bug 1668908. Today is the day we merge mozilla-beta to mozilla-release. I performed the dry-run, and it failed with the following error[1]:

2020-11-09 12:43:50,153 - treescript.merges - INFO - Replacing EARLY_BETA_OR_EARLIER=1 -> EARLY_BETA_OR_EARLIER= inside /app/workdir/src/build/defines.sh
Traceback (most recent call last):
  File "/app/bin/treescript", line 8, in <module>
    sys.exit(main())
  File "/app/lib/python3.8/site-packages/treescript/script.py", line 129, in main
    return sync_main(async_main, default_config=get_default_config())
  File "/app/lib/python3.8/site-packages/scriptworker_client/client.py", line 127, in sync_main
    loop.run_until_complete(_handle_asyncio_loop(async_main, config, task))
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/app/lib/python3.8/site-packages/scriptworker_client/client.py", line 182, in _handle_asyncio_loop
    await async_main(config, task)
  File "/app/lib/python3.8/site-packages/treescript/script.py", line 102, in async_main
    await retry_async(do_actions, args=(config, task, actions_to_perform, repo_path), retry_exceptions=(CheckoutError, PushError))
  File "/app/lib/python3.8/site-packages/scriptworker_client/aio.py", line 322, in retry_async
    return await func(*args, **kwargs)
  File "/app/lib/python3.8/site-packages/treescript/script.py", line 65, in do_actions
    await perform_merge_actions(config, task, actions, repo_path, repo_type)
  File "/app/lib/python3.8/site-packages/treescript/script.py", line 38, in perform_merge_actions
    push_activity = await do_merge(config, task, repo_path)
  File "/app/lib/python3.8/site-packages/treescript/merges.py", line 218, in do_merge
    await apply_rebranding(config, repo_path, merge_config)
  File "/app/lib/python3.8/site-packages/treescript/merges.py", line 124, in apply_rebranding
    replace(os.path.join(repo_path, f), from_, to)
  File "/app/lib/python3.8/site-packages/treescript/merges.py", line 44, in replace
    raise ValueError(f"{file_name} does not contain {from_}")
ValueError: /app/workdir/src/build/defines.sh does not contain EARLY_BETA_OR_EARLIER=1

This is a genuine error because we did unset that environment variable 10 days ago[2]. It's part of the regular release process so it was the right thing to do. Although, bug 1668908 landed a month ago[3] and changed the behavior of the merge automation. While it makes total sense to unset that variable for merge simulations (it was the original intent of that bug), it breaks the real merge.

I backed out that patch[4] from mozilla-beta, in order to get today's merge going. I'm still unsure of the right way of solving this problem. That's why I'm filing this bug.

[1] https://firefox-ci-tc.services.mozilla.com/tasks/auFvwwmrTMika9D9QX37CA/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FauFvwwmrTMika9D9QX37CA%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive_backing.log
[2] https://hg.mozilla.org/releases/mozilla-beta/rev/1ec12ea99d1a0d300224ce331241b9e7a0347aa1
[3] https://hg.mozilla.org/mozilla-central/rev/19db5464f5d1c78d9bdd37b9d23c380fa1da53b2
[4] https://hg.mozilla.org/releases/mozilla-beta/rev/b7857d5bd8cd0e1746f6a838d1036d6374311646

Comment 1

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1676130 - part 1: Backed out changeset 19db5464f5d1 (bug 1668908) r=jcristau

Comment 2

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1676130 - part 2: Let `mach try release` overwrite the content of build/defines.sh r=jcristau

Depends on D96409

Comment 3

[Johan Lorenzo [:jlorenzo]]

I found a way to make this change in release-sim but not in the regular merge tasks.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1668908

Comment 5

[Julien Cristau [:jcristau] (back April 22)]

Too bad, seems we can't have a replacement that may be a no-op. :(
I guess we'll live with the footgun of building a release that pretends to be early beta if we ever forget to unset this manually...

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a1215db7a4f7
part 1: Backed out changeset 19db5464f5d1 (bug 1668908) r=jcristau
https://hg.mozilla.org/integration/autoland/rev/604df6f6576f
part 2: Let `mach try release` overwrite the content of build/defines.sh  r=jcristau

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/a1215db7a4f7
https://hg.mozilla.org/mozilla-central/rev/604df6f6576f

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:jlorenzo, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 9

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9186641 [details]
Bug 1676130 - part 2: Let mach try release overwrite the content of build/defines.sh r=jcristau

Beta/Release Uplift Approval Request

• User impact if declined: Next mergeday will be busted.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It's just an easy-to-reproduce bug that impacts Release Engineering on mergeday.
• String changes made/needed: None

Comment 10

[Julien Cristau [:jcristau] (back April 22)]

Comment on attachment 9186640 [details]
Bug 1676130 - part 1: Backed out changeset 19db5464f5d1 (bug 1668908) r=jcristau

a=npotb

Comment 11

[Julien Cristau [:jcristau] (back April 22)]

https://hg.mozilla.org/releases/mozilla-beta/rev/7fe526e2d404
https://hg.mozilla.org/releases/mozilla-beta/rev/cc3fd4663dad