1556748 - Implement balrog rule to update 64-bit Windows OS users who are running 32-bit Thunderbird

Status: NEW

(Thunderbird :: Build Config, enhancement)

Description

[Rob Lemley [:rjl]]

Attachment: win32Towin64_balrog.py

The way this worked for Firefox, there's two Balrog rules:

• Rule 685 acted as a watershed for all Windows users to install FF 56.0
• Rule 681 then migrated only users running 56.0 and met the other criteria

Criteria for migrating to 64-bit:

• Did not opt out with the registry setting created by the installation of 56.0
• Running 32-bit app on 64-bit Windows
• Have more than 2G of RAM

It looks like we need a watershed for Windows users on the next beta so the updater can set that registry key. We'll have to do the same for the release branch as well.

I've modified the script that releng used for Firefox to mangle the Balrog blob for a given release. It removes the non-windows data and sets the mapping for win32 on x64 users to get the 64 bit version.

Comment 1

[Rob Lemley [:rjl]]

Attachment: FF_Beta_watershed_rule.png

For reference, this is the watershed rule that forced all users to install FF-56.0.

Comment 2

[Rob Lemley [:rjl]]

Attachment: FF_Beta_migration_rule.png

For reference, this is the rule that did the actual migration for eligible users.

Comment 3

[Rob Lemley [:rjl]]

Attachment: TB_win32Towin64.md

Some notes on how to set up Balrog

Comment 4

[Rob Lemley [:rjl]]

The URLs that get generated by the Thunderbird updater should look something like (from a recent nightly):

https://aus5.mozilla.org/update/6/Thunderbird/69.0a1/20190526233640/WINNT_x86-msvc-x64/%LOCALE%/%CHANNEL%/Windows_NT 6.1.1.0.7601 (x64)/ISET:SSE4_2,MEM:32101/default/default/update.xml?mig64=1

The important pieces from a Balrog perspective:

• build target: WINNT_x86-msvc-x64 that indicates an x86 build running on x64 OS
• RAM: MEM:32101 (in MB, must be > 2048 to trigger the upgrade)
• opt-in: mig64=1

I've got this working on the "nightlytest" update channel. You have to start off with a build < 20190604091847. You will get two updates, the first will go to Build 20190604091847 which will set the magic registry value, then on the next update if you meet the criteria you should go to Build 20190605101101 for win64.

I can put this on the regular nightly channel if desired.

Comment 5

[Richard Marti (:Paenglab)]

Do we need to set the magic registry value also on the next, or one of the next, TB 60 updates to make it possible to update from 32bit TB 60 to 64bit TB68?

Comment 6

[Worcester12345]

(In reply to Richard Marti (:Paenglab) from comment #5)

Do we need to set the magic registry value also on the next, or one of the next, TB 60 updates to make it possible to update from 32bit TB 60 to 64bit TB68?

Is there a process in place to go from 32 bit TB60 to 64 bit TB60? THAT would be the most logical first step; to only change one thing at a time. Otherwise, if something breaks, there are too many variables.

Comment 7

[Rob Lemley [:rjl]]

We could flip the registry bit for esr60 on the next release to get the process going. It doesn't do anything except send another parameter to the update server. Balrog will ignore it and continue sending win32 downloads unless we tell it otherwise.

Comment 8

[Rob Lemley [:rjl]]

Attachment: win32Towin64_balrog.py

Updated script, works with a little less interaction from user now.

Comment 9

[Worcester12345]

Just updated to 60.8.0 (32-bit). Still 32 bit.

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to Worcester12345 from comment #9)

Just updated to 60.8.0 (32-bit). Still 32 bit.

Patience grasshopper. This bug is not fixed, so of course this is expected. Migration won't happen until the balrog piece described by this bug is done.

Comment 11

[Worcester12345]

I think I read comment 4 and comment 5 together very quickly, and somehow thought this was done. Thanks for working on it.

Comment 12

[Rob Lemley [:rjl]]

Attachment: Win32 to Win64 Automigrate Testing.html

This test document assumes a watershed for Windows users at 60.8 and that the win64 migration happens on 68.2.
I just verified that the script to create the special migration release in Balrog still works.

Comment 13

[Wayne Mery (:wsmwk)]

Comment on attachment 9091141 [details]
Win32 to Win64 Automigrate Testing.html

I think win64 for 68.2 is reasonable, so SGTM.  I'd just double check with Magnus, so adding NI.

We still have Bug 1509918 - "use thunderbird as default client" prompt on every startup on WIndows 7 using TB 64bit.  However the issue also appears with 32bit Thunderbird so I don't think it blocks this bug - but we should still commit to fixing it.

Comment 14

[Magnus Melin [:mkmelin]]

Comment on attachment 9091141 [details]
Win32 to Win64 Automigrate Testing.html

Looks good. 
While testing that migration works, can we also at the same time add an item to check that it found the existing profile after migration?

Comment 15

[Worcester12345]

This title looks whacky.
"Implement balrog rule to update 32-bit Windows Thunderbird users running 64-bit OS"

Is this what it really wants to say?
"Implement balrog rule to update 64-bit Windows OS users who are running 32-bit Thunderbird"

Comment 16

[Rob Lemley [:rjl]]

Attachment: Win32 to Win64 Automigrate Testing.html

Updated testing documentation version numbers, and added notes to check that existing profiles are used.

Comment 17

[Rob Lemley [:rjl]]

Attachment: Win32 to Win64 Automigrate Testing.html

Last minute notes about testing setup. Scenario C can be tested effectively.

Comment 18

[Rob Lemley [:rjl]]

Attachment: image.png

release-localtest is configured with automigration. The migration version being tested is 68.2.0. When 68.2.1 is released I will make the appropriate changes and copy to the regular release channel.

Comment 19

[Rob Lemley [:rjl]]

Attachment: Win32 to Win64 Automigrate Testing.html

I've completed my testing of automigration using the release-locatest channel. Everything looks good to me, profiles remained in tact.
The opt out methods both work. I've updated the testing documentation with the actual registry keys that need to be set. Bypassing migration by installing the 32-bit version of 68.2.1 will be easiest for most people though.

Comment 20

[Worcester12345]

What is left to get this bug closed out?

Comment 21

[Wayne Mery (:wsmwk)]

(In reply to Worcester12345 from comment #20)

What is left to get this bug closed out?

In general terms ... more important stuff which has emerged (profile issues), moving more users from 60 (because we don't want to move users from 60 and 64bit at the same time), and retesting. That's probably an incomplete list.

It's fluid and dynamic, and so it's not possible to give more specifics, dates or version numbers. And, to be clear about relative priorities, shipping 64bit hasn't been and still is not critical.

Comment 22

[Worcester12345]

(In reply to Wayne Mery (:wsmwk) from comment #21)

(In reply to Worcester12345 from comment #20)

What is left to get this bug closed out?

In general terms ... more important stuff which has emerged (profile issues), moving more users from 60 (because we don't want to move users from 60 and 64bit at the same time), and retesting. That's probably an incomplete list.

It's fluid and dynamic, and so it's not possible to give more specifics, dates or version numbers. And, to be clear about relative priorities, shipping 64bit hasn't been and still is not critical.

It goes beyond that. It also shows a commitment to moving forward. Most all other software is now 64 bit. It is a bit embarrassing for Thunderbird to be behind, to be honest. It's about optics; and it isn't very difficult to do that last 1%.

Comment 24

[Worcester12345]

Just added before the 1598866 bug was marked a duplicate:
Blocks: https://bugzilla.mozilla.org/show_bug.cgi?id=1556222

Comment 25

[Worcester12345]

(In reply to Wayne Mery (:wsmwk) from comment #21)

(In reply to Worcester12345 from comment #20)

What is left to get this bug closed out?

In general terms ... more important stuff which has emerged (profile issues), moving more users from 60 (because we don't want to move users from 60 and 64bit at the same time), and retesting. That's probably an incomplete list.

It's fluid and dynamic, and so it's not possible to give more specifics, dates or version numbers. And, to be clear about relative priorities, shipping 64bit hasn't been and still is not critical.

Are there any statistics on how many users are still on version 60? At what point does 64 bit BECOME more "critical"?

Comment 26

[Worcester12345]

(In reply to Wayne Mery (:wsmwk) from comment #21)

(In reply to Worcester12345 from comment #20)

What is left to get this bug closed out?

In general terms ... more important stuff which has emerged (profile issues), moving more users from 60 (because we don't want to move users from 60 and 64bit at the same time), and retesting.

Would it be safe to say that the "move users from 60" is mostly done now?

(In reply to Worcester12345 from comment #25)

Are there any statistics on how many users are still on version 60? At what point does 64 bit BECOME more "critical"?

?

Comment 28

[Worcester12345]

Please add key word: "support-win64-thunderbird" from duplicate bug. Thank you.

Comment 29

[Wayne Mery (:wsmwk)]

(In reply to Worcester12345 from comment #28)

Please add key word: "support-win64-thunderbird" from duplicate bug. Thank you.

That's already used by bug 634233. Besides, we don't need an alias for this bug.

Comment 30

[Worcester12345]

I think the original intent of this bug was to be identical to:

Migrate 32-bit Firefox (WOW64) users to 64-bit Firefox (Win64)
https://bugzilla.mozilla.org/show_bug.cgi?id=1274659

except for Thunderbird.

Comment 31

[Worcester12345]

OK, what is still holding this bug back from installing 64 bit Thunderbird? Based on numerous crashes, is it time to move this forward?

Comment 32

[Worcester12345]

From the submitted attachment 11 [details] months ago:
"The current plan is to enable automigration with the 68.2.1 release"

Comment 33

[Worcester12345]

(In reply to Wayne Mery (:wsmwk) from comment #21)

(In reply to Worcester12345 from comment #20)

What is left to get this bug closed out?

In general terms ... more important stuff which has emerged (profile issues), moving more users from 60 (because we don't want to move users from 60 and 64bit at the same time), and retesting. That's probably an incomplete list.

So are users off 60 yet? Retesting done?

It's fluid and dynamic, and so it's not possible to give more specifics, dates or version numbers. And, to be clear about relative priorities, shipping 64bit hasn't been and still is not critical.

Another year has passed. And it was critical enough that you suggested it to me as a solution to many crashes I was having, and it actually did work as you suggested it might. I think it is time to ship this out. Put it on the schedule, so it will be ready before Halloween, or at the worst, Columbus Day. Thank you.

Comment 34

[Wayne Mery (:wsmwk)]

I recall providing a potential time line in a different bug report, so I don't see the value in asking the same question in a different bug report or setting arbitrary dates. Also note that migrating users to 64bit will itself create some initial instability for some users - at least that is what we see in support circles, some users encounter profiles issues. So additional testing will be required beyond what has already occurred.

I understand your passion for this issue. But those driving releases do not have capacity to do everything we would wish beyond the most important - currently that is fixing important version 78 bugs, bringing users to version 78, and solidifying Openpgp functionality, to name a few. Do you have an analysis and data which shows what specific user segment will benefit from 64bit, and that indicates how critical that need is?

Comment 35

[Worcester12345]

(In reply to Wayne Mery (:wsmwk) from comment #34)

... I don't see the value in... setting arbitrary dates.

It is not arbitrary. It is a solid date, a month or two out. Most of the work is already done as of long ago.

I understand your passion for this issue.
Only from the frustration of repeated crashes. I wish you to spare others the same agony.

But those driving releases do not have capacity to do everything we would wish beyond the most important - currently that is fixing important version 78 bugs, bringing users to version 78,

But isn't this PRECISELY what you said about "moving more users from 60", literally right above here?

Do you have an analysis and data which shows what specific user segment will benefit from 64bit, and that indicates how critical that need is?

I'm sorry, I am not a software analyst nor a programmer. I am a volunteer "Tester", who is helping you with this little project. I was suckered into believing it was a "community project" or something like that, where everybody contributes somehow and does what they can to help out. I've sent in my share of bugs, crashes, suggestions, etc. over the years; for both Thunderbird, and Firefox, and even "Seamonkey" and "Sunbird" and "Lightning". As I said, it was YOUR suggestion for me to try 64-bit, which helped a lot.

Comment 36

[Wayne Mery (:wsmwk)]

Rob, I'm reevaluating whether we have any outstanding issues in bug 634233, etc. Assuming we find nothing major, and after retesting various scenarios users might encounter ...

For beta, what do you think about doing this during the 87 beta cycle, i.e. in March? (not at release, but during the cycle)
For release, what do you think about doing this ~8 weeks later during the 78.10, i.e in May. (not at release, but during the cycle)

In both cases, we probably want to avoid doing this very close to some other major change, for example enabling a major portion of e10s.
Is there any aspect that we'd want to discuss on tb-planning?

Comment 37

[Rob Lemley [:rjl]]

That timeline sounds good. I just verified that the 32to64 migration registry keys are set in current win32 on x64 installs and are sending the "mig64=1" parameter to Balrog.

During the release itself, after promotion but before calls for testing go out, I will need to mangle the release blob in Balrog. Then testing can proceed via the -localtest/-cdntest channels.

One thing I want to clarify: Thunderbird must have been installed using the installer. That's where the registry key to enable is created. ZIP installs don't get the registry key until they go through at least one upgrade.

Comment 38

[Worcester12345]

Are you going to use the same process as Firefox to do this?

If so, you would need a Thunderbird version of this bug, among others:
https://bugzilla.mozilla.org/show_bug.cgi?id=1340936

No need to reinvent the wheel here, just follow the same process and steps.

Comment 39

[Rob Lemley [:rjl]]

We don't have a stub installer.

Comment 40

[Worcester12345]

(In reply to Rob Lemley [:rjl] from comment #39)

We don't have a stub installer.

Here's the bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1558573

Comment 41

[Worcester12345]

(In reply to Wayne Mery (:wsmwk) from comment #36)

Rob, I'm reevaluating whether we have any outstanding issues in bug 634233, etc. Assuming we find nothing major, and after retesting various scenarios users might encounter ...

For beta, what do you think about doing this during the 87 beta cycle, i.e. in March?
...

(In reply to Rob Lemley [:rjl] from comment #37)

That timeline sounds good.
...

Fantastic! Thank you. This should help others who were having issues similar to what I had, that putting on 64 bit build helped alleviate.

Comment 42

[Worcester12345]

The "87 beta cycle, i.e. in March" has come and gone. Now on "89 beta cycle".

Probably want to change "version" up top from 68 also.

Comment 43

[Wayne Mery (:wsmwk)]

(In reply to Worcester12345 from comment #42)

The "87 beta cycle, i.e. in March" has come and gone. Now on "89 beta cycle".

We nixed the time line in part because of the pandemic and in part because of lack of the release teams available time to test. So we're going to need a new plan.

Probably want to change "version" up top from 68 also.

version is meaningless for enhancement bug reports

Comment 44

[Worcester12345]

There is no programming ("patch") involved in this bug, which is outstanding, is there?

And, there are no blocking bugs?

If so, what is the reasoning for not finishing this and holding it back?

Thanks.

Comment 45

[Wayne Mery (:wsmwk)]

(In reply to Worcester12345 from comment #44)

There is no programming ("patch") involved in this bug, which is outstanding, is there?

Correct, it's not a Thunderbird code patch. The implementation details are here in comment 0.

And, there are no blocking bugs?
If so, what is the reasoning for not finishing this and holding it back?

Try rereading comment 43 from 11 days ago. When there is a new plan it will be posted here.

Comment 46

[Worcester12345]

How about this for the new plan:

"Criteria for migrating to 64-bit:
Running 32-bit app on 64-bit Windows
Have more than 2G of RAM
It looks like we need a watershed for Windows users on the next beta so the updater can set that registry key. We'll have to do the same for the release branch as well."

Any deviations from this new plan can be posted here.

Comment 47

[Rob Lemley [:rjl]]

The registry key change was done back in Thunderbird 69 (and uplifted to Thunderbird 68). There is a watershed on version 68.12.1 and another on 78.14.0 that will take care of actually setting the registry key so there is no need for another.

(In reply to Worcester12345 from comment #46)

How about this for the new plan:

"Criteria for migrating to 64-bit:
Running 32-bit app on 64-bit Windows
Have more than 2G of RAM
It looks like we need a watershed for Windows users on the next beta so the updater can set that registry key. We'll have to do the same for the release branch as well."

Any deviations from this new plan can be posted here.

Comment 48

[Rob Lemley [:rjl]]

We recently had an enterprise admin do a mass migration of various Thunderbird versions from 32-bit to 64-bit using this method. It's a self-maintained update server, but the principles used are the same.

This is very encouraging. I feel a lot better about migrating 102.x installs now.

On 12/25/22 11:58, [e-mail] wrote:

Thanks so much for the help,

It was actually simpler than I thought The transition from X86 to X64 took place without problems.
I confirm that the pointing remained for C: \ Program File (x86) \ for the X86 clients but the program is correctly x64

For clients 60.2.1 the written point in my thunderbird.cfg file
defaultpref ("Autoadmin.global_config_url", "http://mysite.com/tb/thunderbird.js");
It did not work, perhaps because it is not yet developed.

To work I inserted this pointing in the channel-prefs.js file
pref ("App.update.url", "http://mysite.com/tb/update.xml");

Now the clients, thanks to your help, are migrating to the latest version and all on a X64 base

<?xml version="1.0"?>
<updates>
<update type="minor" appVersion="60.9.1" displayVersion="60.9.1" buildID="20191031083309">
       <patch type="complete" URL="Http://mysite.com/TB/win64/thunderbird-60.9.1.complete.mar" size="37803251"/>
   </update>
<update type="minor" appVersion="68.12.0" displayVersion="68.12.0"  promptWaitTime="10" buildID="20200820223055">
       <patch type="complete" URL="Http://mysite.com/TB/win64/thunderbird-68.12.0.complete.mar" size="45857870"/>
   </update>
<update type="minor" appVersion="78.14.0" displayVersion="78.14.0"  promptWaitTime="10" buildID="20210901192859">
       <patch type="complete" URL="Http://mysite.com/TB/win64/thunderbird-78.14.0.complete.mar" size="55521106"/>
   </update>
<update type="minor" appVersion="91.13.1" displayVersion="91.13.1"  promptWaitTime="10" buildID="20220908191136">
       <patch type="complete" URL="Http://mysite.com/TB/win64/thunderbird-91.13.1.complete.mar" size="59596142"/>
   </update>
<update type="minor" appVersion="102.6.0" displayVersion="102.6.0" promptWaitTime="10" buildID="20221212195513">
       <patch type="complete" URL="Http://mysite.com/TB/win64/thunderbird-102.6.0.complete.mar" size="60478856"/>
   </update>
<update type="major" appVersion="102.6.1" displayVersion="102.6.1" promptWaitTime="10" buildID="20221219215418">
       <patch type="complete" URL="Http://mysite.com/TB/win64/thunderbird-102.6.1.complete.mar" size="60479524"/>
   </update>
</updates>

Comment 49

[Wayne Mery (:wsmwk)]

Rob says nightly users haven't been updated. So, in order, we do a balrog rule for nightly, then beta, then release.

Andrei chatted about the telemetry we need (bug 1808254), "I think that xpcomabi already gives us that [telemetry] info for Windows anyway". Andrei, can you schedule that work?

Comment 50

[Worcester12345]

What is the anticipated time frame on this being done?

Comment 51

[Neustradamus]

Any news about it?
Thunderbird 115 exists...

Comment 52

[Wayne Mery (:wsmwk)]

Per matrix next step is nightly channel. At your convenience.

Comment 53

[Andrei Hajdukewycz [:sancus]]

I think we want Bug 1879717 before we do this on release builds. Nightly and Beta are fine without.

Comment 54

[Worcester12345]

From bug https://bugzilla.mozilla.org/show_bug.cgi?id=1911076
" gene smith
Comment 20 • 3 days ago
It appears that 128/32-bit can't handle compacting of mbox files of more than 4GB. I tried running 115.13.0/32-bit and deleted a few messages in 8GB mbox folder and then compacted them and the size just went down a small amount. Tried this on several large mbox folders with no problem. Then installed 128.0.1/32-bit again and repeated the experiment and the foldes are truncated to 4GB with half the messages inaccessible."

Why won't it install the 64 bit version then, if that one WILL work on files of more than 4GB? I just checked for updates on a 64 bit Windows 10 computer, and it only does the 32 bit installation of Thunderbird. Seems like your stepping on your own privates here.