Closed Bug 1336514 Opened 7 years ago Closed 7 years ago

Publish jsshell-*.zip as part of release

Categories

(Release Engineering :: Release Automation: Other, defect, P1)

Product:
Release Engineering
Component:
Release Automation: Other
Type:
defect

Tracking

(firefox58 fixed, firefox59 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox58 --- fixed
firefox59 --- fixed

People

(Reporter: mathias, Assigned: rail)

References

(Blocks 1 open bug)

Details

(Keywords: good-first-bug)

Attachments

(2 files)

Bug 1336514 - Publish jsshell-*.zip as part of release
59 bytes, text/x-review-board-request
nthomas
: review+
Details
Regexes!!!!
1012 bytes, patch
nthomas
: review+
Details | Diff | Splinter Review 
`jsshell-*.zip` files are generated as a part of release (candidates):

https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/
https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-macosx64/1486087028/

…and nightly builds:

https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/

…but none of these are officially published. This means the above URLs can’t be fully relied on for build scripts, as they might be removed at any point, or the files overwritten (for nightly builds).

Please publish them officially as versioned, static URLs as part of releases, i.e. add them to folders like:

https://archive.mozilla.org/pub/firefox/releases/51.0.1/

This would make it easier for developers to test different SpiderMonkey versions: instead of having to build them all from source, they could just `wget` and extract a `.zip` file.

(For example, projects such as Homebrew could then re-use the pre-built binaries instead of hardcoding the macOS-specific build steps. https://github.com/Homebrew/homebrew-core/pull/9607)
Ritu, what you say?
Flags: needinfo?(rkothari)
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #1)
> Ritu, what you say?

I don't know if I fully understand the implications but in general it seems like a good idea if it helps reuse pre-built files and not needing to rebuild it. If there are any disadvantages of doing this, we should reconsider.
Flags: needinfo?(rkothari)
Priority: -- → P4
FWIW, Chakra now publishes pre-built binaries of tagged releases at predictable, stable URLs.

homebrew-ecmascript uses those URLs to provide easy installers for macOS users:

    brew install mathiasbynens/ecmascript/chakra

https://github.com/mathiasbynens/homebrew-ecmascript/blob/master/Formula/chakra.rb

I can’t wait to do the same thing for SpiderMonkey!
Should be as easy as editing https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/beetmover/en_us_build.yml.tmpl and adding the files to the list.
Keywords: good-first-bug
Priority: P4 → P3
It would be good if you could also publish checksums and signatures for release versions of jsshell*.zip.

For example, on old candidate builds I could verify the download of jsshell*.zip by downloading something like:

https://archive.mozilla.org/pub/firefox/candidates/52.0esr-candidates/build4/linux-x86_64/en-US/firefox-52.0esr.checksums

and

https://archive.mozilla.org/pub/firefox/candidates/52.0esr-candidates/build4/linux-x86_64/en-US/firefox-52.0esr.checksums.asc

Inside firefox-52.0esr.checksums there's the sha512 sum for jsshell-linux-x86_64.zip :

9c63a013a4936afd1721c7fa97b587319887d061247ffe7608b3b13e8003d0ab9d06a0a4eb954ecbddf73c8d8d0dfc650a637caf41f350dd1b7cbfc0e255e4ad sha512 9061568 jsshell-linux-x86_64.zip

on later candidate builds the signed checksum files containing the checksums of jsshell are missing eg:

https://archive.mozilla.org/pub/firefox/candidates/52.1.2esr-candidates/build1/linux-x86_64/en-US/

is missing those files.

Or could the jsshell checksums be added to the root of each build? eg to:

https://archive.mozilla.org/pub/firefox/candidates/52.1.2esr-candidates/build1/SHA256SUMS
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #4)
> Should be as easy as editing
> https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/
> beetmover/en_us_build.yml.tmpl and adding the files to the list.

I don't fully understand how this all works, but it looks like the jsshell.zip is already in the list in that file. Does it perhaps need to be added to https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/releases/bouncer_firefox_release.py in order to show up on archive.mozilla.org?
Flags: needinfo?(rail)
I think I was wrong in comment 4, that file controls the "candidates" directory only. We need to remove the corresponding pattern from https://dxr.mozilla.org/mozilla-central/rev/cd9c8c48e4b3ded47a776f757008f3dcf570c59c/testing/mozharness/scripts/release/push-candidate-to-releases.py#47 so we don't ignore those files when we copy the files to the "releases" directory.

(In reply to Till Schneidereit [:till] from comment #6)
> I don't fully understand how this all works, but it looks like the
> jsshell.zip is already in the list in that file. Does it perhaps need to be
> added to
> https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/
> releases/bouncer_firefox_release.py in order to show up on
> archive.mozilla.org?

This would also add the to Bouncer (aka download.mozilla.org), but it's not required to have the files in the releases directory.
Flags: needinfo?(rail)
Assignee: nobody → rail
Priority: P3 → P1
Comment on attachment 8913842 [details]
Bug 1336514 - Publish jsshell-*.zip as part of release

https://reviewboard.mozilla.org/r/185226/#review190406

I'd prefer them stashed in a jsshell/ dir rather than the top level but *shrug*. 

There must be another blacklist which is excluding https://archive.mozilla.org/pub/firefox/candidates/57.0b4-candidates/build1/beetmover-checksums/jsshell-linux-i686.zip.beet and smilar from SHA256SUMs etc.
Attachment #8913842 - Flags: review?(nthomas) → review+
Comment on attachment 8913842 [details]
Bug 1336514 - Publish jsshell-*.zip as part of release

(In reply to Nick Thomas [:nthomas] from comment #9)
> I'd prefer them stashed in a jsshell/ dir rather than the top level but
> *shrug*. 

Good point. Shouldn't be hard. :)

> There must be another blacklist which is excluding
> https://archive.mozilla.org/pub/firefox/candidates/57.0b4-candidates/build1/
> beetmover-checksums/jsshell-linux-i686.zip.beet and smilar from SHA256SUMs
> etc.

Bah, forgot about it and AV.

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #10)
> Comment on attachment 8913842 [details]
> Bug 1336514 - Publish jsshell-*.zip as part of release
> 
> Review request updated; see interdiff:
> https://reviewboard.mozilla.org/r/185226/diff/1-2/

This one should just work.
Attachment #8913842 - Flags: review+ → review?(nthomas)
Comment on attachment 8913842 [details]
Bug 1336514 - Publish jsshell-*.zip as part of release

https://reviewboard.mozilla.org/r/185226/#review190792

Need to update https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/release/generate-checksums.py#129 to include them in the checksums files, r+ with that.
Attachment #8913842 - Flags: review?(nthomas) → review+
Can we land this?
Oh can you list some full URLs where the shell is going to be available?
(In reply to Tom Schuster [:evilpie] from comment #13)
> Can we land this?

Sure! Sorry, I was a bit busy with other stuff.

(In reply to Tom Schuster [:evilpie] from comment #14)
> Oh can you list some full URLs where the shell is going to be available?

http://archive.mozilla.org/pub/firefox/releases/58.0/jsshell/jsshell-linux-x86_64.zip (it's 404 now)
Pushed by raliiev@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/518b1b92bdae
Publish jsshell-*.zip as part of release r=nthomas
Thanks for doing this! \o/

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #15)
> (In reply to Tom Schuster [:evilpie] from comment #14)
> > Oh can you list some full URLs where the shell is going to be available?
> 
> http://archive.mozilla.org/pub/firefox/releases/58.0/jsshell/jsshell-linux-
> x86_64.zip (it's 404 now)

To see the results of this patch, do we have to wait until the Firefox 58 release in January? Or does it affect other directories as well?
> To see the results of this patch, do we have to wait until the Firefox 58
> release in January? Or does it affect other directories as well?

I realized too late that Firefox 57 hasn’t gone stable yet. Same question, but with s/January/November/ :)
I'll uplift it to beta next week.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #22)
> beta:
> https://hg.mozilla.org/releases/mozilla-beta/rev/
> 26e5b4f6b584f7e88990577cf35edf5cd2007e46

Thanks! Could you please list an example URL where this is available?
Will do when we get another beta build available (this Friday).
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #24)
> Will do when we get another beta build available (this Friday).

Thanks.
We can edit that line and remove the [^k] pattern because SDK is not a thing anymore:  https://hg.mozilla.org/mozilla-central/rev/724aad1423e6
I gave this bug a break because we have been working on the in-tree release scheduling project where uploads will be handled a bit differently. It would be better to address both issues at once, so we don't forget to port the logic to the new system.
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #28)
> I gave this bug a break because we have been working on the in-tree release
> scheduling project where uploads will be handled a bit differently. It would
> be better to address both issues at once, so we don't forget to port the
> logic to the new system.

:( Any idea when the new system is going to be ready?
Blocks: sm-embedding
We target Firefox 59, so probably mid January for betas.
Actually I can look at this some time next week if we want this to be backported. The blocking part of the new system is almost done.
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #31)
> Actually I can look at this some time next week if we want this to be
> backported. The blocking part of the new system is almost done.

That'd be fantastic: the week after next is the next tc39 meetup, and members of that committee have been asking for these builds to be available to make cross-engine test automation easier. It'd be great to give a good update at the meeting.
Attached patch Regexes!!!! — — Splinter Review 
This should address the push-to-cdns issue. The new super regexp :P should match zip and zip.asc files unless they contain "jshell-".

[re.search(r"^(?!.*jsshell-).*\.zip(\.asc)?$", f) for f in
    ("candidates/xx/yy/jsshell/jsshell-win64.zip",
     "candidates/xx/yy/zz/jsshell-mac.zip",
     "candidates/xx/yy/zz/jsshell_linux.zip",
     "candidates/xx/yy/zz/firefox.zip")]

[None,
 None,
 <_sre.SRE_Match at 0x7fb817d04dc8>,
 <_sre.SRE_Match at 0x7fb817d04d50>]
Attachment #8929146 - Flags: review?(nthomas)
Comment on attachment 8929146 [details] [diff] [review]
Regexes!!!!

Review of attachment 8929146 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm. We could copy the jsshell files into firefox/releases/57.0/ too.
Attachment #8929146 - Flags: review?(nthomas) → review+
Pushed by raliiev@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/bb9035ea4c09
Publish jsshell-*.zip as part of release r=nthomas DONTBUILD
I hope to see the files published as a part of the beta build today.
https://hg.mozilla.org/mozilla-central/rev/bb9035ea4c09
Status: REOPENED → RESOLVED
Closed: 7 years ago → 7 years ago
status-firefox59: --- → fixed
Resolution: --- → FIXED
Published beetmoverscript 3.3.0 containing this change as well. As soon as the patch gets merged in puppet, later on today, we'll have the behavior enabled too.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: