1210582 - Allow local testing of spidermonkey built as aurora or release

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect)

Description

[Steve Fink [:sfink] [:s:]]

efaust wanted a way to pass a configure flag that would allow testing what will happen during an uplift.

Comment 1

[Steve Fink [:sfink] [:s:]]

Attachment: Add --with-milestone configure option to fake aurora and release builds

It would be straightforward to just give options for setting NIGHTLY_BUILD and RELEASE_BUILD directly, but those are confusing and nobody understands them at first, so I fear such flags would be misused and fail to serve the purpose. It seems more robust to make it go through the normal milestone parsing logic.

One intended use is if something's failing after an uplift, for local testing you can pull out the milestone from milestone.txt and do --with-milestone=44.0a2 or whatever.

But hopefully the more common use case would be to test what will happen during the next uplift: --with-milestone=aurora

Comment 2

[Mike Hommey [:glandium]]

Comment on attachment 8668673 [details] [diff] [review]
Add --with-milestone configure option to fake aurora and release builds

Review of attachment 8668673 [details] [diff] [review]:
-----------------------------------------------------------------

I'm not particularly thrilled by adding more configure code to do something that could be done by just editing a file in your try pushes or local builds, specifically, you just need to edit config/milestone.txt.

Ryan, since you are or were doing that (and more) often, do you have an opinion?

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8668673 [details] [diff] [review]
Add --with-milestone configure option to fake aurora and release builds

Thing is, the simulation runs on Try are more than just version number tweaks. I'm worried that this option might actually give people a false sense of security that they're testing a configuration that they really aren't. This might be enough for a basic JS shell build, but as you said, so would changing milestone.txt directly, so it feels like this is extra complexity without much overall gain.

I've CCed philor as another relevant stakeholder in case he has more input as well.

Comment 4

[Mike Hommey [:glandium]]

In light of comment 2 and comment 3, I'm marking this WONTFIX.

Comment 5

[Phil Ringnalda (:philor)]

I don't have much skin in this game, despite being the genesis of this bug, because what I wanted was already wontfixed on IRC. jseng have wholeheartedly embraced NIGHTLY_BUILD, despite frequently forgetting they have when they write tests, so they've already landed two bits of busted tests in the first 10 days of this train. I wanted an on-push !NIGHTLY_BUILD shell build running tests, little enough since they only run on their pushes, but didn't get it. Whether or not efaust will find it acceptable to keep a patch in his queue to modify the version number, rebasing it every six weeks, I don't know.

Comment 6

[Steve Fink [:sfink] [:s:]]

(In reply to Phil Ringnalda (:philor) from comment #5)
> I wanted an on-push !NIGHTLY_BUILD
> shell build running tests, little enough since they only run on their
> pushes, but didn't get it. Whether or not efaust will find it acceptable to
> keep a patch in his queue to modify the version number, rebasing it every
> six weeks, I don't know.

Sorry, I should have mentioned that as another intended use case here: I wanted to have CI builds that run tests with the different versions set. (I didn't really want them to fire on every push to js/src, though; I was hoping we could schedule it daily and come up with a way to report the results without requiring a meat-and-bones sheriff in the loop.)

I can still do such builds, it's just that the build script would now be hacking its own source code (by editing milestone.txt). Which is tolerable, it just doesn't seem like the cleanest way to do this.

I'm not going to push this any further for now. I hope to find some time to work on some related scheduling stuff, at which time I'll take a look at doing this. But I was curious about:

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #3)
> Comment on attachment 8668673 [details] [diff] [review]
> Add --with-milestone configure option to fake aurora and release builds
> 
> Thing is, the simulation runs on Try are more than just version number
> tweaks. I'm worried that this option might actually give people a false
> sense of security that they're testing a configuration that they really
> aren't. This might be enough for a basic JS shell build, but as you said, so
> would changing milestone.txt directly, so it feels like this is extra
> complexity without much overall gain.

What else is there? For the automation jobs, I would certainly like to make these as useful as possible for predicting uplift problems before uplift, even if it's only for the JS shell. Are there other things these builds could do to usefully simulate uplift and predict breakage?

It's feeling like it's time to fork off another bug here.

Comment 7

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/try/rev/b8647e9cd5b7 and https://hg.mozilla.org/try/rev/699ecc18758d - if you get burned by branding or update channel then you've done a bad bad thing, though you could legitimately get burned by removing --enable-profiling.