Closed Bug 1470948 Opened 6 years ago Closed 5 years ago

beta release candidate builds not appearing on buildhub

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: wlach, Assigned: bhearsum)

References

Details

Just before release, beta users get upgraded to the same build that will be shipped to the release population. I believe this gets tagged as an "rc" build of some kind in the display version. However, I'm not seeing any such builds in buildhub: https://mozilla-services.github.io/buildhub/?channel[0]=beta I'm not sure why this is, my understanding is that for something to get ingested into buildhub two things need to happen: 1. Files need to get posted to archive.mozilla.org 2. Buildhub needs to ingest the metadata in those files So the first step in solving this is to see what (1) is, then fix (2) to ingest the data. I don't know the exact name of the files or versions, but I do know that I have seen significant number of beta users on a "release" version of some kind in missioncontrol before I was filtering the data on what was on buildhub, so I am pretty sure this is happening. We just need someone on relman to tell us/buildhub what the right files are. :) :catlee, could you direct this needinfo to the appropriate person on your team? The context of this request is :phillip/relman's wish to see relative crash data for the build that went to release vs. previous beta versions: https://github.com/mozilla/missioncontrol/issues/308
Flags: needinfo?(catlee)
(In reply to William Lachance (:wlach) (use needinfo!) from comment #0) > Just before release, beta users get upgraded to the same build that will be > shipped to the release population. I believe this gets tagged as an "rc" > build of some kind in the display version. However, I'm not seeing any such > builds in buildhub: > > https://mozilla-services.github.io/buildhub/?channel[0]=beta > > I'm not sure why this is, my understanding is that for something to get > ingested into buildhub two things need to happen: > > 1. Files need to get posted to archive.mozilla.org > 2. Buildhub needs to ingest the metadata in those files > > So the first step in solving this is to see what (1) is, then fix (2) to > ingest the data. I don't know the exact name of the files or versions, but I > do know that I have seen significant number of beta users on a "release" > version of some kind in missioncontrol before I was filtering the data on > what was on buildhub, so I am pretty sure this is happening. We just need > someone on relman to tell us/buildhub what the right files are. :) > > :catlee, could you direct this needinfo to the appropriate person on your > team? > > The context of this request is :phillip/relman's wish to see relative crash > data for the build that went to release vs. previous beta versions: > https://github.com/mozilla/missioncontrol/issues/308 Bug 1443873 is tracking the implementation of [1] and [2] that you mentioned. We've got everything ready to land but we're waiting for 61.0 to release tomorrow to deploy things on our end. Until that automation happens from our RelEng side, I'm not sure the mechanics of how Buildhub is currently pulling/scraping that information. Roping :peterbe and :mostlygeek in this conversation too.
Flags: needinfo?(catlee)
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #1) > Bug 1443873 is tracking the implementation of [1] and [2] that you > mentioned. We've got everything ready to land but we're waiting for 61.0 to > release tomorrow to deploy things on our end. Until that automation happens > from our RelEng side, I'm not sure the mechanics of how Buildhub is > currently pulling/scraping that information. > > Roping :peterbe and :mostlygeek in this conversation too. Looks like bug 1443873 is pretty close to being fixed? If so, happy to mark this one as a duplicate.
(In reply to William Lachance (:wlach) (use needinfo!) from comment #2) > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #1) > > Bug 1443873 is tracking the implementation of [1] and [2] that you > > mentioned. We've got everything ready to land but we're waiting for 61.0 to > > release tomorrow to deploy things on our end. Until that automation happens > > from our RelEng side, I'm not sure the mechanics of how Buildhub is > > currently pulling/scraping that information. > > > > Roping :peterbe and :mostlygeek in this conversation too. > > Looks like bug 1443873 is pretty close to being fixed? If so, happy to mark > this one as a duplicate. Bug 1443873 is only about adding another build artifact and it's *going* to be used by Buildhub2. I don't think it affects which builds are built at all.
(In reply to Peter Bengtsson [:peterbe] from comment #3) > (In reply to William Lachance (:wlach) (use needinfo!) from comment #2) > > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #1) > > > Bug 1443873 is tracking the implementation of [1] and [2] that you > > > mentioned. We've got everything ready to land but we're waiting for 61.0 to > > > release tomorrow to deploy things on our end. Until that automation happens > > > from our RelEng side, I'm not sure the mechanics of how Buildhub is > > > currently pulling/scraping that information. > > > > > > Roping :peterbe and :mostlygeek in this conversation too. > > > > Looks like bug 1443873 is pretty close to being fixed? If so, happy to mark > > this one as a duplicate. > > Bug 1443873 is only about adding another build artifact and it's *going* to > be used by Buildhub2. I don't think it affects which builds are built at all. Yeah, sorry, I meant the RelEng side of transfering those files in the right place. Indeed it doesn't affect any build stuff. It's merely about transferring and updating the information with the proper bits.
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #4) > > Bug 1443873 is only about adding another build artifact and it's *going* to > > be used by Buildhub2. I don't think it affects which builds are built at all. > > Yeah, sorry, I meant the RelEng side of transfering those files in the right > place. > Indeed it doesn't affect any build stuff. It's merely about transferring and > updating the information with the proper bits. Ok, so if bug 1443873 has nothing to do with this, where *are* the artifacts from release candidate builds being uploaded, and why aren't they appearing in buildhub? I am confused. :)
So my take on things as I understand currently is as follows (Peter or Benson, feel free to correct me if I'm wrong). Currently, Buildhub[1] is a service that services engineering maintains that collects meta data on various release artifacts that is then consumed by other services. Currently this is populated by extracting artifacts and parsing files and thus is very brittle and fragile. More here[2]. I don't know the mechanics of how currently Buildhub scrapes and populates existing data (namely for on-push or nightlies/release). The motivation in bug 1443873, AIUI, is to have instead a file (buildhub.json) encompassing accurate information about the installers at the build generation time. All the build scripts would generate it, while other RelEng (during a release process) mechanics downstream will consume this and update with the proper size (installers during release workflow change as we sign + repackage them) following which will transfer that (updated) buildhub.json artifact in the S3 for Buildhub to consume. As far as I understand this, this rethinking of how Buildhub is getting its logic is part of the "Buildhub2" which is yet to be released. Therefore: a) I don't know why RC builds are currently missing from Buildhub database, I think :peterbe might know more. b) once bug bug 1443873 lands and Buildhub2 deployed, information about release artifacts will be more accurate and less fragile, redy to be used by Buildhub2. [1]: https://mozilla-services.github.io/buildhub/ [2]: https://bugzilla.mozilla.org/show_bug.cgi?id=1443873#c0
Good summary. Yeah, Buildhub2 is about being able to act "ignorant" about how or which files are built. If a buildhub.json is shipped, we index it. Buildhub2 will NOT monitor releases that are added to archive.mozilla.org that lack a buildhub.json file. Half-jokingly I would say that Buildhub2 is a glorified index of all buildhub.json files in S3 (archive.mozilla.org). Every few years I hear a similar confusion/complaint/question that there's something odd about releases vs. final beta. I.e. it's not actually built, but just copied and renamed. I used to understand it but admit amnesia. I.e. when you make a release (pardon me if I get this wrong) instead of going back to Taskcluster to compile and ship this release we just copy the final beta and call it release candidate. Or something like that. I dare not venture deeper into my fading memory on this because I suspect a lot has changed. I suspect catlee can better explain it. What we need, as far as BuildhubX is concerned is whatever uniquely named binaries users can get hold of, that there is a buildhub.json for it. Meaning, if the step is to copy-and-rename, we need to do something similar with the buildhub.json. Over to you catlee to herd us on this.
See Also: → 1443873
Hi Chris, can you give some guidance on this? See comment 7.
Flags: needinfo?(catlee)
(In reply to William Lachance (:wlach) (use needinfo!) from comment #8) > Hi Chris, can you give some guidance on this? See comment 7. As Mihai mentioned, the work to publish buildhub.json files for all our automation isn't yet complete. When it's done, I would expect buildhubX to have a complete set of data. Until that point though, I'm not sure why the previous scraping method wouldn't be picking up the candidate releases for buildhub1. Perhaps there is some confusion as to how these RC builds are identified? There is nothing in the version string that identifies them as "RC", since these are the same builds that get shipped out to the general release users.
Flags: needinfo?(catlee)
William, If we ignore Buildhub2 for the moment; Is there a release that became available (e.g. on archive.mozilla.org) that did not get picked up by Buildhub1?
Flags: needinfo?(wlachance)
(In reply to Peter Bengtsson [:peterbe] from comment #10) > William, > If we ignore Buildhub2 for the moment; Is there a release that became > available (e.g. on archive.mozilla.org) that did not get picked up by > Buildhub1? So that was part 1 of my original question (see comment 0): I don't even know where those release builds that got shipped to the beta channel are. Wherever they may be, they are not being picked up by buildhub.
Flags: needinfo?(wlachance)
(In reply to William Lachance (:wlach) (use needinfo!) from comment #11) > (In reply to Peter Bengtsson [:peterbe] from comment #10) > > William, > > If we ignore Buildhub2 for the moment; Is there a release that became > > available (e.g. on archive.mozilla.org) that did not get picked up by > > Buildhub1? > > So that was part 1 of my original question (see comment 0): I don't even > know where those release builds that got shipped to the beta channel are. > Wherever they may be, they are not being picked up by buildhub. If you can find them in archive.mozilla.org we can figure out why they didn't get sucked up by Buildhub1.
(In reply to Peter Bengtsson [:peterbe] from comment #12) > (In reply to William Lachance (:wlach) (use needinfo!) from comment #11) > > (In reply to Peter Bengtsson [:peterbe] from comment #10) > > > William, > > > If we ignore Buildhub2 for the moment; Is there a release that became > > > available (e.g. on archive.mozilla.org) that did not get picked up by > > > Buildhub1? > > > > So that was part 1 of my original question (see comment 0): I don't even > > know where those release builds that got shipped to the beta channel are. > > Wherever they may be, they are not being picked up by buildhub. > > If you can find them in archive.mozilla.org we can figure out why they > didn't get sucked up by Buildhub1. Ok, got an answer in the channel meeting. Apparently the release builds don't migrate out of the candidates directory, they just get shipped to beta users directly from there: https://archive.mozilla.org/pub/firefox/candidates/61.0-candidates/ I'm not sure if we want buildhub to just scrape this directory, as there's almost certainly a bunch of stuff in there that never gets shipped to users.
(In reply to William Lachance (:wlach) (use needinfo!) from comment #11) > (In reply to Peter Bengtsson [:peterbe] from comment #10) > > William, > > If we ignore Buildhub2 for the moment; Is there a release that became > > available (e.g. on archive.mozilla.org) that did not get picked up by > > Buildhub1? > > So that was part 1 of my original question (see comment 0): I don't even > know where those release builds that got shipped to the beta channel are. > Wherever they may be, they are not being picked up by buildhub. Ignoring Buildhub for a sec, I'll add some context to our release automation, as per channel mtg, When shipping major release versions (e.g. 61.0), we create Release Candidates (RCs). We "test" RCs on the Beta channel prior to shipping them to Release channel users. The way we do that is somewhat unusual to other releases. Background: Normally whenever we do a release, we upload all the release files to the candidates dir: e.g. https://archive.mozilla.org/pub/firefox/candidates/*. When we are confident with our release testing, we copy those release files to the final location we serve releases from, the releases dir: e.g. https://archive.mozilla.org/pub/firefox/releases/*. This holds true for dot releases (e.g. 60.0.1), ESR releases, and normal beta releases. How RC differs: This holds true except for RC releases. For RC, we actually serve beta users release files from the candidates dir prior to copying them to the final destination under 'releases/*'. When we want to release to Release channel users, we then do our normal copy to 'releases/*' and serve Release users from there. Does that help?
Ok, after some back and forth on #release-drivers, I realized that we are already indexing these candidate builds, but only for the release channel: https://mozilla-services.github.io/buildhub/?versions[0]=62.0b3rc1&versions[1]=62.0b3 I think all we need is to add .0 releases to the beta channel as well as the release channel, to reflect the fact that these builds are also shipped to users. Looks like this should be doable with some modifications to the code here: https://github.com/mozilla-services/buildhub/blob/master/jobs/buildhub/utils.py :peterbe is this something you could do? I don't think this is hugely urgent, but it might be nice to be able to do in time for the 62 uplift (so 6 weeks from today)
Flags: needinfo?(peterbe)
(In reply to Jordan Lund (:jlund) from comment #14) > Normally whenever we do a release, we upload all the release files to the > candidates dir: e.g. https://archive.mozilla.org/pub/firefox/candidates/*. > When we are confident with our release testing, we copy those release files > to the final location we serve releases from, the releases dir: e.g. > https://archive.mozilla.org/pub/firefox/releases/*. This holds true for dot > releases (e.g. 60.0.1), ESR releases, and normal beta releases. > Can you flesh that out a little bit more? In particular "we copy those release files". In particular, what BuildhubX cares about is **writes to the S3 bucket**. In both Buildhub1 and Buildhub2 we look for files added to S3 which triggers Python code that write to our index. Also, in particular with the future in mind; if there your automation does: s3copydirectory -r https://archive.mozilla.org/pub/firefox/candidates/61.0b14-candidates/build1/win32/en-US/ https://archive.mozilla.org/pub/firefox/releases/61.0b14/win32/en-US/ ...then any .exe AND its associoated buildhub.json will get copied. > How RC differs: > > This holds true except for RC releases. For RC, we actually serve beta users > release files from the candidates dir prior to copying them to the final > destination under 'releases/*'. When we want to release to Release channel > users, we then do our normal copy to 'releases/*' and serve Release users > from there. > Again, it would really help to understand how the copying actually works. At least so we understand what it means for BuildhubX. The thing I'm concerned about is that Buildhub wants to be a searchable database of all BUILDS that users might get installed. Now an searchable database of all COMPILED things in Taskcluster. If it helps the argument; BuildhubX actually just a fancy scraper of a S3 bucket. So as long as the files are put there correctly we don't care how they got there. Or why. :)
Flags: needinfo?(peterbe) → needinfo?(jlund)
(In reply to William Lachance (:wlach) (use needinfo!) from comment #15) > Ok, after some back and forth on #release-drivers, I realized that we are > already indexing these candidate builds, but only for the release channel: > > https://mozilla-services.github.io/buildhub/?versions[0]=62. > 0b3rc1&versions[1]=62.0b3 > > I think all we need is to add .0 releases to the beta channel as well as the > release channel, to reflect the fact that these builds are also shipped to > users. Looks like this should be doable with some modifications to the code > here: > > https://github.com/mozilla-services/buildhub/blob/master/jobs/buildhub/utils. > py > > :peterbe is this something you could do? I don't think this is hugely > urgent, but it might be nice to be able to do in time for the 62 uplift (so > 6 weeks from today) This is getting complex and the discussion spans multiple teams and stuff. So excuse me for being a bit slow. Is there a file on archive.mozilla.org that did NOT get indexed by Buildhub? If so, can you point to the file/directory and I'll try to patch Buildhub1 so that it picks it up. Every valid downloadable thing in archive.mozilla.org should be index in Buildhub as long as it's distinct. Also, see my response to Jordan above.
(In reply to Peter Bengtsson [:peterbe] from comment #17) > (In reply to William Lachance (:wlach) (use needinfo!) from comment #15) > This is getting complex and the discussion spans multiple teams and stuff. > So excuse me for being a bit slow. > Is there a file on archive.mozilla.org that did NOT get indexed by Buildhub? > If so, can you point to the file/directory and I'll try to patch Buildhub1 > so that it picks it up. > > Every valid downloadable thing in archive.mozilla.org should be index in > Buildhub as long as it's distinct. Also, see my response to Jordan above. No, everything is being indexed by buildhub. It's just that .0 releases also appear on the beta channel, so we need to create duplicate entries for them in the buildhub database. Does that make sense? It seems like the logic to determine the channel a build is on is based on the version string here, note that it only ever returns a single value: https://github.com/mozilla-services/buildhub/blob/1a9a00582830bff19306da056e675f66a56ae6f2/jobs/buildhub/utils.py#L102 I *think* that answers your question to :jlund as well (at least from the perspective of this bug), so cancelling the NI there.
Flags: needinfo?(jlund)

I believe a reasonable solution to this is to ensure that all release builds are also added to buildhub2 on the beta channel.

There have been cases where we have shipped a dot release candidate to beta as well, so it's not just .0 releases that go to beta users.

(In reply to Chris AtLee [:catlee] from comment #19)

I believe a reasonable solution to this is to ensure that all release builds are also added to buildhub2 on the beta channel.

There have been cases where we have shipped a dot release candidate to beta as well, so it's not just .0 releases that go to beta users.

As discussed in a conversation with :catlee, this works for me. It basically encodes the assumption we have that any release version may also be sent to the beta channel.

Hello catlee - wondering if we have a timeline for the work that has to be done in Comment 19. Thanks.

Flags: needinfo?(catlee)

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #21)

Hello catlee - wondering if we have a timeline for the work that has to be done in Comment 19. Thanks.

This is on my plate. I should have something to show this week or next.

Assignee: nobody → bhearsum
Flags: needinfo?(catlee)

I think I have a fix for this in https://github.com/mozilla-releng/buildhub2/pull/610 per comment #19. There's a screenshot there of what things should look like once landed. Will that work for folks?

Also, do we need to backfill this?

Adding a ni to get the answer to the question in Comment 23.

Flags: needinfo?(wlachance)

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #23)

I think I have a fix for this in https://github.com/mozilla-releng/buildhub2/pull/610 per comment #19. There's a screenshot there of what things should look like once landed. Will that work for folks?

Also, do we need to backfill this?

Yay! This sounds like exactly the right approach. I suspect we will not need to backfill this data since this is mainly intended for beta channel monitoring in mission control going forward. In the event we find a use case for performing this operation on historical data, we can easily file another bug.

Flags: needinfo?(wlachance)

bhearsum - We discussed the backfill part during our meeting today and there was general agreement as to it not being needed now. In terms of timing, will this be ready when 71 enters RC? Just trying to get the timing down. Thanks again for your work on this - it is very helpful.

Flags: needinfo?(bhearsum)

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #26)

bhearsum - We discussed the backfill part during our meeting today and there was general agreement as to it not being needed now. In terms of timing, will this be ready when 71 enters RC? Just trying to get the timing down. Thanks again for your work on this - it is very helpful.

I expect this to land in production either this week or next -- so I think it's safe to say it'll be ready for 71 RC.

Flags: needinfo?(bhearsum)

The fix for this landed in production today. I hear there's a point release coming up, which is probably the best way to verify that it's working. If that doesn't come to pass I'll find another way to verify it ahead of 71.0.

I verified this today by re-publishing a 64.0 buildhub.json file, which resulted in a new entry on Buildhub: https://buildhub.moz.tools/?versions[0]=64.0&platform[0]=win64&channel[0]=beta&products[0]=firefox

I'm pretty sure this is going to work for 71.0, but I'll leave this bug open until we build it to confirm.

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #29)

I verified this today by re-publishing a 64.0 buildhub.json file, which resulted in a new entry on Buildhub: https://buildhub.moz.tools/?versions[0]=64.0&platform[0]=win64&channel[0]=beta&products[0]=firefox

I'm pretty sure this is going to work for 71.0, but I'll leave this bug open until we build it to confirm.

71.0 build2 showed up correctly: https://buildhub.moz.tools/?versions[0]=71.0&channel[0]=beta

I'm calling this fixed. If this fix doesn't end up addressing one or more use cases, let's use a new bug to track any follow-ups.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.