1262460 - fennec-beta-latest download links are 404

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Alex Gibson [:agibson]]

Our automated test run just picked up that fennec-beta-latest links are all 404 :(

https://ci.us-west.moz.works/job/bedrock_integration_tests_runner/4595/testReport/

Comment 1

[Alex Gibson [:agibson]]

Example 404 link from the test report:

https://download.mozilla.org/?product=fennec-beta-latest&os=android&lang=multi

Comment 2

[Liz Henry (:lizzard) (privacy/fingerprinting team)]

I can download beta firefox for android and it also is updating correctly to beta 7. What does this test failure mean? Do I need to worry about it?

Comment 3

[Alex Gibson [:agibson]]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #2)
> I can download beta firefox for android and it also is updating correctly to
> beta 7. What does this test failure mean? Do I need to worry about it?

It means all the download links here are broken:

https://www.mozilla.org/en-US/firefox/android/beta/all/

Comment 4

[Alex Gibson [:agibson]]

And also anywhere else we provide a direct download link for Fennec Beta on the site.

Comment 5

[Rail Aliiev [:rail]]

Looks like we haven't published files to CDNs (http://ftp.mozilla.org/pub/mobile/releases/), but updated the bouncer aliases (in postrelease)? Kim, can you look at this, please.

Comment 6

[Kim Moir [:kmoir] ET]

I fixed it.

I looked like so the fennec checksums, copy to releases, push to mirrors jobs didn't run.  I'll open a bug against releng to see why these jobs are failing silently.

Comment 7

[Alex Gibson [:agibson]]

Looks like there is still one failure: 

https://download.mozilla.org/?product=fennec-beta-latest&os=android-api-9&lang=nl

Comment 8

[Alex Gibson [:agibson]]

Kim, could you please take a look at this last broken download link?

Comment 9

[Kim Moir [:kmoir] ET]

I just got to work, looking at it now.

Comment 10

[Kim Moir [:kmoir] ET]

I just spent an hour or so looking at s3 and our candidates versus releases dir

from #releaseduty
<kmoir> so the missing nl apk for android 9 isn't in the candidates dir either
10:25 AM <kmoir> there are 152 apks in the android 15 dir, 138 in the android 9
10:25 AM <kmoir> going to go figure out why/if they should be different

Comment 11

[Kim Moir [:kmoir] ET]

Figured out the problem
the repack 7/10 failed a couple of times
When I looked at it on the morning of the april 5, I rechecked the failures and saw that it was green
but looking at it again the api15 one is green and the api9 is red
so I will rerun that repack 
generate checksums etc
and copy those files over manually to s3

Comment 12

[Alex Gibson [:agibson]]

(In reply to Kim Moir [:kmoir] from comment #11)
> Figured out the problem
> the repack 7/10 failed a couple of times
> When I looked at it on the morning of the april 5, I rechecked the failures
> and saw that it was green
> but looking at it again the api15 one is green and the api9 is red
> so I will rerun that repack 
> generate checksums etc
> and copy those files over manually to s3

Great - thanks for the investigation, Kim!

Comment 13

[Kim Moir [:kmoir] ET]

Can you rerun your tests? cloudops has copied over the files now to s3

Comment 14

[Josh Mize [:jgmize]]

Verified that https://download.mozilla.org/?product=fennec-beta-latest&os=android-api-9&lang=nl is working now, and I just kicked off an integration test run at https://ci.us-west.moz.works/job/bedrock_test_stage_eu_west/928/

Comment 15

[Josh Mize [:jgmize]]

(In reply to Josh Mize [:jgmize] from comment #14)
> Verified that
> https://download.mozilla.org/?product=fennec-beta-latest&os=android-api-
> 9&lang=nl is working now, and I just kicked off an integration test run at
> https://ci.us-west.moz.works/job/bedrock_test_stage_eu_west/928/

The test succeeded; all the download links are working again.