1281644 - Log URLs in builds-4hr should all use HTTPS to avoid mixed content breakage of the reftest analyser

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Mason Chang [Inactive] [:mchang]]

When trying to view a reftest log through try pushes on Windows, the log never shows up. See for example: https://treeherder.mozilla.org/#/jobs?repo=try&revision=afa0986611d8&selectedJob=22738463

Click on the reftest log, see "loading". From the log URL, the file doesn't actually exist, so something with either linking the correct log or the log isn't being saved seems to be the problem.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Seems to be working here?

Comment 2

[Mason Chang [Inactive] [:mchang]]

Sorry, wrong description. The reftest analyzer. Try to launch the reftest analyzer and it says "loading log" forever.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Someone else commented on IRC today that they were having issues with the reftest analyzer. Given that it's a pretty simple redirect and not something that ever changes, I'm wondering if it's possibly a regression on trunk.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

From the browser console:
Blocked loading mixed active content "http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-afa0986611d88b7707d195e4ab9982276477bfe4/try-win64-debug/try_win8_64-debug_test-reftest-no-accel-bm110-tests1-windows-build2.txt.gz"

Sure sounds relevant!

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Ed, was there anything in the recent Treeherder deploy to production that might be causing this? I'm able to reproduce this issue on builds going back to April, so this doesn't appear to be a regression in the browser itself.

Comment 6

[Ed Morley [:emorley]]

Console says:

Blocked loading mixed active content “http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-afa0986611d88b7707d195e4ab9982276477bfe4/try-win64-debug/try_win8_64-debug_test-reftest-no-accel-bm110-tests1-windows-build2.txt.gz”

Regression from bug 1117583, though with bug 1200501 coming up we'd have hit it anyway.

Log URLs will likely need to be all HTTPS - or else a server-side log fetching component created, given I don't believe there's any way to work around the mixed content blocking client side (even though it's "only" a log).

Ben, is there a bug filed for this/does it seem viable?
(The archive.mozilla.org log URL mentioned above works over HTTPS too).

Comment 7

[Ed Morley [:emorley]]

Bah midair broke the needinfo request.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

As a workaround for now, you can click the broken lock icon on the URL bar and disable protection from the ensuing dropdown.

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Ed Morley [:emorley] from comment #6)
> Console says:
> 
> Blocked loading mixed active content
> “http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-
> afa0986611d88b7707d195e4ab9982276477bfe4/try-win64-debug/try_win8_64-
> debug_test-reftest-no-accel-bm110-tests1-windows-build2.txt.gz”
> 
> Regression from bug 1117583, though with bug 1200501 coming up we'd have hit
> it anyway.
> 
> Log URLs will likely need to be all HTTPS - or else a server-side log
> fetching component created, given I don't believe there's any way to work
> around the mixed content blocking client side (even though it's "only" a
> log).
> 
> Ben, is there a bug filed for this/does it seem viable?
> (The archive.mozilla.org log URL mentioned above works over HTTPS too).

I'm probably not the best person to ask, but since HTTPS already works I would assume it would continue to. Rail is probably a better person to ask, but away at the moment.

Comment 10

[Ed Morley [:emorley]]

Rail, I don't suppose you could help us with this? :-)

archive.mozilla.org is accessible over HTTPS, so I'm hoping it shouldn't be too hard to make all log URLs in builds-4hr be HTTPS rather than the current HTTP URLs, which will avoid the mixed content issues we're seeing with the reftest analyser, now that we serve the reftest analyser over HTTPS.

Thanks!

Comment 11

[Rail Aliiev [:rail]]

I'm not 100% familiar with that part but there are some clues. :)

The URLs are set implicitly in 
https://dxr.mozilla.org/build-central/source/buildbotcustom/bin/postrun.py#108
and
https://dxr.mozilla.org/build-central/source/buildbotcustom/bin/try_mailer.py#50
(the latter should be fixed to accept https).

The URLs are printed by https://dxr.mozilla.org/build-central/source/buildbotcustom/bin/log_uploader.py#408, which uses https://dxr.mozilla.org/build-central/source/buildbotcustom/process/factory.py#144 post_upload.py, which is actually a Go program :D https://github.com/mozilla-services/product-delivery-tools/tree/master/post_upload

post_upload accepts --url-prefix, which is http://archive.mozilla.org/ by default.

We can either ask the cloudops team to change the default value or explicitly pass it.

Comment 12

[Ed Morley [:emorley]]

Ah thank you :-)

I must admit it's slightly confusing that post_upload.py is not:
https://dxr.mozilla.org/build-central/source/tools/stage/post_upload.py

but instead that Go program, but at least that explains why https://dxr.mozilla.org/build-central/ came up empty for where we were setting 'archive.mozilla.org'.

Comment 13

[Ed Morley [:emorley]]

Nick, as someone who's touched those files most recently, do you have a preference here? (As to changing the product-delivery-tools post_upload default vs passing the --url-prefix param)

I'm not familiar with how many things this will affect (eg do binary or mar uploads use the same tool? in which case just using --url-prefix for the log cases might be less risky)

In addition to the locations mentioned in comment 11, I've also found:
https://dxr.mozilla.org/build-central/source/tools/lib/python/build/upload.py

...which seems to be a dupe of the factory.py function.

Comment 14

[Rail Aliiev [:rail]]

I totally agree! :) But was the easiest way to switch to S3 for everybody. Probably it's a good idea to kill https://dxr.mozilla.org/build-central/source/tools/stage/post_upload.py. Let me do that.

Comment 15

[Ed Morley [:emorley]]

> Probably it's a good idea to kill
> https://dxr.mozilla.org/build-central/source/tools/stage/post_upload.py

Ah that would be great :-)
Perhaps also a comment at the couple of mentions of `post_upload.py` pointing to the GitHub repo would help too?

Comment 16

[Rail Aliiev [:rail]]

Attachment: Bug 1281644 - Kill post_upload.py

Review commit: https://reviewboard.mozilla.org/r/62428/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/62428/

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8768111 [details]
Bug 1281644 - Kill post_upload.py

https://reviewboard.mozilla.org/r/62428/#review59344

With fire, kill it.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

I thought this bug seemed familiar and then recalled bug 1225303. Looks like I was pointing some machines may not cope with new certs with old CA bundles if we globally went to https:// in post_upload output. 

Things have moved on a bit since then - we're using funsize for all update generation, files for tests get downloaded from taskcluster, and release promotion moved that way a lot too - all of which helps. Bug 1250629 might have brought l10n into play  through download of the en-US build, but probably not - we're using https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles in some places, and that's actually a common cert with archive.m.o. ie we'd be broken already.

So I think we should be OK to make a global change in post_upload. If we find some reason that's not true I'd suggesting modifying log_uploader.py to pass --url-prefix.

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 8768111 [details]
Bug 1281644 - Kill post_upload.py

https://hg.mozilla.org/build/tools/rev/a7e91263a4f1

Comment 21

[Ed Morley [:emorley]]

Attachment: post_upload: Default to HTTPS archive.mozilla.org URLs in logs

Comment 22

[Ed Morley [:emorley]]

Attachment: try_mailer: Support extracting HTTPS URLs from logs

Comment 23

[Ed Morley [:emorley]]

Comment on attachment 8775903 [details] [diff] [review]
try_mailer: Support extracting HTTPS URLs from logs

remote:   https://hg.mozilla.org/build/buildbotcustom/rev/d4c85e2ea3b4

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Merged to production - https://hg.mozilla.org/build/buildbotcustom/rev/61610b08ca2d

Comment 25

[Ed Morley [:emorley]]

The post_upload GitHub PR was merged, and I've opened an issue asking for a new release to pick up the fix:
https://github.com/mozilla-services/product-delivery-tools/issues/45

Once that's released, what's the process for updating the binary use in production?

Thanks :-)

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

A new binary deployed by issue 45 should be all you need.

Comment 28

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #25)
> The post_upload GitHub PR was merged, and I've opened an issue asking for a
> new release to pick up the fix:
> https://github.com/mozilla-services/product-delivery-tools/issues/45

Jeremy, I don't suppose you could take a look at this?

Comment 29

[Jeremy Orem [:oremj]]

I've commented on the issue.

Comment 30

[Ed Morley [:emorley]]

This is now in production.

Thank you all :-)