Closed Bug 1279952 Opened 4 years ago Closed 4 years ago

Remove checks related to mxr.mozilla.org

Categories

(Infrastructure & Operations :: MOC: Service Requests, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: fubar, Assigned: ryanc)

References

(Depends on 1 open bug)

Details

mxr.mozilla.org is hardhat'ed for a maintenance issue on the web head. 

Anyone that needs to search code in the main firefox trees (mozilla-central, -release, -aurora, -beta, -er45) can use dxr.mozilla.org in the meantime.
Please set redirects to dxr or rewrite mxr links on Bugzilla so that we don't lose track.

Permalinks such as https://mxr.mozilla.org/mozilla-central/source/toolkit/components/osfile/modules/osfile_win_back.jsm?rev=4663c05c869c&mark=104-112#104 should be supported.
(In reply to Masatoshi Kimura [:emk] from comment #1)
> Please set redirects to dxr or rewrite mxr links on Bugzilla so that we
> don't lose track.
> 
> Permalinks such as
> https://mxr.mozilla.org/mozilla-central/source/toolkit/components/osfile/
> modules/osfile_win_back.jsm?rev=4663c05c869c&mark=104-112#104 should be
> supported.

Rewriting links is completely out of scope for this issue, sorry. I'm not sure if that's actually something that we'll be able to do with DXR, as they are two entirely different applications, but you're welcome to file a bug in Webtools:DXR for it.
(In reply to Masatoshi Kimura [:emk] from comment #1)
> Please set redirects to dxr or rewrite mxr links on Bugzilla so that we
> don't lose track.
> 
> Permalinks such as
> https://mxr.mozilla.org/mozilla-central/source/toolkit/components/osfile/
> modules/osfile_win_back.jsm?rev=4663c05c869c&mark=104-112#104 should be
> supported.

however, it turns out that :erikrose is a very clever person and designed dxr with maintaining mxr's path structure. queries, obviously, will not carry over but links like you suggest should.
(In reply to Kendall Libby [:fubar] from comment #2)
> Rewriting links is completely out of scope for this issue, sorry. I'm not
> sure if that's actually something that we'll be able to do with DXR, as they
> are two entirely different applications,

You will notice DXR supports permalinks when once you use DXR.

  https://mxr.mozilla.org/mozilla-central/source/path/to/file?rev=<revid>
Could be rewritten to
  https://dxr.mozilla.org/mozilla-central/rev/<revid>/path/to/file
.

> but you're welcome to file a bug in
> Webtools:DXR for it.

I don't have to file a DXR bug because DXR already supports permalinks. What component is appropriate to request URL rewrite?
DXR also supports multiline highlight (e.g. "?mark=104-112#104" to "#104-112"), but unfortunately it is not possible to rewrite it on the server because server side redirects cannot contain a hash.
Can we restore service of mxr.mozilla.org/addons separately? That repo is private and not accessible (and so not rewritable) to DXR). It being unavailable is hurting our ability to check for add-on compat issues etc...
Flags: needinfo?(klibby)
Depends on: 1280113
Is there an alternate URL where we can redirect mozroots until mxr is available again?
http://mxr.mozilla.org/seamonkey/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1
(In reply to greg.marr from comment #7)
> Is there an alternate URL where we can redirect mozroots until mxr is
> available again?
> http://mxr.mozilla.org/seamonkey/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1

https://hg.mozilla.org/mozilla-central/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt
?
(Files under http://mxr.mozilla.org/seamonkey/ are very old because they are frozen since Firefox 3.5. Is it really a file that you want?)
It's the file currently being requested by the version of mozroots that comes with the version of mono that we're using (3.2.8 I think).
Is this the file that we should be using all the time now?
(In reply to greg.marr from comment #9)
> It's the file currently being requested by the version of mozroots that
> comes with the version of mono that we're using (3.2.8 I think).
> Is this the file that we should be using all the time now?

It depends on the reason why the URL has not been updated since Oct 2010:
https://github.com/mono/mono/commit/0240df34432f52483cbdac42f5acd26042d6d012

Is it intentional to refer the NSS 3.12 certdata.txt forever?
(In reply to greg.marr from comment #7)
> Is there an alternate URL where we can redirect mozroots until mxr is
> available again?
> http://mxr.mozilla.org/seamonkey/source/security/nss/lib/ckfw/builtins/
> certdata.txt?raw=1

Many things appear to be pulling files directly from MXR when they should not. If anything, they should pull from the source code repository, not from a code indexing site. 

The future of MXR is extremely bleak, to be honest. I expect that all efforts next week will be getting trees indexed on DXR.


(In reply to :Gijs Kruitbosch from comment #6)
> Can we restore service of mxr.mozilla.org/addons separately? That repo is
> private and not accessible (and so not rewritable) to DXR). It being
> unavailable is hurting our ability to check for add-on compat issues etc...

No, but I will make having addons available on DXR a priority.
Flags: needinfo?(klibby)
There are a scary number of people that are pulling certdata.txt from MXR, in many cases over HTTP (!!) - 60,000 code hits on GitHub alone:
https://github.com/search?utf8=%E2%9C%93&q=%22mxr.mozilla.org%22+certdata.txt&type=Code&ref=searchresults

And similarly, 2000 hits for effective_tld_names.dat:
https://github.com/search?utf8=%E2%9C%93&q=%22mxr.mozilla.org%22+effective_tld_names.dat&type=Code&ref=searchresults

Plus a handful of other files:
https://github.com/search?p=3&q=%22mxr.mozilla.org%22+raw&ref=searchresults&type=Code&utf8=%E2%9C%93
(including a VIM plugin that uses a testing web-server pulled from mozilla-central over HTTP for actual functionality)

If we do set up redirects for old MXR links I *really really* think we should explicitly break any links that use `?raw=1`, so people can't silently unsafely use these resources over HTTP. (Whilst DXR is HTTPS-only and sets HSTS headers, the initial redirect can be MITMed).

For anyone visiting this bug after breakage - the recommended location to pull the Public Suffix List from is:
https://publicsuffix.org/list/
I found that mozroots 3.2.8 was unable to pull the hg.mozilla.org version over https, and also unable to process it using --file when I used curl to download it.
I downloaded the mxr version from archive.org, processed that using --file, and was then able to pull the hg version using --url.
Works for me? (You are using the raw file link, yeah?)

[~/src]$ curl -I https://hg.mozilla.org/projects/nss/raw-file/tip/lib/ckfw/builtins/certdata.txt
HTTP/1.1 200 Script output follows
Server: Apache
Vary: Accept-Encoding
Content-Type: text/plain; charset="UTF-8"
Date: Fri, 17 Jun 2016 13:35:05 GMT
Access-Control-Allow-Origin: *
ETag: 1466168830
Content-Disposition: inline; filename="certdata.txt"
Connection: Keep-Alive
Content-Length: 1633400

That said, I'd recommend writing a script to pull this file into your repo periodically, rather than having a third party site as an external dependency to your project. hg.mozilla.org really shouldn't be acting as a CDN for your users.

Anyway, sorry for more noise in the bug :-)
Summary: mxr outage → mxr.mozilla.org outage
Ed, this isn't going in a public repo or to users.  It's building a docker image for a mono service deployment.
Ah - is this different from the mono-related mozroots tool? (Which is being run by users)
eg: https://github.com/boogie-org/boogie/issues/37
It's the same.  In this case, we are the users of mono.  We're using mono to create a service,
and our service needs to access https resources.  Our users don't use mozroots.
We're also fixed to a particular version of mono at this point for various reasons, so we can't
upgrade to a later version, which presumably uses the newer URL.
I'm confused about whether there is any intention to actually restore mxr based on the comments here. Is the expectation that this will be restored, or has it been taken offline for good?
(In reply to Kent James (:rkent) from comment #18)

We'll be sending out a update about mxr by mid-week.
FYI, DXR doesn't provide access to the l10n repos. If mxr isn't going to be back soon, we'll need those on dxr.
(In reply to Mike Kaply [:mkaply] from comment #20)
> FYI, DXR doesn't provide access to the l10n repos. If mxr isn't going to be
> back soon, we'll need those on dxr.

I've filed bug 1281218 and made it block the "add missing trees to DXR" bug 820531, which itself blocks the "make MXR retireable" bug 1097091 tracker. If there are any other bugs, please add :-)
:arr OK but you did not answer the question. I can only assume that you are still deciding whether to keep mxr or not, and that will be answered in the update this week.

:emorley is the expectation that we file bugs for all missing tree on dxr that were on mxr?
(In reply to Kent James (:rkent) from comment #22)
> :emorley is the expectation that we file bugs for all missing tree on dxr
> that were on mxr?

Even before this outage, the long term plan from what I can tell (I have nothing to do with MXR/DXR) was always to bring DXR up to feature parity with MXR and then switch MXR off. Regardless of the outcome for this bug, it seems prudent to make sure any missing features (or repos) have bugs filed so the DXR team can plan their roadmap. If it isn't reported, the DXR team will have no idea that they need to implement it...
See Also: → 1281389
Please can we avoid doing redirects to https://hardhat.cdn.mozilla.net/ when there are issues on any site so that session history isn't lost for any tabs which were pointing to the site. A rewrite/proxy rule can preserve the URL in the address bar while still showing the hardhat site. Having the URLs return a iframe pointing to https://hardhat.cdn.mozilla.net/… would also work. Doing what I suggest would also mean we would be returning a more appropriate HTTP error code (e.g. 503) instead of 302.
(In reply to Matthew N. [:MattN] (behind on reviews) from comment #24)
> Please can we avoid doing redirects to https://hardhat.cdn.mozilla.net/ when
> there are issues on any site so that session history isn't lost for any tabs
> which were pointing to the site. A rewrite/proxy rule can preserve the URL
> in the address bar while still showing the hardhat site. Having the URLs
> return a iframe pointing to https://hardhat.cdn.mozilla.net/… would also
> work. Doing what I suggest would also mean we would be returning a more
> appropriate HTTP error code (e.g. 503) instead of 302.

Sorry, that's out of my hands; filed bug 1281567 with WebOps for it, though!
Depends on: 1279877
Based on comment 26, turning this into a bug to yank monitoring for anything related to MXR with the OK from Kendall.

We will keep in mind of any inquires related to comment 0 for mxr.m.o -> dxr.m.o
Flags: needinfo?(klibby)
(In reply to Ryan C [:ryanc] from comment #27)
> Based on comment 26, turning this into a bug to yank monitoring for anything
> related to MXR with the OK from Kendall.
> 
> We will keep in mind of any inquires related to comment 0 for mxr.m.o ->
> dxr.m.o

Nearly all of the monitoring, except for system-level checks, has been removed, I think; go ahead and clean up the rest of it.
Flags: needinfo?(klibby)
Removed remaining checks in sysadmins r119454.
Assignee: nobody → rchilds
Status: NEW → RESOLVED
Closed: 4 years ago
Component: MOC: Incidents → MOC: Service Requests
Resolution: --- → FIXED
Summary: mxr.mozilla.org outage → Remove checks related to mxr.mozilla.org
You need to log in before you can comment on or make changes to this bug.