ACAO header not being served up on some MDN pages

RESOLVED FIXED

Status

RESOLVED FIXED
4 years ago
4 months ago

People

(Reporter: openjck, Assigned: cturra)

Tracking

Details

(Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/588] )

(Reporter)

Description

4 years ago
We enable ACAO across the site in our main htaccess file [1] but it appears the ACAO headers are only served on some pages.

Served up correctly:

> curl -I https://developer.mozilla.org/en-US/docs/Web/HTML
> HTTP/1.1 200 OK
> Server: Apache
> Vary: Cookie, Accept-Encoding
> X-Backend-Server: developer1.webapp.scl3.mozilla.com
> Content-Type: text/html; charset=utf-8
> Date: Thu, 24 Jul 2014 20:33:42 GMT
> X-kuma-revision: 643463
> Access-Control-Allow-Origin: *
> Accept-Ranges: bytes
> ETag: "c5e5763cc6a1429709485b923d999a1596c3decd"
> X-Frame-Options: DENY
> Last-Modified: Wed, 23 Jul 2014 15:30:38 GMT
> Content-Length: 55331
> Connection: Keep-Alive

Not served up:

> $ curl -I https://developer.mozilla.org/en-US/
> HTTP/1.1 200 OK
> Server: Apache
> Vary: Cookie, Accept-Encoding
> X-Backend-Server: developer3.webapp.scl3.mozilla.com
> Content-Type: text/html; charset=utf-8
> Date: Thu, 24 Jul 2014 19:50:58 GMT
> Accept-Ranges: bytes
> X-Frame-Options: DENY
> Content-Length: 39683
> Connection: Keep-Alive
> X-Cache-Info: cached

Not served up:

> curl -I https://developer.mozilla.org/en-US/demos/
> HTTP/1.1 200 OK
> Server: Apache
> Vary: Cookie, Accept-Encoding
> X-Backend-Server: developer1.webapp.scl3.mozilla.com
> Content-Type: text/html; charset=utf-8
> Date: Thu, 24 Jul 2014 20:34:38 GMT
> Accept-Ranges: bytes
> X-Frame-Options: DENY
> X-Cache-Info: caching

ACAO is also not served up on editing pages, like [2].

I don't want to rule out that something is mis-configured on our end, but I haven't found anything yet.

[1] https://github.com/mozilla/kuma/blob/master/configs/htaccess#L78-81
[2] https://developer.mozilla.org/en-US/docs/Web/HTML$edit

Updated

4 years ago
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/588]
(Assignee)

Comment 1

4 years ago
i wonder if the header you're seeing is in fact coming from the work done in bug 881315 and not your htaccess file. if you'd like, we can merge these headers into the root apache config?

  https://github.com/mozilla/kuma/blob/master/configs/htaccess#L78-81
Flags: needinfo?(jkarahalis)
(Assignee)

Comment 2

4 years ago
grabbing this bug so it doesn't page.
Assignee: server-ops-webops → cturra
(Assignee)

Comment 3

4 years ago
*dropping priority while waiting on a response*
Severity: major → normal
(Reporter)

Comment 4

4 years ago
Sounds great. We also want to be sure that the /media/fonts/.htaccess ACAO directive takes precedence for files in that directory.

Any idea why the ACAO directive in our source-controlled root .htaccess is not having the intended effect?
Flags: needinfo?(jkarahalis)
(Assignee)

Comment 5

4 years ago
(In reply to John Karahalis [:openjck] from comment #4)
(In reply to John Karahalis [:openjck] from comment #4)
> Sounds great. We also want to be sure that the /media/fonts/.htaccess ACAO
> directive takes precedence for files in that directory.
> 
> Any idea why the ACAO directive in our source-controlled root .htaccess is
> not having the intended effect?

this cannot be done via an .htaccess file because the requests are being served by mod_wsgi, which you cannot provide directives to via htaccess.

i can move these rules into this vhosts apache config tho, which will ensure they're served correctly via mod_wsgi.
(Reporter)

Comment 6

4 years ago
Sounds perfect. Thanks, cturra!
(Assignee)

Comment 7

4 years ago
i have applied this change on stage. can you please give it a test and let me know how things look before we push this to prod?

$ svn diff
svn ci Index: modules/webapp/files/developer-stage/etc-httpd/domains/developer.allizom.org.conf
===================================================================
--- modules/webapp/files/developer-stage/etc-httpd/domains/developer.allizom.org.conf	(revision 90985)
+++ modules/webapp/files/developer-stage/etc-httpd/domains/developer.allizom.org.conf	(working copy)
@@ -54,6 +54,11 @@
     WSGIProcessGroup developer_stage
     WSGIPassAuthorization On

+    # bug 1043604
+    Header set Access-Control-Allow-Origin "*" env=CORS
+    Header set Access-Control-Allow-Methods "GET" env=CORS
+    Header set Access-Control-Allow-Credentials "false" env=CORS
+
Flags: needinfo?(jkarahalis)
(Reporter)

Comment 8

4 years ago
Hm, doesn't appear to be showing up on stage pages.

> $ curl -I https://developer.allizom.org/en-US/
> HTTP/1.1 200 OK
> Server: Apache
> Vary: Cookie, Accept-Encoding
> X-Backend-Server: developer1.stage.webapp.scl3.mozilla.com
> Content-Type: text/html; charset=utf-8
> Date: Tue, 29 Jul 2014 16:14:14 GMT
> Accept-Ranges: bytes
> X-Frame-Options: DENY
> Content-Length: 39453
> Connection: Keep-Alive
> X-Cache-Info: cached

> $ curl -I https://developer.allizom.org/en-US/demos/
> HTTP/1.1 200 OK
> Server: Apache
> Vary: Cookie, Accept-Encoding
> X-Backend-Server: developer1.stage.webapp.scl3.mozilla.com
> Content-Type: text/html; charset=utf-8
> Date: Tue, 29 Jul 2014 16:14:49 GMT
> Accept-Ranges: bytes
> X-Frame-Options: DENY
> Content-Length: 57479
> Connection: Keep-Alive
> X-Cache-Info: cached
Flags: needinfo?(jkarahalis)
(Assignee)

Comment 9

4 years ago
i suspected that would be the case based on the `env=CORS` the header set directives had. i went ahead and removed those and the headers are present as expected. are you happy with this?

$ curl -I https://developer.allizom.org/en-US/
HTTP/1.1 200 OK
Server: Apache
Vary: Cookie, Accept-Encoding
X-Backend-Server: developer1.stage.webapp.scl3.mozilla.com
Content-Type: text/html; charset=utf-8
Access-Control-Allow-Credentials: false
Date: Tue, 29 Jul 2014 16:55:57 GMT
Access-Control-Allow-Origin: *
Accept-Ranges: bytes
X-Frame-Options: DENY
Access-Control-Allow-Methods: GET
Content-Length: 39544
Connection: Keep-Alive
X-Cache-Info: cached

$ curl -I https://developer.allizom.org/en-US/demos/
HTTP/1.1 200 OK
Server: Apache
Vary: Cookie, Accept-Encoding
X-Backend-Server: developer1.stage.webapp.scl3.mozilla.com
Content-Type: text/html; charset=utf-8
Access-Control-Allow-Credentials: false
Date: Tue, 29 Jul 2014 16:54:56 GMT
Access-Control-Allow-Origin: *
Accept-Ranges: bytes
X-Frame-Options: DENY
Access-Control-Allow-Methods: GET
Content-Length: 57570
Connection: Keep-Alive
X-Cache-Info: cached
Flags: needinfo?(jkarahalis)
(Reporter)

Comment 10

4 years ago
Looking good. Just a couple of things:

1. I notice that fonts are also being served up with the header. The .htaccess
   file in our fonts directory [1] limits CORS to Mozilla domains, but I guess
   that doesn't help much if ACAO directives in htaccess files are ignored. Is
   this something that also needs to be done in the root apache config?

> $ curl -I https://developer.allizom.org/media/fonts/BebasNeue-webfont.eot
> HTTP/1.1 200 OK
> Server: Apache
> X-Backend-Server: developer1.stage.webapp.scl3.mozilla.com
> Content-Type: application/vnd.ms-fontobject
> Access-Control-Allow-Credentials: false
> Date: Tue, 29 Jul 2014 20:41:28 GMT
> Accept-Ranges: bytes
> Access-Control-Allow-Origin: *
> ETag: "6390"
> Last-Modified: Thu, 21 Jun 2012 20:46:45 GMT
> Access-Control-Allow-Methods: GET
> X-Cache-Info: caching
> Content-Length: 25488

2. ACAO headers appear to be served correctly on edit pages (like [2]). Upon
   saving the page we do an Optimizely trackEvent, but it appears this is
   being blocked in Firefox with the error "Cross-Origin Request Blocked".
   I thought the ACAO header would have resolved this, but maybe it's something
   else?

[1] https://github.com/mozilla/kuma/blob/master/media/fonts/.htaccess
[2] https://developer.allizom.org/en-US/docs/Web/CSS/CSS3$edit
Flags: needinfo?(jkarahalis)
(Assignee)

Comment 11

4 years ago
this got me thinking - clearly we don't want to be the blockers for you being able to apply apache configs on your environments. what we've done for other projects, like bedrock, is include an apache config that's managed in the projects repo. here is an example of some bedrock uses:

  https://github.com/mozilla/bedrock/tree/master/etc/httpd


i'd like to propose developer does the same. this will give you full control of these sorts of changes. once you've created a config in your repo, let me know and i will include it from the base apache vhost config.
Flags: needinfo?(jkarahalis)
(Reporter)

Updated

4 years ago
Blocks: 949465
(Reporter)

Comment 12

4 years ago
Sounds good to me. I always appreciated this on Bedrock.

It sounds like this new, source-controlled file would replace the existing Apache root config. Can we just copy the existing configuration to source control, or are there reasons to avoid that?
Flags: needinfo?(jkarahalis)
(Assignee)

Comment 13

4 years ago
we would still control the "core" apache vhost configs, but anything app specific (after mod_wsgi stuff) would be imported to that config. this is exactly the same way it's done in bedrock.

just let me know when you've got a file (even a dummy one) created in your repo and i will get dev pointed to it for you.
(Reporter)

Comment 14

4 years ago
Looks like we have an Apache configuration file hooked up, after all. Would the directives work if placed here?

https://github.com/mozilla/kuma/blob/master/puppet/files/etc/apache2/conf.d/mozilla-kuma-apache.conf
(Assignee)

Comment 15

4 years ago
^ that file looks like it was created for local dev testing by Les. i would check with him if you can take that over.

the file will have to change since the <virtualhost> directives will continued to be managed by us. just app specific directives, like redirects and headers will be in your control. your config file will resemble:

  https://github.com/mozilla/bedrock/blob/master/etc/httpd/global.conf
(Assignee)

Comment 16

4 years ago
just a ping on this.
Flags: needinfo?(jkarahalis)
(Reporter)

Comment 17

4 years ago
Jannis and Luke had some concerns about our team managing the configuration. Jannis, Luke: Are you able to say more?
Flags: needinfo?(lcrouch)
Flags: needinfo?(jkarahalis)
Flags: needinfo?(jezdez)
Sure, :openjck.

I simply don't see anyone inside the MDN dev team with enough Apache configuration knowledge to take the responsibility of maintenance. In that sense I'm not in favor for putting this on our plate but find a different way to improve the way we handle updates to that file.

How about a separate repository (e.g. mozilla/mdn-deploy) to store the config files that we'd use for collaborative code reviews together with :cturra and team? We'd still open Bugzilla tickets for new additions to the config file, but instead being blocked by a busy schedule on webops' side we could attach a pull request to that ticket with what we think may work. Then in the code review you can help us understand Apache configuration better. That way we find out if that's something we can handle on the long run.
Flags: needinfo?(jezdez)
(Assignee)

Comment 19

4 years ago
that sounds fair. tho, i want to clarify, i am not proposing you take over the entire apache config, but rather just the bits like these headers. this aligns with the way we've allowed bedrock (www) to manage their own redirects.
(Assignee)

Comment 20

4 years ago
any word on the direction we want to go with this?
Flags: needinfo?(jkarahalis)
(Reporter)

Comment 21

4 years ago
Hi Chris,

Our team has a work week starting Monday. I'm going to flag down a couple of people to discuss before we move forward. I'll know more by the end of next week. Thanks for following up.
Flags: needinfo?(jkarahalis)
Flags: needinfo?(lcrouch)
(Reporter)

Updated

4 years ago
No longer blocks: 949465
Depends on: 949465
(Reporter)

Updated

4 years ago
Blocks: 949465
No longer depends on: 949465
(Reporter)

Comment 22

4 years ago
We have a pull request open for adding a global Apache configuration file.

https://github.com/mozilla/kuma/pull/2774

Once that lands, I'll update this bug so that we can start hooking it up. Thanks so much for your help.

Comment 23

4 years ago
Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/0e6cb58f1c4ccb5ab26112ab3ed0a0c71b3d890b
Bug 1043604: Add global Apache configuration

A new file is added, all-servers.conf, which contains Apache settings
that should be used on all servers. The file is hooked up to our local
environment via Puppet and the WebOps team will ensure that it's also
used on our public servers.

https://github.com/mozilla/kuma/commit/26d32fca727a9682078f6a77b3b17a58c538bad7
Merge pull request #2774 from openjck/1043604-root-apache-config

Bug 1043604: Add global Apache configuration
I wasn't around when this was discussed, :openjck can you clarify who is going to maintain that of the MDN dev team?
(Reporter)

Comment 25

4 years ago
(In reply to Jannis Leidel [:jezdez] from comment #24)
> I wasn't around when this was discussed, :openjck can you clarify who is
> going to maintain that of the MDN dev team?

I can help maintain it. I don't expect that we will need to do much.

Like Chris was saying in comment 19, we won't be maintaining the whole Apache config. Just some specific bits that can't live in .htaccess, like these ACAO headers.
(Reporter)

Comment 26

4 years ago
Hi Chris,

We added the configuration file to our repository and pushed it to staging and production.

https://github.com/mozilla/kuma/blob/master/etc/apache/all-servers.conf
Flags: needinfo?(cturra)
(Assignee)

Comment 27

4 years ago
i have updated the apache configs in both dev and stage to include your all-server.conf. let me know when your testing is complete and you want this pushed out to prod.
Flags: needinfo?(cturra) → needinfo?(jkarahalis)
(Reporter)

Comment 28

4 years ago
Looks good on stage. The ACAO header was wrong on stage, so I did a push to trigger an Apache restart. Now we're getting 500s from stage.
Flags: needinfo?(jkarahalis) → needinfo?(cturra)
(Reporter)

Comment 29

4 years ago
Looks good on dev*...
(Assignee)

Comment 30

4 years ago
(In reply to John Karahalis [:openjck] from comment #28)
> Looks good on stage. The ACAO header was wrong on stage, so I did a push to
> trigger an Apache restart. Now we're getting 500s from stage.

doh! it was a typo in the import directive of of the apache config. fixed now.
Flags: needinfo?(cturra)
(Reporter)

Comment 31

4 years ago
Looks great on stage! Ready to move forward with prod. Thanks for all your help on this, Chris.
(Assignee)

Comment 32

4 years ago
pushed to prod and i ran a quick configtest to be sure there weren't any typos this time!

 [root@developer1.webapp.scl3 ~]# apachectl configtest
 Syntax OK


looks like we can now mark this bug as r/fixed. nice work everyone \o/
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED

Updated

4 months ago
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.