1312673 - Better blocklist XML NGINX caching

Status: RESOLVED FIXED

(Cloud Services :: Operations: AMO, task)

Description

[Rémy Hubscher (:natim)]

We are having a lot of different hits like that [0]

However the kinto views is only using the first parameters to generate the XML file:

- /v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.13/
- /v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.10/

Would it be possible to configure the blocklists-settings proxy to call the Kinto server endpoint using an URL cleaned from useless informations so that we could be more efficient in caching those?

/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.13/Firefox/20101203075014/WINNT_x86-msvc/ru/release/Windows_NT%25205.1/default/default/
    →  /v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.13/

/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.10/Firefox/20100914125854/WINNT_x86-msvc/en-US/release/Windows_NT%25206.1/default/default/
    → /v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.10/

etc.

[0]
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.13/Firefox/20101203075014/WINNT_x86-msvc/ru/release/Windows_NT%25205.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.10/Firefox/20100914125854/WINNT_x86-msvc/en-US/release/Windows_NT%25206.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.5.17/Firefox/20110121150729/WINNT_x86-msvc/en-US/release/Windows_NT%25205.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6b4/Firefox/20091124213835/WINNT_x86-msvc/en-US/beta/Windows_NT%25206.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/38.1.1/Firefox/20150806001528/Linux_x86-gcc3/en-US/esr/Linux%25203.14.25-tinycore%252B%2520(GTK%25202.24.20)/default/default/1/2/1035/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.5.5/Firefox/20091102152451/WINNT_x86-msvc/es-ES/release/Windows_NT%25206.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.5.12/Iceweasel/20100927120848/Linux_x86_64-gcc3/chrome://global/locale/intl.properties/default/Linux%25202.6.32-5-amd64%2520(GTK%25202.12.12)/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.0.13/Firefox/2009073021/Darwin_Universal-gcc3/en-US/release/Darwin%25208.11.0/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/4.0b5/Firefox/20100831080731/WINNT_x86-msvc/en-US/beta/Windows_NT%25206.2/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/31.7.0/Firefox/20150504194141/WINNT_x86-msvc/en-US/esr/Windows_NT%25206.3/default/default/29/29/1/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/31.6.0.1/Firefox/20150414033101/WINNT_x86-msvc/en-US/default/Windows_NT%25206.3/SurfEasy/1.5.0/223/661/1/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/31.6.0.1/Firefox/20150414033101/WINNT_x86-msvc/en-US/default/Windows_NT%25206.1/SurfEasy/1.5.0/278/756/1/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/31.5.0/Firefox/20150217104802/WINNT_x86-msvc/ja/esr/Windows_NT%25206.1/default/default/363/407/1/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/31.4.0/Firefox/20150105205548/Darwin_x86_64-gcc3-u-i386-x86_64/en-GB/esr/Darwin%252015.3.0/default/default/1/1/new/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/31.3.0/Firefox/20141125031119/WINNT_x86-msvc/en-US/esr/Windows_NT%25206.1/default/default/1/1/483/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/31.1.0esr/Firefox/20141120173533/WINNT_x86-msvc/en-US/default/Windows_NT%25205.1/default/default/173/612/1/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.7a5pre/Firefox/20100514200938/WINNT_x86_64-msvc/en-US/default/Windows_NT%25206.0/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6b4/Firefox/20091124213835/WINNT_x86-msvc/en-US/beta/Windows_NT%25206.2/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6b1/Firefox/20091029151354/Darwin_Universal-gcc3/en-US/beta/Darwin%25209.8.0/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.7/Firefox/20100713121444/Darwin_Universal-gcc3/en-US/release/Darwin%252010.8.0/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.4/Firefox/20100611143157/WINNT_x86-msvc/en-US/release/Windows_NT%25205.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.28/Firefox/20120306064154/WINNT_x86-msvc/ja/release/Windows_NT%25206.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.28/Firefox/20120306064154/Darwin_Universal-gcc3/en-US/release/Darwin%25208.11.0/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.27/Firefox/20120216013254/WINNT_x86-msvc/de/release/Windows_NT%25205.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.20/Firefox/20110803131630/Darwin_Universal-gcc3/ko/release/Darwin%25208.11.0/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.18/Firefox/20110622065727/Linux_x86_64-gcc3/chrome://global/locale/intl.properties/default/Linux%25202.6.18-238.19.1.el5%2520(GTK%25202.10.4)/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.18/Firefox/20110614230723/WINNT_x86-msvc/en-US/release/Windows_NT%25206.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.18/Firefox/20110614230723/WINNT_x86-msvc/en-US/release/Windows_NT%25205.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.17/Firefox/20110420140830/WINNT_x86-msvc/ja/release/Windows_NT%25205.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.16/Firefox/20110323162424/Linux_x86-gcc3/chrome://global/locale/intl.properties/default/Linux%25202.6.35-25-generic%2520(GTK%25202.20.1)/canonical/1.0/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.16/Firefox/20110319135224/Linux_x86-gcc3/fr/release/Linux%25202.6.32.21-168.fc12.i686.PAE%2520(GTK%25202.18.3)/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.15/Firefox/20110303024726/WINNT_x86-msvc/id/release/Windows_NT%25206.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.15/Firefox/20110303024726/WINNT_x86-msvc/es-ES/release/Windows_NT%25205.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.13/Firefox/20101203074432/Darwin_Universal-gcc3/ja-JP-mac/release/Darwin%25208.11.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.12/Firefox/20101026210630/WINNT_x86-msvc/es-ES/release/Windows_NT%25205.1/default/default/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.11pre/Firefox/20100922090109/Linux_x86-gcc3/chrome://global/locale/intl.properties/default/Linux%25202.6.28-18-generic%2520(GTK%25202.16.1)/canonical/1.0/
/v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.11/Firefox/20101012113537/WINNT_x86-msvc/pt-BR/release/Windows_NT%25206.1/default/default/

Comment 1

[Jason Thomas [:jason]]

https://github.com/mozilla-services/cloudops-deployment/pull/212 and deployed to stage. Initial testing looks good. I would like to let it sit on stage for a few days and verify the requests that are coming into the kinto service before deploying this to production.

2016/10/26 21:35:16 [notice] 5358#0: *14 "^/v1/blocklist/([0-9]+)/([^/]+)/([^/]+)/.*" matches "/v1/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/3.6.13/Firefox/20101203075014/WINNT_x86-msvc/ru/release/Windows_NT%205.1/default/default/", client: 127.0.0.1, server: blocklists-settings.stage.mozaws.net, request: "GET /v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.13/Firefox/20101203075014/WINNT_x86-msvc/ru/release/Windows_NT%25205.1/default/default/ HTTP/1.1", host: "blocklists-settings.stage.mozaws.net"
2016/10/26 21:35:16 [notice] 5358#0: *14 rewritten data: "/v1/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/3.6.13/", args: "", client: 127.0.0.1, server: blocklists-settings.stage.mozaws.net, request: "GET /v1/blocklist/3/%257Bec8030f7-c20a-464f-9b0e-13a3a9e97384%257D/3.6.13/Firefox/20101203075014/WINNT_x86-msvc/ru/release/Windows_NT%25205.1/default/default/ HTTP/1.1", host: "blocklists-settings.stage.mozaws.net"

Comment 2

[Jason Thomas [:jason]]

These changes have been deployed to prod. I've verified kinto prod app logs that requests are now coming through without the additional parameters. Sample request from logs 'GET /v1/blocklist/3/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/47.0a2/'