Rollout a Normandy recipe that turns off (pref'able) TLS 1.3 until Avast rolls out a fix for Bug 1468892

VERIFIED FIXED

Status

--
major
VERIFIED FIXED
5 months ago
3 months ago

People

(Reporter: ritu, Assigned: mythmon)

Tracking

trunk

Firefox Tracking Flags

(firefox61blocking verified)

Details

(Reporter)

Description

5 months ago
Can we use Normandy to push a TSL 1.3 pref off recipe on Firefox clients that has Avast installed? This is a potential mitigation while we wait for Avast to rollout a fix for Bug 1468892.
(Reporter)

Comment 1

5 months ago
Hi Mike, Rob, Ryan, I just filed this bug to explore using Normandy for this critical issue since 61 rollout. Please note if Avast is able to update their clients super fast we may not need to use Normandy. See https://bugzilla.mozilla.org/show_bug.cgi?id=1468892#c40
Flags: needinfo?(ryanvm)
Flags: needinfo?(rhelmer)
Flags: needinfo?(mcooper)
(Reporter)

Updated

5 months ago
Assignee: nobody → nobody
Component: Normandy Server → Libraries
Product: Firefox → NSS
Version: 61 Branch → trunk
(Assignee)

Comment 2

5 months ago
In Windows 8 and above, I believe there should be a telemetry item along the lines of `environment.system.sec.antivirus`. If we can verify that the value of that item correlates well with a user having Avast installed, then we can target it and use Normandy to change the TLS 1.3 pref temporarily.

I don't have a Windows machine with Avast handy to verify the above. I can work on that today, but perhaps someone already has a setup we can test.
Flags: needinfo?(mcooper)
I checked with the SV team in Vegas, but they don't have any machines handy with Avast/AVG. Andrei, can your team provide the info?
Flags: needinfo?(ryanvm) → needinfo?(andrei.vaida)
Flags: needinfo?(rhelmer)
See Also: → bug 1468892
(Reporter)

Comment 4

5 months ago
Hi Philipp, can we get release/SUMO users who are running into this problem (bug 1468892) to share the "security software" section of their about:support? We need to reliably know what strings to look for as an indication of an Avast-enabled Firefox client. Thanks!
Flags: needinfo?(madperson)

Comment 5

5 months ago
it's too late for me today to start reaching out to users. i myself see the following strings in about:support when test running the security software:

Avast:

Security Software
Type 	Name
Antivirus 	Avast Antivirus
Antispyware 	Avast Antivirus
Firewall 	Windows Firewall

===

AVG:

Security Software
Type 	Name
Antivirus 	AVG Antivirus
Antispyware 	AVG Antivirus
Firewall 	Windows Firewall

this doesn't work under windows 7 though, where the whole section isn't present - see the implementation details in bug 1418131
Flags: needinfo?(madperson)
(Assignee)

Comment 6

5 months ago
I was able to get a Windows 10 VM running Avast, and confirmed that

a) Changing security.tls.version.max via Normandy fixes the problem.
b) Avast can be targeted by Normandy via Telemetry.

I used the filter expression

    'Avast Antivirus' in normandy.telemetry.main.environment.system.sec.antivirus

Which could be easily modified to work for both Avast and AVG Antivirus, if that's desirable.
Flags: needinfo?(andrei.vaida)

Comment 7

5 months ago
Indeed, Windows 7 doesn't provide the needed information:

	Application Basics
	------------------

	Name: Firefox
	Version: 61.0
	Build ID: 20180621125625
	Update Channel: release
	User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0
	OS: Windows_NT 6.1


	Security Software
	----------------- Type:

"Raw" version of "Security Software" section:

	"securitySoftware": {
	    "registeredAntiVirus": "",
	    "registeredAntiSpyware": "",
	    "registeredFirewall": ""
	  },
(Reporter)

Comment 8

5 months ago
Hi Mike, based on comment 5, we should also look for "AVG Antivirus" in addition to "Avast Antivirus". Ryan, can correct me if I am wrong but we have issues with both of those security software and TLS 1.3. Thanks!
Flags: needinfo?(ryanvm)
Flags: needinfo?(mcooper)
(Reporter)

Updated

5 months ago
status-firefox61: --- → affected
tracking-firefox61: --- → blocking
From an email thread with mythmon:
> I've updated my stage recipe to target both users with Avast and users
> with Windows versions <8 (that is, Win7 users). I don't have a Windows 7
> box to test this on, so I'm not 100% sure this approach works. Hopefully
> QA or someone else with Windows 7 can test it.
> 
> The recipe, for QA and others with access to Normandy, is
> https://normandy-admin.stage.mozaws.net/recipe/508/approval_history/
> 
> The data in the API for those without VPN access is
> https://normandy.cdn.mozilla.net/api/v1/recipe/508/

I'm submitting a PI request to cover testing of this recipe. As Ritu noted, we need to make sure it covers both Avast *and* AVG. So basically if Fx61 && (Win7 || Avast || AVG), set security.tls.version.max to 3. Otherwise, do nothing.

We will continue to monitor Avast's rollout of an updated version so that we can turn this recipe off as soon as is feasible.
Flags: needinfo?(ryanvm)
Make that https://normandy.stage.mozaws.net/api/v1/recipe/508/ for the recipe on stage.  It does not include AVG right now as far as I can tell.
(In reply to Ryan VanderMeulen [:RyanVM] from comment #3)
> I checked with the SV team in Vegas, but they don't have any machines handy
> with Avast/AVG. Andrei, can your team provide the info?

We have test machines ready for this.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #9)
> I'm submitting a PI request to cover testing of this recipe. As Ritu noted,
> we need to make sure it covers both Avast *and* AVG. So basically if Fx61 &&
> (Win7 || Avast || AVG), set security.tls.version.max to 3. Otherwise, do
> nothing.
> 
> We will continue to monitor Avast's rollout of an updated version so that we
> can turn this recipe off as soon as is feasible.

Thank you for filing the request, we created a test plan [1] and started testing. We'll post our results here, and on the associated PI email thread as soon as we're done.


[1]  https://public.etherpad-mozilla.org/p/pi-request_bug1471672
As tests using https://normandy.stage.mozaws.net/api/v1/recipe/508/ fail to enroll, I've modified the staging recipe as follows:
Changed 'Avast Antivirus' to "Avast Antivirus" 
Fixed a typo in the second part of the filter "telementry" to "telemetry".

After the above changes, the targeting seems to be good.

One note related to using the telemetry env. matches is that in the cases of first run, the telemetry env. instantiates later than Normandy runs, which will mean a miss match in this particular case. Ofc, after 4h, when normandy runs again, the telemetry filters will match and recipe executed. This translates that first time users/new profiles on Win7 for example will have for ~4 hrs the tls 1.3, before the recipe is executed.
Replaced this filter section (which targets windows 7)
(
                normandy.telemetry.main.environment.system.os.name == "Windows_NT"
                && normandy.telemetry.main.environment.system.os.version[0] != "1"
                && normandy.telemetry.main.environment.system.os.version < "8"
               )

with:

normandy.telemetry.main.environment.system.os.version == 6.1


and added filter for AVG.
So I *think* the filter we want is

(
     "AVG Antivirus" in normandy.telemetry.main.environment.system.sec.antivirus
  || "Avast Antivirus" in normandy.telemetry.main.environment.system.sec.antivirus
  || (normandy.telemetry.main.environment.system.os.name == "Windows_NT"
      && normandy.telemetry.main.environment.system.os.version == "6.1")
)

i.e. add back the system.os.name check and add quotes around 6.1.
done, currently the filters for staging server recipe for "TLS 1.3 Avast rollback test" are:

(
     "AVG Antivirus" in normandy.telemetry.main.environment.system.sec.antivirus
     ||  "Avast Antivirus" in normandy.telemetry.main.environment.system.sec.antivirus
     ||    
      (        normandy.telemetry.main.environment.system.os.version == "6.1" 
               &&  normandy.telemetry.main.environment.system.os.name == "Windows_NT" )
)


FYI: In my Cu.import tests, both 6.1 and "6.1" return the correct results :)
(Assignee)

Comment 16

5 months ago
I've tweaked the filter slightly, and added the recipe on prod in a disabled state. The filter is:

(
  "AVG Antivirus" in normandy.telemetry.main.environment.system.sec.antivirus
  || "Avast Antivirus" in normandy.telemetry.main.environment.system.sec.antivirus
  || (
    normandy.telemetry.main.environment.system.os.name == "Windows_NT" 
    && normandy.telemetry.main.environment.system.os.version == "6.1" 
  )
  || (normandy.isFirstRun || !normandy.telemetry.main.environment)
)

To summarize the above, in Ekr's and my words:

1. If you are upgrading, then we disable TLS 1.3 if you are running Avast.
2. If you are a new install, we disable TLS 1.3 unconditionally and then reenable it in 6 hours if you are not running Avast.
3. If you are on  Windows 7, then we disable TLS 1.3 in all cases, since we can't be sure you have Avast or not.

Can we get a formal sign off in this bug from relman? Then we should be ready to go
Flags: needinfo?(mcooper)
QA has tested this and signed off and the deployment plan in comment 16 looks good to me. Signing off for RelMan.
(Assignee)

Comment 18

5 months ago
This recipe is now live on Normandy to 100% of 61, given the above filtering rules. We'll monitor enrollment and the effect via Telemetry over the next few days.
(Assignee)

Updated

5 months ago
Status: NEW → RESOLVED
Last Resolved: 5 months ago
Resolution: --- → FIXED
Assignee: nobody → mcooper
status-firefox61: affected → fixed
Severity: normal → major
Status: RESOLVED → VERIFIED
status-firefox61: fixed → verified
You need to log in before you can comment on or make changes to this bug.