Closed Bug 1529943 Opened 5 years ago Closed 2 years ago

Add a policy for Fx update pinning

Categories

(Firefox :: Enterprise Policies, enhancement, P3)

Product:
Firefox
Component:
Enterprise Policies
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: mkaply, Assigned: bytesized)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

Attachments

(1 file)

updates.png
104.14 KB, image/png
Details

Google provides policies for update separate from their regular policies:

https://support.google.com/chrome/a/answer/6350036?hl=en

A number of these are quite interesting and we should look at them, but the one of interest right now is pinning updates (Windows only).

I'd like to add this policy for Firefox. The goal is that enterprises can test a version of Firefox before updating everyone.

This is especially important with our new downgrade protection which will prevent enterprises from downgrading Firefox if they run into a problem.

I was just thinking about this, and I can foresee a problem with implementing this.

I'm not 100% sure how this policy is expected to work and be used, but I imagine that the usage will look something like this:

  1. Network administrator sets policy "update-pin: 50" and installs Firefox 50 on all machines.
  2. Firefox 51 is released by Mozilla, but is not downloaded by installations due to update pinning.
  3. Company tests Firefox 51 to ensure compatibility
  4. Network administrator sets policy "update-pin: 51", allowing Firefox installations to update to version 51.

However, now consider this scenario:

  1. Network administrator sets policy "update-pin: 50" and installs Firefox 50 on all machines.
  2. Firefox 51 is released by Mozilla, but is not downloaded by installations due to update pinning.
  3. Company starts tests Firefox 51 to ensure compatibility
  4. Firefox 52 is released and is also not downloaded by installations.
  5. Company finishes compatibility testing of Firefox 51.
  6. Network administrator sets policy "update-pin: 51".
  7. Firefox installations start up and, as usual, check for updates. They receive an update XML containing download links for the update to Firefox 52.
  8. Updates are not installed because the only advertised update is 52, but the installations are pined to 51.
  9. This situation potentially repeats forever. Updates are never installed. Installations remain at Firefox 50 forever.

There is a solution to this. I believe that we would need to make changes to the update server, Balrog. We would need to add a parameter to the update URL specifying the latest acceptable update version. This would allow Firefox to request an update to the pinned version and update directly to it.

There is a solution to this. I believe that we would need to make changes to the update server, Balrog. We would need to add a parameter to the update URL specifying the latest acceptable update version. This would allow Firefox to request an update to the pinned version and update directly to it.

Totally agree.

I bet Callek has an opinion here.

Callek asked me to comment on this wearing Balrog maintainer hat. A policy which could specify update-pin:60.* to stay on ESR60 might be quite workable, and fit with the idea that organisations only go through the big QA cycles for ESR branches. Balrog already offers the right release until we hit the X.2.0 release of a new ESR, but after that we'd need something new.

It looks like we support policies for regular releases too, and Kirk's scenario would be much harder to deal with. Balrog doesn't keep track of the latest 65.0 release, or the sequence of releases that we've offered which could be walked down until Balrog or Firefox finds the first one OK by the policy. Adding that kind of manifest approach would negate the ability to set rules to handle special situations (OS deprecation, issues with security suites etc). So my initial reaction isn't positive but I'll think on it some more, and would welcome more info on common use cases and background on why this would be useful.

From the policy point of view, whenever this policy is on and there's a newer update pending, I'd support displaying a message somewhere (in the About dialog or about:preferences), something along the lines of "A newer version of Firefox is available but your organization has not approved it yet".

This would probably ease some of the concerns about the case that Kirk mentioned, at least making people aware if they are stuck because of that.

And probably would be nice if this policy is explicitly designed and documented to not block dot releases

Priority: -- → P3

Are there any news on this one? Are we still intending to implement this policy? There is definitely a need for this.

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #3)

Callek asked me to comment on this wearing Balrog maintainer hat. A policy which could specify update-pin:60.* to stay on ESR60 might be quite workable, and fit with the idea that organisations only go through the big QA cycles for ESR branches.

Having an update-pin:78.* policy would be extremely useful for Firefox / Thunderbird in enterprise contexts!

See Also: → 1643271

Yes, implementing a policy for updates would make Enterprise's update process much more effective (see also: 1643271 - https://bugzilla.mozilla.org/show_bug.cgi?id=1643271)

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #3)

Callek asked me to comment on this wearing Balrog maintainer hat. A policy which could specify update-pin:60.* to stay on ESR60 might be quite workable, and fit with the idea that organisations only go through the big QA cycles for ESR branches. Balrog already offers the right release until we hit the X.2.0 release of a new ESR, but after that we'd need something new.

It looks like we support policies for regular releases too, and Kirk's scenario would be much harder to deal with. Balrog doesn't keep track of the latest 65.0 release, or the sequence of releases that we've offered which could be walked down until Balrog or Firefox finds the first one OK by the policy. Adding that kind of manifest approach would negate the ability to set rules to handle special situations (OS deprecation, issues with security suites etc). So my initial reaction isn't positive but I'll think on it some more, and would welcome more info on common use cases and background on why this would be useful.

I had a look and think about this myself, and everything Nick says here still holds IMO. It's not that difficult to figure how to find the authoritative version Release blob of eg: 65, 90, 105 -- but ensuring we still respect OS deprecation and other rules needs a lot of careful thought, and may not be simple or easy.

Attached image updates.png —

My naive thinking would be that Firefox already has all the information it needs to make the policy work.

When I set Firefox to ask before installing updates and then go to help, I can see the pending update. At this point, the pin-policy can decide, if the update should be installed or if it should be prevented.

So the update in the bottom scenario in the attached screenshot must be prevented, if there is a pin policy for 78.

I personally think the concerns in Comment #1 do not apply, as we are not talking about dot releases and probably also not about updates for regular releases, but for ESR jumps.

Let us consider a Company, which is using Firefox 68 ESR and has finished their testing for Firefox 78 ESR only after Firefox 91 ESR has been released. They change the pin policy to 78 and nothing happens. Their instances do not update. I would not consider this a failure of the pin policy. To overcome this, they can manually deploy Firefox 78 ESR.

This is even valid, if a company is indeed using the pin policy on regular releases.

The reason balrog is needed is for the middle cases.

So let's say you pinned to 91, and kept it pinned and in the meantime we updated to 94.

And then you want to change the pin to 92.

There's no way to tell our update server that you want an update to 92, you can only tell it that you want the latest Firefox.

So we need a way to tell balrog "Give me this version of Firefox, not the latest"

Is that how Google has implemented this policy? Then we probably should follow.

But one of the bugs closed as duplicate for this one aimed just at preventing upgrades beyond the tested range. For me for example the middle cases are not an issue. I can always deploy the version I want independent of the Firefox update server. In my scenario, I want to keep getting ESR updates automatically but not being upgraded to the next ESR.

So if the balrog based implementation turns out to be too complicated, a simple update stopper set via policy would be sufficient for some of us.

Summary: Add a policy for update pinning → Add a policy for Fx update pinning

trying to disambiguate from "tls certificate pinning", which also involves balrog

(In reply to John Bieling (:TbSync) from comment #12)

Let us consider a Company, which is using Firefox 68 ESR and has finished their testing for Firefox 78 ESR only after Firefox 91 ESR has been released. They change the pin policy to 78 and nothing happens. Their instances do not update. I would not consider this a failure of the pin policy. To overcome this, they can manually deploy Firefox 78 ESR.

This sounds dangerous to me. Consider this scenario:

  1. Pin policy updated to ESR78 while that is the newest version.
  2. A week later, ESR91 is released.

Most machines would update during that week. But some machines would be loaners or would belong to people on vacation and would not get updated. Now those machines are unable to update until the update pin is moved again.

In the best case, the enterprise is aware of this problem and has a mechanism for deploying the newest Firefox version to machines that need it. In this case, what did they gain from the pinning policy? What is the purpose of an update mechanism that requires that you also have your own, independent update mechanism? It seems to me that they would be served equally well by simply disabling update altogether.

In the worst case, the pinning policy effectively hides the problem from the enterprise. Some installations are out of date. Especially infrequently used machines may never be updated again.

The point is to remain on a tested ESR, still getting updates for that ESR, but not being updated to the next ESR. The Company is doing the update to the next ESR by global rollout, after that next ESR has been tested and approved.

The company would move the pin with the global rollout. They would not change the pin to get auto updated. The pin is used to pin down and lock onto the approved ESR.

I changed the wording of my last comment, to better express what I wanted to say. Sorry for that.

I guess I never directly responded to this.

(In reply to John Bieling (:TbSync) from comment #17)

The company would move the pin with the global rollout. They would not change the pin to get auto updated. The pin is used to pin down and lock onto the approved ESR.

I sort of already covered this in Comment 16. If you want to use a policy to prevent Firefox from getting updated, rather than to allow it to auto-update to a particular version, that policy already exists.

I've been spending some time looking into what will be necessary to implement this policy. Currently, it looks like there will be, roughly speaking, three pieces needed to make this work:

  1. Small change to Firefox - This would allow it to read the update pin policy and use it to add a query parameter to the update URL.
  2. Change to Balrog (the update server) - Balrog will probably need to do something like this: run through the existing rules to get the latest version that Firefox can update to. If that version is greater than the update pin, look up the newest release corresponding to the update pin.
  3. Change to release automation (I'm still figuring out this part) - In order for Balrog to be able to look up the newest release corresponding to a particular version, release automation will need to basically tell Balrog "this is the newest release of version X.Y" so that Balrog can update its version listing.
Assignee: nobody → bytesized
Depends on: 1762957
Depends on: 1762979
Depends on: 1762986
Depends on: 1770827
Depends on: 1772799

(In reply to Release mgmt bot [:suhaib / :marco/ :calixte] from comment #20)

The bug assignee is inactive on Bugzilla.

Odd, I really am not.

I'll take this as an opportunity to give a quick update though. The work for this has been done. But I was planning to keep this tracker bug open until the QA is finished. We want to be able to make sure that this works properly on Release and ESR channels. We want to run tests on Release 103 and 104, and ESR 102.1 and 102.2. Once that has completed, I'll close this bug.

Assignee: nobody → bytesized
Depends on: 1787125

I think we can mark this fixed, Kirk?

Yes, I agree.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: