Closed Bug 1636539 Opened 5 years ago Closed 4 years ago

Integrate check with remote settings for simple bool on/off switch for WDBA

Categories

(Toolkit :: Default Browser Agent, task, P1)

Product:
Toolkit
Component:
Default Browser Agent
Type:
task

Tracking

()

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox80 --- fixed

People

(Reporter: rachel, Assigned: nalexander)

References

(
URL
)

Details

(Whiteboard: [iu_tracking])

Attachments

(5 files, 2 obsolete files)

Bug 1636539 - Part 2: Add defaultagent-static Rust crate fetching remote disabled flag. r?agashlin,bytesized
47 bytes, text/x-phabricator-request
Details | Review
Bug 1636539 - Part 1: Mirror preferences for remote settings to Windows Default Browser Agent. r?agashlin,bytesized
47 bytes, text/x-phabricator-request
Details | Review
Bug 1636539 - Part 3: Use Rust remote settings client to fetch remote kill switch. r?agashlin,bytesized
47 bytes, text/x-phabricator-request
Details | Review
Bug 1636539 - Part 4: Note remote disablement in browser default agent docs. r?agashlin,bytesized
47 bytes, text/x-phabricator-request
Details | Review
Bug 1636539 - Part 0: Expose default agent's Windows event log name. r?agashlin
47 bytes, text/x-phabricator-request
Details | Review

Background

It's going to be increasingly important that we have a way to remotely disable WDBA as we start showing user facing notifications via 1619699.

Prefs as they work today within Firefox won't work here as the browser has to be open in order to read the updated value. Since WDBA is run outside of Firefox, it's possible for a user who never opens Firefox to never get the updated pref set into their profile, which effectively prevents us from using a pref read from FF to enable/disable the feature.

Requirements

  • on/off bool flag that can be passed from remote settings directly to the agent. Should be able to remotely enable/disable for all users.

Considerations

  • Should be built with the intention to eventually also use within the background update agent.
  • Should be built with the assumption that we'd eventually like to use it to enable scaled rollouts/experiments.

Notes

  • Details on working with remote settings here. - With a caveat from Benson that there might be things missing there (like REST API docs) so ping him or ask in #delivery as needed.

  • Here's a list of other Remote Settings Clients for reference as well.

See Also: → 1635507
Severity: -- → N/A
Priority: P2 → P1
See Also: wdba-phase2 →

This lays out a Rust crate and statically links it into
windows-default-agent.exe. It shows how to declare and invoke a
single-method API.

The API "fails open": if Remote Settings cannot be reached for any
reason -- network errors, data errors, or coding errors -- its assumed
that the agent has not been remotely disabled.

TODO:

  • Determine how much error propagation we care to do. Are we going to
    report errors to Telemetry, for example?

  • Keep a registry entry of the last disabled flag read from Remote
    Settings so that we can fallback to that value when we witness
    errors.

  • Finish up the Viaduct backend: support sending a body; add granular
    logging.

  • Consider disabling env_logger in non-debug configurations.

  • Consider mirroring Remote Settings configuration to the registry so
    that we can use prod/stage/dev easily.

  • Verify that code bloat in release/PGO/LTO configuratons isn't too extreme.

  • Write overview doc.

leplatrem: just FYI, the WIP commit above is what I meant by "Vaiduct around wininet.dll". You can see that I just fetch a record directly from Kinto, but y'all could build nice APIs that actually reflect Kinto's bucket/collection/record, and then add additional features around those.

Flags: needinfo?(mathieu)
Assignee: nobody → nalexander
Status: NEW → ASSIGNED

So more TODOs:

  • Make sure wininet flags are locked down: no cookies, no cache, etc.

  • Keep a timestamp/etag so that we get some 204s. This might not be worth the hassle, since then the caller has to maintain that data (in the registry, I guess).

The pattern with /mfbt/Poison.cpp and IMPL_MFBT is
well-established in the tree.

This isn't quite ready and tested yet, but I might as well throw it
out for initial comments. This originally mirrored the
services.settings.server pref. For WDBA, however, we only care
about one record; and it's quite handy to be able to name that
specific record (say, to one that disables things!). So eventually
I'll make this mirror some new pref with just that URL.

However, there's really no reason to do this in native code; it could
just as easily be JavaScript that runs a little after startup. That
would make it lighter to modify. But this location is next to the
same pattern for the launcher process. Trade offs, trade offs.

Do we have opinions about the form of the pref, and about where this
mirroring logic lives?

Depends on D80766

Attachment #9154689 - Attachment description: Bug 1636539 - WIP: Add defaultagent-static Rust crate that queries remote enabled flag. → Bug 1636539 - Part 3: Add defaultagent-static Rust crate fetching remote disabled flag. r?agashlin,bytesized

My try build is unhappy: https://treeherder.mozilla.org/#/jobs?repo=try&revision=dcbad051581a79971415d22d0419ec7046610552. The errors look minor but it'll have to wait until tomorrow for me to address.

I have a happy (debug) try build.

PLEASE SEE UPDATED TEST PLAN BELOW

For testing, I've exposed a debug command that simply runs the code and prints the output. I don't really know how to verify that the WDBA task is/is not run, since it's essentially silent. Perhaps we can witness Telemetry being sent? Whatever was used for the initial WDBA testing should be possible here. In any case, here's how I've tested this locally:

  • Install from try build. Elevated privileges should not matter, but I tested without them so please test with them. DO NOT RUN THE INSTALLED FIREFOX IMMEDIATELY

  • Open a command prompt. Again, elevated privileges should not matter. Navigate to the location of firefox.exe, and run the default browser agent debug commands:

  C:\Users\nalexander\AppData\Local\Firefox Nightly>default-browser-agent.exe debug-remote-disabled
  default-browser-agent: Last remote disabled: 0
  default-browser-agent: Remote service disabled state URL: 'https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173ffff'
  [ERROR defaultagent_static] Using remote settings URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173ffff
  [ERROR defaultagent_static] Error::NetworkError
  default-browser-agent: Error code: 0x7d01
  IsAgentRemoteDisabled: 0

Witness the URL, which should fallback to the production URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173ffff. Right now this will fail since we don't have the records in production, only in staging. But it's important to test that not having the setting in the registry doesn't crash.

  • Run the installed Firefox. This populates certain registry entries with settings. In the absence of those settings (the URL to fetch), we should fallback to the production URL, as verified above.

  • (Optional) Look in regedit under the key Computer\HKEY_CURRENT_USER\Software\Mozilla\Firefox\Default Browser Agent and witness entries with names like
    C:\Users\nalexander\AppData\Local\Firefox Nightly|DefaultAgentRemoteServicesStateURL with long URL defaulting to
    https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173ffff

  • In Firefox about:config, set
    default-browser-agent.remote_services_state_url to
    https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173ffff
    (Here, only the origin changes; the path remains the same.) This is a staging record that enables WDBA.

  • Open a command prompt. Again, elevated privileges should not matter. Navigate to the location of firefox.exe, and run the default browser agent debug commands:

  C:\Users\nalexander\AppData\Local\Firefox Nightly>default-browser-agent.exe debug-remote-disabled
  default-browser-agent: Last remote disabled: 0
  default-browser-agent: Remote service disabled state URL: 'https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173ffff'
  [ERROR defaultagent_static] Using remote settings URL: https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173ffff
  default-browser-agent: Error code: 0x0
  default-browser-agent: Next remote disabled: 0
  IsAgentRemoteDisabled: 0

Witness the configured URL. If the network is available, you should see IsAgentRemoteDisabled: 0.

  • (Optional) witness the last witnessed value (0) in regedit, like C:\Users\nalexander\AppData\Local\Firefox Nightly|DefaultAgentLastRemoteDisabled

  • Disable the network, or update the URL to something that will 404.

  • Run debug-remote-disabled again. Witness an error code but that we fall back to the last witnessed remote disabled value (here, 0, meaning "not disabled").

  C:\Users\nalexander\AppData\Local\Firefox Nightly>default-browser-agent.exe debug-remote-disabled
  default-browser-agent: Last remote disabled: 0
  default-browser-agent: Remote service disabled state URL: 'https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173fffg'
  [ERROR defaultagent_static] Using remote settings URL: https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173fffg
  [ERROR defaultagent_static] Error::NetworkError
  default-browser-agent: Error code: 0x7d01
  IsAgentRemoteDisabled: 0

  • If necessary, re-enable the network.

  • In Firefox about:config, then set
    default-browser-agent.remote_services_state_url to
    https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f0ff0ff0ff
    (Here, the origin is still stage but the path points to a known record that flips the kill switch to disabled.) THIS RECORD SHOULD EXIST TOMORROW -- it is waiting for Rachel Tublitz to "review" it so that it's published.

  • Run debug-remote-disabled again. Witness no error code and that we're remotely disabled (with a 1, meaning "yes disabled").

  C:\Users\nalexander\AppData\Local\Firefox Nightly>default-browser-agent.exe debug-remote-disabled
  default-browser-agent: Last remote disabled: 0
  default-browser-agent: Remote service disabled state URL: 'https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f0ff0ff0ff`
  [ERROR defaultagent_static] Using remote settings URL: https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f0ff0ff0ff
  default-browser-agent: Error code: 0x0
  default-browser-agent: Next remote disabled: 1
  IsAgentRemoteDisabled: 1

  • (Optional) witness the last witnessed value (1) in regedit, like C:\Users\nalexander\AppData\Local\Firefox Nightly|DefaultAgentLastRemoteDisabled

  • Disable the network, or update the URL to something that will 404.

  • Run debug-remote-disabled again. Witness an error code but that we fall back to the last witnessed remote disabled value (here, 1, meaning "yes disabled").

  C:\Users\nalexander\AppData\Local\Firefox Nightly>default-browser-agent.exe debug-remote-disabled
  default-browser-agent: Last remote disabled: 0
  default-browser-agent: Remote service disabled state URL: 'https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173fffg'
  [ERROR defaultagent_static] Using remote settings URL: https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173fffg
  [ERROR defaultagent_static] Error::NetworkError
  default-browser-agent: Error code: 0x7d01
  IsAgentRemoteDisabled: 1

  • If necessary, re-enable the network.

  • In Firefox about:config, set
    default-browser-agent.remote_services_state_url to the original "not disabled" record:
    https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173ffff

  • Run debug-remote-disabled again. Witness no error code and the correct "not disabled" value of 0:

  C:\Users\nalexander\AppData\Local\Firefox Nightly>default-browser-agent.exe debug-remote-disabled
  default-browser-agent: Last remote disabled: 0
  default-browser-agent: Remote service disabled state URL: 'https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173ffff'
  [ERROR defaultagent_static] Using remote settings URL: https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/ffffffff-0c5f-49a0-bb48-11f17173ffff
  default-browser-agent: Error code: 0x0
  default-browser-agent: Next remote disabled: 0
  IsAgentRemoteDisabled: 0

Those are the scenarios I have thought of and tested against.

Some small issues were found during testing, a new case was suggested, and the preferences for configuration details will change for the landed version.

Two issues

The two issues were:

  1. A typo in the production URL, v2 instead of v1.
  2. The observation that the registry entry for DefaultAgentLastRemoteDisabled wasn't set after the first run in the previous test plan. I believe this is, in fact, appropriate. In the previous test plan, the production URL was incorrect, so the first run was a 404. We return "0" -- not disabled -- as a default, but since we didn't actually fetch the remote pref we don't store it for later.

Updated test plan

(Link to new try build will arrive shortly.)

For testing, I've exposed a debug command that simply runs the code and prints the output. To witness the actual task being run, the initial WDBA testing approach should be possible here: use the Windows Task Scheduler GUI and use a netcap program (like Fiddler) to witness Telemetry outbound pings.

For testing, the default/prod URL will be "enabled":

https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/windows-default-browser-agent/records/state

and the corresponding stage URL is "disabled":

https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/state

To change the URL, change just the "server URL" under services.settings.server. Production is

https://firefox.settings.services.mozilla.com/v1

and stage is

https://settings.stage.mozaws.net/v1

  • Install from try build. Elevated privileges should not matter, but I tested without them so please test with them. DO NOT RUN THE INSTALLED FIREFOX IMMEDIATELY

  • Open a command prompt. Again, elevated privileges should not matter. Navigate to the location of firefox.exe, and run the default browser agent debug commands:

  C:\Users\nalexander\AppData\Local\Firefox Nightly>default-browser-agent.exe debug-remote-disabled
  default-browser-agent: Last remote disabled: 0
  default-browser-agent: Remote service disabled state URL: 'https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/windows-default-browser-agent/records/state'
  [ERROR defaultagent_static] Using remote settings URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/windows-default-browser-agent/records/state
  IsAgentRemoteDisabled: 0

Witness the URL, which should fallback to the production URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/windows-default-browser-agent/records/state. This tests that not having the service URL setting in the registry doesn't cause any problems.

  • Run the installed Firefox. This populates certain registry entries with settings. In the absence of those settings (the server root URL to fetch from), we should fallback to the production URL, as verified above.

  • (Optional) Look in regedit under the key Computer\HKEY_CURRENT_USER\Software\Mozilla\Firefox\Default Browser Agent and witness entries with names like
    C:\Users\nalexander\AppData\Local\Firefox Nightly|ServicesSettingsServer with server root defaulting to
    https://firefox.settings.services.mozilla.com/v1

  • (Optional) witness the last witnessed value (0) in regedit, like C:\Users\nalexander\AppData\Local\Firefox Nightly|DefaultAgentLastRemoteDisabled

  • In Firefox about:config, set
    services.settings.server to
    https://settings.stage.mozaws.net/v1. This is a staging record that disables WDBA.

  • Open a command prompt. Again, elevated privileges should not matter. Navigate to the location of firefox.exe, and run the default browser agent debug commands:

  C:\Users\nalexander\AppData\Local\Firefox Nightly>default-browser-agent.exe debug-remote-disabled
  default-browser-agent: Last remote disabled: 0
  default-browser-agent: Remote service disabled state URL: 'https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/state
  [ERROR defaultagent_static] Using remote settings URL: https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/state
  default-browser-agent: Error code: 0x0
  default-browser-agent: Next remote disabled: 1
  IsAgentRemoteDisabled: 1

Witness the configured URL. If the network is available, you should see IsAgentRemoteDisabled: 1.

  • Disable the network.

  • Run debug-remote-disabled again. Witness an error code but that we fall back to the last witnessed remote disabled value (here, 1, meaning "yes disabled").

  C:\Users\nalexander\AppData\Local\Firefox Nightly>default-browser-agent.exe debug-remote-disabled
  default-browser-agent: Last remote disabled: 0
  default-browser-agent: Remote service disabled state URL: 'https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/state'
  [ERROR defaultagent_static] Using remote settings URL: https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/state
  [ERROR defaultagent_static] Error::NetworkError
  default-browser-agent: Error code: 0x7d01
  IsAgentRemoteDisabled: 1

  • Re-enable the network.

  • (Optional) witness the last witnessed value (1) in regedit, like C:\Users\nalexander\AppData\Local\Firefox Nightly|DefaultAgentLastRemoteDisabled

  • In Firefox about:config, set
    services.settings.server to the original "not disabled" server root:
    https://firefox.settings.services.mozilla.com/v1.

  • Run debug-remote-disabled again. Witness no error code and the correct "not disabled" value of 0:

  C:\Users\nalexander\AppData\Local\Firefox Nightly>default-browser-agent.exe debug-remote-disabled
  default-browser-agent: Last remote disabled: 0
  default-browser-agent: Remote service disabled state URL: 'https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/state'
  [ERROR defaultagent_static] Using remote settings URL: https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/state'
  default-browser-agent: Error code: 0x0
  default-browser-agent: Next remote disabled: 0
  IsAgentRemoteDisabled: 0

Rachel had the wise idea to test against a malformed record. That turns out to be non-trivial since Remote Settings has a schema preventing syntactic errors, but I've added a "malformed" record that has semantic errors (specifically, "version": "v0"). And indeed:

default-browser-agent: Last remote disabled: 0
default-browser-agent: Remote service disabled state URL: 'https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/malformed'
[ERROR defaultagent_static] Using remote settings URL: https://settings.stage.mozaws.net/v1/buckets/main/collections/windows-default-browser-agent/records/malformed
[ERROR defaultagent_static] Error::SettingsError
default-browser-agent: Error code: 0x7d02
IsAgentRemoteDisabled: 0

Our own, explicit URL is easier for testing, but
services.settings.server fits into Firefox better. Now that we have
a production record in place we should use that, too.

This patch will be folded/absorbed into the earlier patches in the
series.

Depends on D80768

Depends on: 1648617
Attachment #9158733 - Attachment description: Bug 1636539 - Part 2: Mirror preferences for remote settings to Windows Default Browser Agent. r?agashlin,bytesized → Bug 1636539 - Part 1: Mirror preferences for remote settings to Windows Default Browser Agent. r?agashlin,bytesized
Attachment #9154689 - Attachment description: Bug 1636539 - Part 3: Add defaultagent-static Rust crate fetching remote disabled flag. r?agashlin,bytesized → Bug 1636539 - Part 2: Add defaultagent-static Rust crate fetching remote disabled flag. r?agashlin,bytesized
Attachment #9158734 - Attachment description: Bug 1636539 - Part 4: Use Rust remote settings client to fetch remote kill switch. r?agashlin,bytesized → Bug 1636539 - Part 3: Use Rust remote settings client to fetch remote kill switch. r?agashlin,bytesized
Attachment #9159420 - Attachment description: Bug 1636539 - Part 5: Note remote disablement in browser default agent docs. r?agashlin,bytesized → Bug 1636539 - Part 4: Note remote disablement in browser default agent docs. r?agashlin,bytesized
Attachment #9158732 - Attachment is obsolete: true
Attachment #9159419 - Attachment is obsolete: true

This is an easy way to expose MOZ_APP_DISPLAYNAME to Rust code. A
visibility declaration is required to expose the symbol; the MOZ_*
macros follow the model of /mfbt/Poison.{cpp,h}.

Depends on D81232

This feature is no longer confidential.

Group: mozilla-employee-confidential
Blocks: 1680171
Flags: needinfo?(mathieu)
Component: Installer → Default Browser Agent
Product: Firefox → Toolkit
Target Milestone: Firefox 80 → ---
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: