1340535 - Create a QA test-plan sending shutdown pings with the PingSender

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect, P1)

Description

[Alessio Placitelli [:Dexter]]

We need to write a proper QA test plan for sending the shutdown ping with the PingSender when the browser shuts down.

Some of the things we should cover:

- A shutdown ping must be sent, either trough the PingSender or through the normal Telemetry flow.
- If a ping is successfully sent through the PingSender, it must be pruned from the pending pings directory (if not, we should list the exceptional cases).

Comment 1

[Alessio Placitelli [:Dexter]]

Abe kindly produced a test-plan [1] and test cases [2] for the shutdown ping.

[1] - https://wiki.mozilla.org/QA/Shutdown_Ping
[2] - https://docs.google.com/a/mozilla.com/spreadsheets/d/1BclUXq-VjR26UDevJZLA_6vJeAAxre6Nty3U0NVGnM4/edit?usp=sharing

Comment 2

[Alessio Placitelli [:Dexter]]

Mark, Chris can you think of any other risk/test cases from comment 1? Can you think of anything we missed?

Comment 3

[Chris H-C :chutten]

Do we need to put any time controls on this? (e.g. A shutdown ping must be sent by pingSender within X seconds of shutdown)

Should we put any integrity controls on this? (The shutdown ping sent by pingSender must be well-formed, compressed, etc. In any exceptional cases when more than one ping is sent, they must be identical in content...)

Comment 4

[Mark Reid [:mreid]]

We should make sure that the submission URL from pingSender vs. normal Telemetry are the same (ie. things external to the submitted payload).

pingSender should include the standard HTTP "Date" header.

Do we care if pingSender sends the same value for the DNT header?

Comment 5

[Alessio Placitelli [:Dexter]]

(In reply to Chris H-C :chutten from comment #3)
> Do we need to put any time controls on this? (e.g. A shutdown ping must be
> sent by pingSender within X seconds of shutdown)

I would personally stay away from time sensitive checks even for manual QA.
Moreover, the pingsender is not required to send pings within X seconds from the browser shutdown: if it does, good, if it doesn't that's still good!

> Should we put any integrity controls on this? (The shutdown ping sent by
> pingSender must be well-formed, compressed, etc. In any exceptional cases
> when more than one ping is sent, they must be identical in content...)

Yes, that's already mentioned in the test cases from comment 1 (schema validation,
identical content for duplicate pings, etc.).

Compression is not covered as it will be taken care of in a separate bug. We will update the test cases accordingly after it lands.

(In reply to Mark Reid [:mreid] from comment #4)
> We should make sure that the submission URL from pingSender vs. normal
> Telemetry are the same (ie. things external to the submitted payload).

Good point!

> pingSender should include the standard HTTP "Date" header.

Good point #2! Also, the pingsender wasn't sending the date header.
I just added it today!

> Do we care if pingSender sends the same value for the DNT header?

I don't think so.

Comment 6

[Alessio Placitelli [:Dexter]]

@Abe, given the discussion in the previous comments, could we add the following:

- We should make sure that the submission URL from ping sender vs. normal Telemetry are the same.
- We should make sure that the submission from the ping sender have the following headers:

User-Agent: pingsender/1.0
X-PingSender-Version: 1.0
Date: <as described in https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Date )

@Georg, could you take one look at this as well?

Comment 7

[Abe - QA (:Abe_LV)]

Updated the test cases accordingly. The first one (URL should be identical) is already included in the existing test cases, but the second one (headers verification) is now updated. Thanks for the feedback.

Comment 8

[Alessio Placitelli [:Dexter]]

We should be good to go with the test plan, we'll execute it once the shutdown pignsender change lands.