Currently I don't think there is a good way to reliably validate that this is working as expected. I wrote a query [1] that checks the size and number of pings vs. how many pings were deleted due to size overflow, but that query is broken in two ways: 1. It is looking at error tables, because I do not have permissions to look at the ping tables; 2. It assumes pings with more than 1MB are sent, when in fact they are rejected by the pipeline with a 413 code; Number 2 makes me question the usefulness of the whole "discard pings that are over 1MB" feature, since we kind of already had that since the beggining. A 413, as all 4xx status codes, are considered unrecoverable and will make Glean delete the ping anyways. Having that built in will only save us the trip of going through the whole upload cycle with a ping that will ultimately be discarded. Aaanyways, I am untaking until Bug 1662470 is resolved. When it is I can tweak [1] to use the content-length of sent pings. I can then add the resulting query to our monitoring dashboard, and we can continually monitor discarded pings vs. size / number of pings. Ideally we should never see any discarded pings (and for the time being we haven't seen any, thus the difficulty in validating this), if we monitor this periodically we can then see when we do get them and take action if needed. If we never / rarely see any discarded pings we can revisit the usefulness of this feature, if the size limit should be tweaked, etc. [:chutten] Do you possibly have any clever ways I could possibly validate this? [1]: [Analyze number of discarded pings due to size overflow](https://sql.telemetry.mozilla.org/queries/74318/source#185766)
Bug 1655932 Comment 1 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Currently I don't think there is a good way to reliably validate that this is working as expected. I wrote a query [1] that checks the size and number of pings vs. how many pings were deleted due to size overflow, but that query is broken in two ways: 1. It is looking at error tables, because I do not have permissions to look at the ping tables; 2. It assumes pings with more than 1MB are sent, when in fact they are rejected by the pipeline with a 413 code; Number 2 makes me question the usefulness of the whole "discard pings that are over 1MB" feature, since we kind of already had that since the beggining. A 413, as all 4xx status codes, are considered unrecoverable and will make Glean delete the ping anyways. Having that built in will only save us the trip of going through the whole upload cycle with a ping that will ultimately be discarded. Aaanyways, I am untaking until Bug 1662470 is resolved. When it is I can tweak [1] to use the content-length of sent pings. I can then add the resulting query to our monitoring dashboard, and we can continually monitor discarded pings vs. size / number of pings. Ideally we should never see any discarded pings (and for the time being we haven't seen any, thus the difficulty in validating this), if we monitor this periodically we can then see when we do get them and take action if needed. If we never / rarely see any discarded pings we can revisit the usefulness of this feature, if the size limit should be tweaked, or even if the metric is broken, etc. [:chutten] Do you possibly have any clever ways I could possibly validate this? [1]: [Analyze number of discarded pings due to size overflow](https://sql.telemetry.mozilla.org/queries/74318/source#185766)