1543728 - Handle connection loss in the upload worker

Status: RESOLVED WORKSFORME

(Toolkit :: Telemetry, defect, P1)

Description

[Alessio Placitelli [:Dexter]]

(In reply to Alessio Placitelli [:Dexter] from bug 1541022 comment #1)

(1) if we loose network connection, given that that's part of our worker constraints, the worker gets killed.

That's a documented behaviour of the WorkManager. I think that this can cause holes in our sequence numbers/data loss. Because:

• The worker calls PingStorageEngine.process

• in turn, this calls PingstorageEngine.processFile for each file in the storage directory. If upload is successful, this function deletes the file

• and, I think, we don't fail on network connectivity when uploading

This ends up deleting pings when network is down, while the worker is running. This is not highly likely to happen, but there's a chance that will happen.

Comment 1

[Alessio Placitelli [:Dexter]]

Looks like I was wrong, we're correctly handling the disconnection case. To prove this, I added a sleep here so that the worker wouldn't finish too quickly and allow me time to turn the connection off in the emulator. Then, I generated a few pings by going to background. After a bit, I saw the following exception in the logcat:

D/glean/PingStorageEngine: Processing ping: 70bbf164-5800-467f-865a-da6c5889dec4
D/glean/HttpPingUploader: Submitting ping to: https://incoming.telemetry.mozilla.org/submit/org-mozilla-samples-glean-debug/baseline/1/70bbf164-5800-467f-865a-da6c5889dec4
I/zygote: Do partial code cache collection, code=60KB, data=53KB
I/zygote: After code cache collection, code=60KB, data=53KB
I/zygote: Increasing code cache capacity to 256KB
W/glean/HttpPingUploader: IOException while uploading ping
java.net.UnknownHostException: Unable to resolve host "incoming.telemetry.mozilla.org": No address associated with hostname
at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:141)
at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:90)
at java.net.InetAddress.getAllByName(InetAddress.java:787)
at com.android.okhttp.Dns$1.lookup(Dns.java:39)
at com.android.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:175)
at com.android.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:141)
at com.android.okhttp.internal.http.RouteSelector.next(RouteSelector.java:83)
at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:174)
at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126)
at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95)
at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281)
at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:461)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:127)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:258)
at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:218)
at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(Unknown Source:0)
at mozilla.components.lib.fetch.httpurlconnection.HttpURLConnectionClientKt$addBodyFrom$$inlined$let$lambda$1.invoke(HttpURLConnectionClient.kt:56)
at mozilla.components.lib.fetch.httpurlconnection.HttpURLConnectionClientKt$addBodyFrom$$inlined$let$lambda$1.invoke(Unknown Source:2)
at mozilla.components.concept.fetch.Request$Body.useStream(Request.kt:92)
at mozilla.components.lib.fetch.httpurlconnection.HttpURLConnectionClientKt.addBodyFrom(HttpURLConnectionClient.kt:55)
at mozilla.components.lib.fetch.httpurlconnection.HttpURLConnectionClientKt.access$addBodyFrom(HttpURLConnectionClient.kt:1)
at mozilla.components.lib.fetch.httpurlconnection.HttpURLConnectionClient.fetch(HttpURLConnectionClient.kt:32)
at mozilla.components.service.glean.net.HttpPingUploader.performUpload$service_glean_debug(HttpPingUploader.kt:119)
at mozilla.components.service.glean.net.HttpPingUploader.upload(HttpPingUploader.kt:71)
at mozilla.components.service.glean.scheduler.PingUploadWorker$Companion$uploadPings$1.invoke(PingUploadWorker.kt:67)
at mozilla.components.service.glean.scheduler.PingUploadWorker$Companion$uploadPings$1.invoke(PingUploadWorker.kt:24)
at mozilla.components.service.glean.storages.PingStorageEngine.processFile(PingStorageEngine.kt:127)
at mozilla.components.service.glean.storages.PingStorageEngine.process(PingStorageEngine.kt:100)
at mozilla.components.service.glean.scheduler.PingUploadWorker$Companion.uploadPings$service_glean_debug(PingUploadWorker.kt:67)
at mozilla.components.service.glean.scheduler.PingUploadWorker.doWork(PingUploadWorker.kt:84)
at androidx.work.Worker$1.run(Worker.java:85)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
at java.lang.Thread.run(Thread.java:764)
Caused by: android.system.GaiException: android_getaddrinfo failed: EAI_NODATA (No address associated with hostname)
at libcore.io.Linux.android_getaddrinfo(Native Method)
at libcore.io.ForwardingOs.android_getaddrinfo(ForwardingOs.java:59)
at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:122)
at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:90) 
at java.net.InetAddress.getAllByName(InetAddress.java:787) 
at com.android.okhttp.Dns$1.lookup(Dns.java:39) 
at com.android.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:175) 
at com.android.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:141) 
at com.android.okhttp.internal.http.RouteSelector.next(RouteSelector.java:83) 
at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:174) 
at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126) 
at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95) 
at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281) 
at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224) 
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:461) 
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:127) 
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:258) 
at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:218) 
at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(Unknown Source:0) 
at mozilla.components.lib.fetch.httpurlconnection.HttpURLConnectionClientKt$addBodyFrom$$inlined$let$lambda$1.invoke(HttpURLConnectionClient.kt:56) 
at mozilla.components.lib.fetch.httpurlconnection.HttpURLConnectionClientKt$addBodyFrom$$inlined$let$lambda$1.invoke(Unknown Source:2) 
at mozilla.components.concept.fetch.Request$Body.useStream(Request.kt:92) 
at mozilla.components.lib.fetch.httpurlconnection.HttpURLConnectionClientKt.addBodyFrom(HttpURLConnectionClient.kt:55) 
at mozilla.components.lib.fetch.httpurlconnection.HttpURLConnectionClientKt.access$addBodyFrom(HttpURLConnectionClient.kt:1) 
at mozilla.components.lib.fetch.httpurlconnection.HttpURLConnectionClient.fetch(HttpURLConnectionClient.kt:32) 
at mozilla.components.service.glean.net.HttpPingUploader.performUpload$service_glean_debug(HttpPingUploader.kt:119) 
at mozilla.components.service.glean.net.HttpPingUploader.upload(HttpPingUploader.kt:71) 
at mozilla.components.service.glean.scheduler.PingUploadWorker$Companion$uploadPings$1.invoke(PingUploadWorker.kt:67) 
at mozilla.components.service.glean.scheduler.PingUploadWorker$Companion$uploadPings$1.invoke(PingUploadWorker.kt:24) 
at mozilla.components.service.glean.storages.PingStorageEngine.processFile(PingStorageEngine.kt:127) 
at mozilla.components.service.glean.storages.PingStorageEngine.process(PingStorageEngine.kt:100) 
at mozilla.components.service.glean.scheduler.PingUploadWorker$Companion.uploadPings$service_glean_debug(PingUploadWorker.kt:67) 
at mozilla.components.service.glean.scheduler.PingUploadWorker.doWork(PingUploadWorker.kt:84) 
at androidx.work.Worker$1.run(Worker.java:85) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636) 
at java.lang.Thread.run(Thread.java:764) 
E/glean/PingStorageEngine: Error processing ping file: 70bbf164-5800-467f-865a-da6c5889dec4

This suggests that the upload function is returning false and that, as a consequence, the ping file is not being deleted, which is the right behaviour.