Open Bug 1302715 Opened 8 years ago Updated 1 month ago

support long running service workers with FetchEvent.respondWith() while controlled window is open

Categories

(Core :: DOM: Service Workers, defect, P3)

49 Branch
defect

Tracking

()

People

(Reporter: marko.seidenglanz, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(1 file, 1 obsolete file)

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36

Steps to reproduce:

1. Installed a service worker that does a lot of background requests to fetch data that should be delivered as blob to the client.

2. client sends a request which gets intercepted by the service worker

Exactly 5.5 minutes after the request was sent by the client we get the following Error: "Script terminated by timeout"

Is there a response timeout defined in Firefox?
As the streaming api seems not to be implemented yet in FF, is there another way to delay the response without trapping into this timeout?
Component: Untriaged → DOM: Service Workers
Product: Firefox → Core
Can you post your service worker script?  Do your background downloads really take 5+ minutes?  What event are you performing these background downloads in?

To answer your question, though, yes we do kill workers that have a respondWith() or waitUntil() holding it alive for excessive time periods.  This is to avoid malicious sites abusing the mechanism to perform bitcoin mining and the like in background threads.
Flags: needinfo?(marko.seidenglanz)
We use the service worker to collect a large amount of data to stream it to the client. In Chrome we can use  the ReadableStream API which works very well. As the Stream API seams yet not be supported by Firefox we collect the data and want to make it downloadable using a BLOB.

As we have to load millions of entities, this operation takes a lot of time which also doesn't seem to be a problem in Chrome. Unfortunately it doesn't work with Firefox since the worker gets killed before finishing his work.

The operation is performed inside the "fetch" event. FYI i've attached the script.
Flags: needinfo?(marko.seidenglanz)
Attached file download-worker.js
Comment on attachment 8792435 [details]
download-worker.js

>/********************************************************************
> * * * * * * * * * * * * * * DEPENDENCIES * * * * * * * * * * * * * * 
> ********************************************************************/
>importScripts('https://cdnjs.cloudflare.com/ajax/libs/co/4.1.0/index.js');
>//importScripts('https://cdnjs.cloudflare.com/ajax/libs/axios/0.14.0/axios.js');
>
>
>/*********************************************************************
> * * * * * * * * * * * * * * CONFIGURATION * * * * * * * * * * * * * * 
> *********************************************************************/
>var worker_count = 30;
>var worker_start_interval = 2000; // DEFAULT: 2000
>var bulk_size = 20;
>var retry_count = 10; // DEFAULT: 10;
>var max_sleep_millis = 60000; // DEFAULT: 60000;
>var useCredentials = true;
>
>// DEFAULT URL
>var BASE_URL = "XXX";

>
>/********************************************************************
> * * * * * * * * * * * * * * EVENT LISTENER * * * * * * * * * * * * * 
> ********************************************************************/
>this.addEventListener('install', event => {
>
>	// Updates sofort installieren
>	// https://developer.mozilla.org/en-US/docs/Web/API/ServiceWorkerGlobalScope/skipWaiting
>	if (self.skipWaiting) {
>		event.waitUntil(self.skipWaiting());
>	}
>
>	console.log("SW: Install contact export worker", event);
>});
>
>this.addEventListener('activate', event => {
>
>	// Worker sofort aktivieren
>	// https://developer.mozilla.org/en-US/docs/Web/API/ServiceWorkerGlobalScope/skipWaiting
>	if (self.clients && self.clients.claim) {
>		event.waitUntil(self.clients.claim());
>	}
>
>	console.log("SW: Activate contact export worker", event);
>});
>
>this.addEventListener('fetch', event => {
>
>	console.log("SW: Fetch request received", event);
>
>	// Fetch only export requests
>	if (!event.request.url.endsWith("/web/js/export/start")) {
>		console.info("SW: No handler defined for " + event.request.url);
>		return;
>	}
>
>	event.respondWith(new Promise(
>		function(resolve, reject) {
>			co(run(event.request, resolve));
>		}));
>});
>
>
>/******************************************************************************
> * * * * * * * * * * * * * * * * * GENERATORS * * * * * * * * * * * * * * * * *
> ******************************************************************************/
>function* run(request, resolve) {
>	try {
>
>		var formData =
>			yield request.text();
>
>		// exportParams=<data>
>		var decodedData = decodeURIComponent(formData.substring("exportParams=".length));
>		var reqData = JSON.parse(decodedData);
>
>		var results = {
>			duration_sec: 0,
>			numContacts: 0,
>			failed: [],
>			errors: [],
>			state: "running"
>		};
>
>		var baseUrl = BASE_URL;
>		var token = reqData.token;
>		var cid = reqData.campaignId;
>		var filterQuery = reqData.searchElements;
>		var fieldNames = reqData.fieldNames;
>
>		// get base url from request (in production use)
>		if (reqData.baseUrl) {
>			baseUrl = reqData.baseUrl;
>			useCredentials = true;
>			max_sleep_millis = 60000;
>		}
>
>		var idQueue = new Queue();
>		var contactsQueue = new Queue();
>
>		// start id fetcher
>		co(idFetcher({
>			baseUrl: baseUrl,
>			token: token,
>			cid: cid,
>			filterQuery: filterQuery,
>			idQueue: idQueue,
>			results: results
>		})).catch(function(e) {
>			results.state = "error";
>			results.errors.push(e.toString());
>			resolve(new Response(e));
>		});
>
>		// start worker
>		co(workerStarter({
>			baseUrl: baseUrl,
>			token: token,
>			cid: cid,
>			fieldNames: fieldNames,
>			idQueue: idQueue,
>			contactsQueue: contactsQueue,
>			results: results
>		})).catch(function(e) {
>			results.state = "error";
>			results.errors.push(e.toString());
>			resolve(new Response(e));
>		});
>
>
>
>		var streamAvailable = typeof ReadableStream !== "undefined";
>		if (streamAvailable) {
>
>			/******************************
>			 * * * * STREAM HANDLER * * * *
>			 ******************************/
>			console.info("SW: Get contacts as stream");
>
>			// Response as stream
>			var stream = new ReadableStream({
>				start(controller) {
>
>					console.log("SW: Start stream controller");
>
>					co(streamProcessor({
>						fieldNames: fieldNames,
>						contactsQueue: contactsQueue,
>						streamController: controller,
>						results: results
>					})).catch(function(e) {
>						console.error(e.stack);
>						results.state = "error";
>						results.errors.push(e.toString());
>						resolve(new Response(e));
>					});
>
>				}
>			});
>
>			resolve(new Response(stream, {
>				headers: {
>					"Content-Type": "text/plain;charset=utf-8",
>					"Content-Disposition": "attachment; filename=contacts_" + new Date().toISOString() + ".txt"
>				}
>			}));
>		} else {
>
>
>			/******************************
>			 * * * * * BLOB HANDLER * * * *
>			 ******************************/
>			console.log("SW: Get contacts as Blob");
>
>			co(blobProcessor({
>				fieldNames: fieldNames,
>				results: results,
>				contactsQueue: contactsQueue,
>				onFinish: function(data) {
>
>					console.info(JSON.stringify(results));
>
>					resolve(new Response(
>						new Blob(data, {
>							type: "text/plain;charset=utf-8"
>						}), {
>							headers: {
>								"Content-Type": "text/plain;charset=utf-8",
>								"Content-Disposition": "attachment; filename=contacts_" + new Date().toISOString() + ".txt"
>							}
>						}
>					));
>				}
>			})).catch(function(e) {
>				console.error(e.stack);
>				resolve(new Response(e));
>			});
>		}
>
>	} catch (e) {
>		console.error(e.stack);
>		resolve(new Response(e));
>		return;
>	}
>}
>
>function* blobProcessor(config) {
>	var tsStart = Date.now();
>
>	// Create Header
>	var csvHeader = fieldNamesToCSVHeader(config.fieldNames);
>	var csvArray = [csvHeader];
>	var contactsIter = config.contactsQueue.iterator();
>
>	while (true) {
>
>		var c = contactsIter.next(); //get next contact
>
>		if (c.done) { //no more contacts left, done
>
>			if (results.state == "error") {
>				config.onError(new Error("Export failed"));
>			} else {
>				results.state == "success";
>				config.onFinish(csvArray);
>			}
>
>			config.results.duration_sec = Math.round((Date.now() - tsStart) / 1000);
>			console.info(JSON.stringify(config.results));
>			return;
>		}
>		var contacts = c.value;
>
>		if (contacts.length > 0) {
>
>			//console.info("EXPORT: Blob processor next contacts", contacts)
>
>			csvArray = csvArray.concat(contacts);
>		} else {
>			var sleepDur = 1000;
>			//console.info("EXPORT: Blob processor sleep " + sleepDur);
>			yield sleep(sleepDur);
>		}
>	}
>}
>
>
>function* streamProcessor(config) {
>	var tsStart = Date.now();
>
>	var contactsIter = config.contactsQueue.iterator();
>	var encoder = new TextEncoder();
>
>	// Create Header
>	var csvHeader = fieldNamesToCSVHeader(config.fieldNames);
>	config.streamController.enqueue(encoder.encode(csvHeader));
>
>	while (true) {
>
>		var c = contactsIter.next(); //get next contact
>
>		//console.log("Emitted contact", c);
>		if (c.done) { //no more contacts left, done
>
>			console.info("SW: Export done!");
>			config.results.duration_sec = Math.round((Date.now() - tsStart) / 1000);
>
>			if (config.results.state == "error") {
>				// Write error to file
>				var errorStr = encoder.encode("\n\n" + JSON.stringify(config.results));
>				config.streamController.enqueue(errorStr);
>			} else {
>				config.results.state == "success";
>			}
>
>			config.streamController.close();
>			console.info(JSON.stringify(config.results));
>			console.info("SW: Close stream clontroller");
>
>
>			return;
>		}
>		var contacts = c.value;
>
>		if (contacts.length > 0) {
>
>			//console.info("SW: Stream processor next contacts", contacts)
>
>			var contactStr = encoder.encode(contacts);
>			config.streamController.enqueue(contactStr);
>		} else {
>			var sleepDur = 1000;
>			//console.info("SW: Stream processor sleep " + sleepDur);
>			yield sleep(sleepDur);
>		}
>	}
>}
>
>function* idFetcher(config) {
>
>	console.info("SW: Start id fetcher");
>
>	var url = config.baseUrl + "!" + config.token + "/api/campaigns/" + config.cid + "/contacts/filter";
>	var cursor = "";
>
>	function stop() {
>		console.info("SW: Stop id fetcher");
>		config.idQueue.finished = true;
>	}
>
>	while (true) {
>
>		var hits;
>		for (var t = 1; t <= retry_count; t++) {
>
>			var query = config.filterQuery.concat({
>				"field": "_cursor_",
>				"values": [cursor],
>				"operator": ""
>			});
>
>			try {
>
>				var params = {
>					method: "POST",
>					body: JSON.stringify(query)
>				};
>				if (useCredentials) {
>					params.credentials = "include";
>					params.mode = "cors";
>				}
>
>				var resp =
>					yield fetch(new Request(url), params);
>
>				if (resp.status === 200) {
>					var respData =
>						yield resp.json();
>					hits = respData.hits;
>					cursor = respData.cursor;
>					break;
>				} else {
>					throw new Error("Request " + url + " failed with status " + resp.status);
>				}
>			} catch (err) {
>				console.error(err);
>
>				if (t == retry_count) {
>					stop();
>					throw err;
>				}
>			}
>
>			// sleep before
>			var sleepDur = Math.min(Math.pow(2, t) * 1000, max_sleep_millis);
>			// TESTING
>			//var sleepDur = 1000;
>			console.error("SW: id fetcher sleep " + sleepDur);
>			yield sleep(sleepDur);
>		}
>
>		// process hits
>		for (var h in hits) {
>			var hit = hits[h];
>			config.idQueue.push(hit.$id);
>		}
>
>		if (!cursor) {
>			stop();
>			return;
>		}
>	}
>}
>
>function* workerStarter(config) {
>
>	config.numWorkers = 0;
>
>	// start multiple contact fetcher
>	var n = 0;
>	while (n < worker_count) {
>
>		//console.info("START WORKER", config.idQueue.size(), config.idQueue.size() % n);
>
>		// Do not start any more worker if all ids have been fetched and existing workers have not more than 100 ids to fetch
>		//if (config.idQueue.finished && (config.idQueue.size() / n) < 100) return;
>		if (n > 0 && config.idQueue.finished) return;
>
>		// Start new worker if existing workers have more than 100 ids to fetch
>		//if (n === 0 || (config.idQueue.size() / n) > 100) {
>		co(contactFetcher(++n, config));
>		/*.catch(function(err) {
>						console.error(err.stack);
>					});*/
>		//}
>
>		// Sleep 2 seconds before starting next worker
>		var sleepDur = worker_start_interval;
>		yield sleep(sleepDur); // sleep before
>	}
>}
>
>// this is the worker function
>function* contactFetcher(id, config) {
>
>	console.info("SW: Start contact fetcher " + id);
>	config.numWorkers++;
>
>	// Create Iterator for ID Queue
>	var idIter = config.idQueue.iterator(bulk_size);
>	var url = config.baseUrl + "!" + config.token + "/api/campaigns/" + config.cid + "/contacts/";
>
>	while (true) {
>
>		var i = idIter.next();
>
>		// sleep if queue empty
>		if (i.done) { //no more contacts left, done
>			console.info("SW: Stop contact fetcher " + id);
>			break;
>		}
>
>		var ids = i.value;
>		if (ids.length > 0) {
>
>			//console.log("SW: Worker " + id + " next: ", ids);
>
>			for (var t = 1; t <= retry_count; t++) {
>				try {
>					var params = {
>						method: "POST",
>						body: JSON.stringify(ids)
>					};
>					if (useCredentials) {
>						params.credentials = "include";
>						params.mode = "cors";
>					}
>
>					var resp =
>						yield fetch(new Request(url), params);
>
>
>					if (resp.status === 200) {
>						var contacts =
>							yield resp.json();
>						//console.log("Add contacts", contacts)
>
>						// Transform to csv
>						var csvData = contactsToCSVData(config.fieldNames, contacts);
>
>						// TESTING PUSH EVERY CONTACT 10.000
>						//for (var s = 0; s < 50; s++) { // REMOVE
>						config.contactsQueue.push(csvData);
>						config.results.numContacts += contacts.length;
>						//}
>						//console.info("Contact finished", contacts[0].$id) // REMOVE
>						break;
>					} else {
>						throw new Error("Request " + url + " failed with status " + resp.status);
>					}
>				} catch (err) {
>					console.error(err);
>					if (t === retry_count) {
>						config.results.failed = config.results.failed.concat(ids);
>					}
>				}
>
>				var sleepDur = Math.min(Math.pow(2, t) * 1000, max_sleep_millis);
>				//console.info("SW: Contact fetcher " + id + " sleep " + sleepDur);
>				yield sleep(sleepDur); // sleep before
>				//console.info("SW: Contact fetcher " + id + " wakeup");
>			}
>
>		} else {
>			var sleepDur = 1000;
>			//console.info("SW: Contact fetcher " + id + " sleep " + sleepDur);
>			yield sleep(sleepDur);
>			//console.info("SW: Contact fetcher " + id + " wakeup");
>			continue;
>		}
>	}
>	if (--config.numWorkers === 0) {
>		console.info("SW: All contact fetcher stopped");
>		config.contactsQueue.finished = true;
>		return;
>	}
>
>	//console.info("SW: Workers left " + config.numWorkers);
>}
>
>/*******************************************************
> * * * * * * * * * * * * * QUEUE * * * * * * * * * * * *
> *******************************************************/
>var Queue = (function() {
>	function Queue() {
>		this.items = [];
>		this.finished = false;
>	}
>
>	Queue.prototype.get = function(count) {
>		return this.items.splice(0, count);
>	};
>
>	Queue.prototype.push = function(items) {
>		this.items = this.items.concat(items);
>	};
>
>	Queue.prototype.size = function() {
>		return this.items.length;
>	};
>
>	Queue.prototype.iterator = function(fetchSize) {
>
>		return function*(queue, fetchSize) {
>
>			fetchSize = fetchSize || 1;
>
>			while (true) {
>
>				if (queue.finished) {
>					if (queue.size() === 0) return;
>					else yield queue.get(fetchSize);
>				} else {
>					if (queue.size() < fetchSize) yield [];
>					else yield queue.get(fetchSize);
>				}
>			}
>		}(this, fetchSize);
>	};
>
>	return Queue;
>})();
>
>
>
>
>/*******************************************************
> * * * * * * * * * * * * HELPER * * * * * * * * * * * *
> *******************************************************/
>/*
>function getLocation(href) {
>	var match = href.match(/^(https?\:)\/\/(([^:\/?#]*)(?:\:([0-9]+))?)(\/[^?#]*)(\?[^#]*|)(#.*|)$/);
>	return match && {
>		protocol: match[1],
>		host: match[2],
>		hostname: match[3],
>		port: match[4],
>		pathname: match[5],
>		search: match[6],
>		hash: match[7]
>	}
>}
>*/
>
>function sleep(ms) {
>	var p = new Promise(function(resolve, reject) {
>		setTimeout(function() {
>			resolve();
>		}, ms);
>	});
>	return p;
>}
>
>
>var csv_options = {
>	separator: "\t",
>	delimiter: "",
>	linebreak: "\r\n"
>};
>
>/******************************************************
> * * * * * * * * * * * CSV PARSER * * * * * * * * * * *
> ******************************************************/
>function fieldNamesToCSVHeader(fieldNames) {
>
>	var line_data = [];
>	for (var idx = 0; fieldNames && idx < fieldNames.length; idx += 1) {
>		var fName = fieldNames[idx];
>		line_data.push(fName);
>	}
>	return line_data.join(csv_options.separator) + csv_options.linebreak;
>}
>
>function contactsToCSVData(fieldNames, contacts) {
>
>	if (contacts.length === 0) return [];
>
>	var contact;
>	var line_data = [];
>	var csv_data = [];
>	for (var c_idx = 0; c_idx < contacts.length; c_idx += 1) {
>		contact = contacts[c_idx];
>		line_data = [];
>		for (var idx = 0; fieldNames && idx < fieldNames.length; idx += 1) {
>			var fName = fieldNames[idx];
>			line_data.push(filterCsv(contact[fName] || ""));
>		}
>
>		var line = line_data.join(csv_options.separator);
>		csv_data.push(line);
>	}
>
>	// Empty line at end
>	csv_data.push("");
>
>	return csv_data.join(csv_options.linebreak);
>}
>
>
>var reFilterCSV = new RegExp("\t|\r|\n", "g");
>
>function filterCsv(text) {
>	if (typeof text === "undefined" || text === null) {
>		return "";
>	}
>	if (typeof text !== "string") {
>		text = text.toString();
>	}
>
>	return text.replace(reFilterCSV, " ");
>}
Group: core-security
How does this work in browsers without a service worker?  Does the server aggregate the requests into a single response?

Also, why did you mark this a security issue?
Flags: needinfo?(marko.seidenglanz)
Also, if I'm reading this right, you are doing all of these secondary fetch() calls in sequence.  You're getting no parallelism.  It seems like you should be able to make this go much faster with something like:

  var urlList = [ /* ... */ ];
  Promise.all(urlList.map(u => fetch(u))).then(responseList => {
    // process the responses
  });

I believe chrome exempts FetchEvent for the "you've been running too long" timeout, but we don't.  Their reasoning is that its safe to do that since the window is there.  I don't think that's really safe, though, since the window can easily be closed over such a long time frame.

It seems to me if you really need to be doing work that takes over 5 minutes you should be doing it directly from the window or a dedicated Worker.  Trying to do this inline of a network request seems really weird to me.  For example, most network requests will timeout within this time frame anyway.
Thank you Ben.

It is a long running operation, since millions of datasets have to be fetched and maximum bulk size is about 100 datasets per request.

I think, I have to rework the implementation. To parallelize the requests we've started several workers. Maybe this was not the right approach.

Marking it a security issue was a mistake. Please excuse.
Flags: needinfo?(marko.seidenglanz)
(In reply to marko.seidenglanz from comment #7)
> I think, I have to rework the implementation. To parallelize the requests
> we've started several workers. Maybe this was not the right approach.

Ok.  Its possible I'm misreading it.  I'm curious how many simultaneous fetches you are getting at once.

There are some workloads that can reasonably take a long time in a fetch event.  Decoding a video stream that is being watched in real time could legitimately hold the worker alive for the length of the video (hours).  This requires stream support, though, which we have not implemented yet.

In this case, though, you are doing somewhat arbitrary work in a fetch event.  It seems like this work could (and probably should) be done from your main window and not the fetch event.  (Again, how would this work in a browser without service workers?)

So while we eventually need to relax our 5 minute limit on respondWith() holding the worker alive for the video case, I'm disinclined to do so now.  I think we will probably adjust that once streaming is implemented.
Andrew, can you remove the sec flag from this bug?  It was added by mistake.  See comment 7.
Flags: needinfo?(continuation)
Group: core-security
Flags: needinfo?(continuation)
Should we morph this bug into relaxing our 5 minute limit on respondWith()?

Marko, Streams support is tracked in bug 1128959 in case you're interested?
Flags: needinfo?(bkelly)
Ben, in cases where service workers are not available it is done inside the window using nearly the same code. We thought that service workers in combination with streams would be a good choice, because the operation continues though the window was already closed. In Firefox we still have another problem, because the blob size is limited to 800MB and so we eventually have to split the data.

Thank you Andrew. I'm interested.
Once streams are implemented we should add support to keep the service worker alive for longer periods if it is holding a `FetchEvent.respondWith()` and the FetchEvent's window is still open.  If the FetchEvent's window is closed, the `respondWith()` keep alive should be revoked.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(bkelly)
Summary: Service worker error "Script terminated by timeout" after 5.5 minutes without response → support long running service workers with FetchEvent.respondWith() while controlled window is open
Priority: -- → P3
I have an experimental build with bug 1128959 and bug 1204254 applied.  This let us actually stream response bodies from the service worker.  I test it with this demo site that infinitely streams data to a couple of iframes from a server and a SW:

https://html-sw-stream.glitch.me/

This more or less works, but after we time out the service worker the second iframe content is replaced with "corrupted content".
Severity: normal → S3
Attachment #9384358 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: