Using a notebook will take 391 days to process this amount of data. There are a few options for to accomplish the processing: 1. High level would be cloud functions however there may be issues installing the required packages. 1. Using app engine or compute engine. This will require more effort due to managing the distributed aspect needed since the predictions have to complete in 1 day but may not be avoidable. As Jesse mentioned loading the batches from BigQuery may be to slow. In addition to that since the predictions will be run daily for a couple weeks it may make sense to pre-process the data into chunks.
Bug 1767979 Comment 1 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Using a notebook will take 38 days to process this amount of data. There are a few options for to accomplish the processing: 1. High level would be cloud functions however there may be issues installing the required packages. 1. Using app engine or compute engine. This will require more effort due to managing the distributed aspect needed since the predictions have to complete in 1 day but may not be avoidable. As Jesse mentioned loading the batches from BigQuery may be to slow. In addition to that since the predictions will be run daily for a couple weeks it may make sense to pre-process the data into chunks.