Closed Bug 1443530 Opened 7 years ago Closed 5 years ago

Add operator for querying athena and putting intermediates into an s3 location

Categories

(Data Platform and Tools :: General, enhancement)

Product:
Data Platform and Tools
Component:
General
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: bugzilla, Unassigned)

Details

Idea for a low-friction, low-effort, sql-based etl platform: Add an operator that reads in queries from a repo, runs them in Athena and saves the intermediates to s3 The queries would run every day in DAGs after Glue The intermediates in s3 would be loaded in as tables in Athena to make them queryable The workflow for end-users would merely be to submit a pull request to the query repository with some additional metadata about which dataset the query works off of and what the intermediate table name should be.

Superseded by bigquery-etl, hooray

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
Component: Scheduling → General
You need to log in before you can comment on or make changes to this bug.