1426332 - queries not running in FxA re:dash

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement)

Description

[Shane Tomlinson [:stomlinson]]

Since Dec 15th, FxA re:dash queries have stopped completing. Every time I kick off a query, they run for several hours before I give up. Ryan Kelly has tried killing off a bunch of the stalled queries, but new queries suffer from the same problem, they just hang.

Comment 1

[Ryan Kelly [:rfkelly]]

> Ryan Kelly has tried killing off a bunch of the stalled queries

This does seem to have helped *some*, inasmuch as the scheduled queries for updating our dashboards like [1] seem to have completed.  Shane, which query are you trying to run that is still giving problems?

[1] https://sql.telemetry.mozilla.org/queries/1751/source

Comment 2

[Ryan Kelly [:rfkelly]]

From the redshift control dashboard, it looks like it's this one:

  https://sql.telemetry.mozilla.org/queries/49975

Comment 3

[Ryan Kelly [:rfkelly]]

Revisiting this after the break, it looks like there's a bug in the above query which was causing redshift to churn something fierce:

    WITH
      treat_flow_events AS (
        SELECT
           ...
        from flow_events
        INNER JOIN flow_metadata on flow_events.flow_id = flow_events.flow_id

That INNER JOIN condition is a tautology, so this was taking the full cartesian product of the two tables, and apparently crushing redshift in the process.  I rebooted the cluster to purge the in-flight queries, and things *seem* to be running OK now, so I'm going to tentatively mark this resolved but keep on eye on things over the next few days.