1510455 - Firefox Monitor GCP: emails fail to send and no sentry log?

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Luke Crouch [:groovecoder]]

I tried to sign up at https://stage.firefoxmonitor.nonprod.cloudops.mozgcp.net/ with [e-mail] and never received the verification email.

I looked for an error in https://sentry.prod.mozaws.net/operations/firefox-monitor-prod/ but I didn't see one. I bet we need to do the whole verify-this-sender dance on the new environment.

So, we'll need to set up a new SES_CONFIG_SET that sets the stage url callback for SES. And set SES_NOTIFICATION_LOG_ONLY=1. Then we'll need to trigger a verification email, look in the app log for the verification url, click it, and set SES_NOTIFICATION_LOG_ONLY= (i.e., blank)

I'm not sure why sentry didn't get this error? :( Other errors have shown up there no problem.

Comment 1

[Bob Micheletto [:bobm]]

(In reply to Luke Crouch [:groovecoder] from comment #0)


Will get a GCP specific configuration set and verification set up.

Comment 2

[Bob Micheletto [:bobm]]

(In reply to Bob Micheletto [:bobm] from comment #1)
> (In reply to Luke Crouch [:groovecoder] from comment #0)

Actually, our GCP stage already has SES_NOTIFICATION_LOG_ONLY set to 1.  We will need a GCP specific bounce handler configuration set, however, I'm going to redeploy with SES_NOTIFICATION_LOG_ONLY set to the empty string to see if that allows mail to work.

Comment 3

[Bob Micheletto [:bobm]]

(In reply to Bob Micheletto [:bobm] from comment #2)

> Actually, our GCP stage already has SES_NOTIFICATION_LOG_ONLY set to 1.  We
> will need a GCP specific bounce handler configuration set, however, I'm
> going to redeploy with SES_NOTIFICATION_LOG_ONLY set to the empty string to
> see if that allows mail to work.

Done.  Please test mail again.

Comment 4

[Luke Crouch [:groovecoder]]

Still don't see an email sending to https://www.mailinator.com/v3/index.jsp?zone=public&query=glom#/#inboxpane ...

There's another sentry error, that I'm not sure is related or not ...

"error insert into "subscribers" ("email", "fx_newsletter", "sha1", "signup_language", "verification_token", "verified") values ($1, $2, $3, $4, $5, $6) returning * - null value in column "id" violates not-null constraint"

Can we check that the new GCP `subscribers.id` column has `not null default nextval('subscribers_id_seq'::regclass)` and we have the corresponding subscribers_id_seq sequence to populate the `id` column.

Comment 5

[Bob Micheletto [:bobm]]

(In reply to Luke Crouch [:groovecoder] from comment #4)

> Can we check that the new GCP `subscribers.id` column has `not null default
> nextval('subscribers_id_seq'::regclass)` and we have the corresponding
> subscribers_id_seq sequence to populate the `id` column.

Here's the whole schema:
fxmonitorstage=> \d+ breached_hashes
 id        | integer | not null      | plain   |              |
 sha1_id   | integer |               | plain   |              |
 breach_id | integer |               | plain   |              |
 notified  | boolean | default false | plain   |              |

fxmonitorstage=> \d+ breaches
 id     | integer                | not null  | plain    |              |
 name   | character varying(255) |           | extended |              |
 meta   | json                   |           | extended |              |

fxmonitorstage=> \d+ email_hashes
 id                 | integer                | not null      | plain    |              |
 sha1               | character varying(255) |               | extended |              |
 email              | character varying(255) |               | extended |              |
 verification_token | character varying(255) |               | extended |              |
 verified           | boolean                | default false | plain    |              |

fxmonitorstage=> \d+ knex_migrations
 id             | integer                  | not null  | plain    |              |
 name           | character varying(255)   |           | extended |              |
 batch          | integer                  |           | plain    |              |
 migration_time | timestamp with time zone |           | plain    |              |

fxmonitorstage=> \d+ knex_migrations_lock
 is_locked | integer |           | plain   |              |

fxmonitorstage=> \d+ subscribers
 id                 | integer                  | not null               | plain    |              |
 sha1               | character varying(255)   |                        | extended |              |
 email              | character varying(255)   |                        | extended |              |
 verification_token | character varying(255)   |                        | extended |              |
 verified           | boolean                  | default false          | plain    |              |
 created_at         | timestamp with time zone | not null default now() | plain    |              |
 updated_at         | timestamp with time zone | not null default now() | plain    |              |
 fx_newsletter      | boolean                  | default false          | plain    |              |
 signup_language    | character varying(255)   |                        | extended |              |

Comment 6

[Bob Micheletto [:bobm]]

It looks like there may not be any sequences?

SELECT * FROM information_schema.sequences; 

sequence_catalog | sequence_schema | sequence_name | data_type | numeric_precision | numeric_precision_radix | numeric_scale | start_value | minimum_value | maximum_value | 
increment | cycle_option
------------------+-----------------+---------------+-----------+-------------------+-------------------------+---------------+-------------+---------------+---------------+-
----------+--------------
(0 rows)

Perhaps there's a better way to check that?

Comment 7

[Luke Crouch [:groovecoder]]

Got this debugged live. Closing.