801956 - Update mozillians push script with elastic search index command.

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P1)

Description

[Giorgos Logiotatidis [:giorgos]]

Hi,

We need to add 

./manage.py cron index_all_profiles

command after a successful push in all dev, stage and production environments of mozillians.org

Thanks!

Comment 1

[Giorgos Logiotatidis [:giorgos]]

And please also run the command on dev.

Comment 2

[Jason Crowe [:jd]]

Giorgos,

This is all done.  Let me know if you have any issues.

Cheers
Jd

Comment 3

[Giorgos Logiotatidis [:giorgos]]

Can you please verify that this command is in stage chief scripts? It doesn't appear in the chief log and things look broken.

Thanks!

Comment 4

[Ben (:bensternthal)]

Bumping to p1, it will prevent us from pushing code today.

Comment 5

[Brandon Burton [:solarce]]

I see it in the stage push script 

+ python -W ignore ./manage.py cron index_all_profiles
+ checkretval
+ retval=0

And when I push stage manually, it works

I am looking into what Chief is doing

Comment 6

[Ben (:bensternthal)]

Brandon, let us know what you find. I checked on stage and did not see search results, this may be a code issue if the update scripts have all run.

However if the indexer did not get kicked off or if other script items have not run... it might be that.

Comment 7

[Ben (:bensternthal)]

Adding daniel as he helped us with this: https://bugzilla.mozilla.org/show_bug.cgi?id=804517

Not sure if related but we are still stuck.

Comment 8

[Ben (:bensternthal)]

Is there a way for me to bump the priority on this (without paging), this is preventing us from pushing code live.

I appreciate the help.

Comment 9

[Ben (:bensternthal)]

Here's a clarification of what we need done mostly based on comment #5.

Currently search (elastic search) is not working on the stage site and we are not sure if this is caused by the update script(s) not running, an issue like 804517, or our code.

We suspect its the deploy script or 804517 as everything is working on dev.

1. Confirm the stage update script has been run, going on the above comments we (devs) are not sure.

2. If the stage scripts have not been run, run them

3. Make sure chief is setup so the scripts are run during a deploy

Once the above is done and we are sure the update scripts have run we can see if search is working and go from there.

Comment 10

[Ben (:bensternthal)]

JD, ran this on stage and confirmed the update script completed, search still does not work.

Giorgos -> we need to investigate if code is causing the problem or something else.

Comment 11

[Giorgos Logiotatidis [:giorgos]]

JD:

Can you please verify that:

The update script restarts the celery server 

and if not restart it and then re-run the indexing command?

Thanks

Comment 12

[Daniel Maher [:phrawzty]]

Giorgos / JD:

I confirm that the update scripts for all three environments are configured to restart their respective celery services as their final action.

Comment 13

[Daniel Maher [:phrawzty]]

It would appear that generic-celery1.stage.seamicro.phx1.mozilla.com does not have a network flow to access the ES service on the ES staging server.  I will open a bug to have that rectified.

I am closing this bug as per comment #12.

Comment 14

[Ben (:bensternthal)]

phrawzty believes this is caused by a network flow problem similar to what got us on dev.

Bug is here: https://bugzilla.mozilla.org/show_bug.cgi?id=807940