Closed Bug 801956 Opened 8 years ago Closed 8 years ago

Update mozillians push script with elastic search index command.

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P1)

x86_64
Linux

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: giorgos, Assigned: jd)

Details

Hi,

We need to add 

./manage.py cron index_all_profiles

command after a successful push in all dev, stage and production environments of mozillians.org

Thanks!
And please also run the command on dev.
Giorgos,

This is all done.  Let me know if you have any issues.

Cheers
Jd
Assignee: server-ops-webops → jcrowe
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Can you please verify that this command is in stage chief scripts? It doesn't appear in the chief log and things look broken.

Thanks!
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Bumping to p1, it will prevent us from pushing code today.
Priority: -- → P1
I see it in the stage push script 

+ python -W ignore ./manage.py cron index_all_profiles
+ checkretval
+ retval=0

And when I push stage manually, it works

I am looking into what Chief is doing
Brandon, let us know what you find. I checked on stage and did not see search results, this may be a code issue if the update scripts have all run.

However if the indexer did not get kicked off or if other script items have not run... it might be that.
Adding daniel as he helped us with this: https://bugzilla.mozilla.org/show_bug.cgi?id=804517

Not sure if related but we are still stuck.
Is there a way for me to bump the priority on this (without paging), this is preventing us from pushing code live.

I appreciate the help.
Here's a clarification of what we need done mostly based on comment #5.

Currently search (elastic search) is not working on the stage site and we are not sure if this is caused by the update script(s) not running, an issue like 804517, or our code.

We suspect its the deploy script or 804517 as everything is working on dev.

1. Confirm the stage update script has been run, going on the above comments we (devs) are not sure.

2. If the stage scripts have not been run, run them

3. Make sure chief is setup so the scripts are run during a deploy

Once the above is done and we are sure the update scripts have run we can see if search is working and go from there.
JD, ran this on stage and confirmed the update script completed, search still does not work.

Giorgos -> we need to investigate if code is causing the problem or something else.
Status: REOPENED → RESOLVED
Closed: 8 years ago8 years ago
Resolution: --- → FIXED
JD:

Can you please verify that:

The update script restarts the celery server 

and if not restart it and then re-run the indexing command?

Thanks
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Giorgos / JD:

I confirm that the update scripts for all three environments are configured to restart their respective celery services as their final action.
It would appear that generic-celery1.stage.seamicro.phx1.mozilla.com does not have a network flow to access the ES service on the ES staging server.  I will open a bug to have that rectified.

I am closing this bug as per comment #12.
Status: REOPENED → RESOLVED
Closed: 8 years ago8 years ago
Resolution: --- → FIXED
phrawzty believes this is caused by a network flow problem similar to what got us on dev.

Bug is here: https://bugzilla.mozilla.org/show_bug.cgi?id=807940
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.