844256 - please pull a few lines from ship-it.mozilla.org http/app logs

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

We're hitting an intermittent issues on ship-it.m.o where the server seems to be returning very weird response. It would help us to debug if we had http + app logs from a few minutes on either side of:
2013-02-21 10:40:29
2013-02-22 09:12:10

The application log is in /data/www/ship-it-dev.allizom.org/release-kickoff/logs/kickoff.log, I think.

Comment 1

[[e-mail] (:bhearsum)]

Sorry, also from around 2013-02-20 03:35:16.

Comment 2

[Rick Bryce [:rbryce]]

These are the only lines from near the times you mention in the log file.  The apache error and access logs also don't show anything around those time stamps either.  Let me know what I else I can help with.

[[e-mail]1 logs]$ pwd
/data/www/ship-it-dev.allizom.org/release-kickoff/logs

[[e-mail]1 logs]$ tail -10 kickoff.log 
2013-02-12 14:30:49,409 - kickoff.add_xframe_options#22: Adding X-Frame-Options: SAMEORIGIN to response
2013-02-12 14:31:19,940 - kickoff.add_xframe_options#22: Adding X-Frame-Options: SAMEORIGIN to response
2013-02-12 14:31:50,457 - kickoff.add_xframe_options#22: Adding X-Frame-Options: SAMEORIGIN to response
2013-02-12 14:32:20,597 - kickoff.add_xframe_options#22: Adding X-Frame-Options: SAMEORIGIN to response
2013-02-20 08:14:31,570 - kickoff.add_xframe_options#22: Adding X-Frame-Options: SAMEORIGIN to response
2013-02-20 08:14:42,778 - kickoff.add_xframe_options#22: Adding X-Frame-Options: SAMEORIGIN to response
2013-02-20 08:26:37,773 - kickoff.add_xframe_options#22: Adding X-Frame-Options: SAMEORIGIN to response
2013-02-22 06:08:15,303 - kickoff.add_xframe_options#22: Adding X-Frame-Options: SAMEORIGIN to response
2013-02-22 06:08:27,273 - kickoff.add_xframe_options#22: Adding X-Frame-Options: SAMEORIGIN to response
2013-02-22 06:13:23,481 - kickoff.add_xframe_options#22: Adding X-Frame-Options: SAMEORIGIN to response

Comment 3

[[e-mail] (:bhearsum)]

Huh, those are just normal operational messages. I wonder if the requests I'm looking at failed at the load balancer instead? Is it it possible to pull logs from it, too?

Thanks for your help!

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #3)
> Huh, those are just normal operational messages. I wonder if the requests
> I'm looking at failed at the load balancer instead? Is it it possible to
> pull logs from it, too?
> 
> Thanks for your help!

ping?

Comment 5

[Rick Bryce [:rbryce]]

(In reply to Ben Hearsum [:bhearsum] from comment #4)
> (In reply to Ben Hearsum [:bhearsum] from comment #3)
> > Huh, those are just normal operational messages. I wonder if the requests
> > I'm looking at failed at the load balancer instead? Is it it possible to
> > pull logs from it, too?
> > 
> > Thanks for your help!
> 
> ping?

Ben -
I will turn logging on today. This will have to be temporary, as we usually dont store log on the LBs for very long.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Rick Bryce [:rbryce] from comment #5)
> (In reply to Ben Hearsum [:bhearsum] from comment #4)
> > (In reply to Ben Hearsum [:bhearsum] from comment #3)
> > > Huh, those are just normal operational messages. I wonder if the requests
> > > I'm looking at failed at the load balancer instead? Is it it possible to
> > > pull logs from it, too?
> > > 
> > > Thanks for your help!
> > 
> > ping?
> 
> Ben -
> I will turn logging on today. This will have to be temporary, as we usually
> dont store log on the LBs for very long.

Ack. I'm not sure how helpful that will be. We don't have STR for the bugs I'm investigating :(. I'll certainly try to repro, but is it possible to leave logging on until we hit the errors again? (Could be days/weeks.)

Comment 7

[Rick Bryce [:rbryce]]

 > Ack. I'm not sure how helpful that will be. We don't have STR for the bugs
> I'm investigating :(. I'll certainly try to repro, but is it possible to
> leave logging on until we hit the errors again? (Could be days/weeks.)

Logging is enabled on the generic-http-dev VIP on the phx zeus cluster

We can leave it on as long as you need.  I didn't think about it before, but this is dev.  I can't image a huge amount of logs.  We can probably leave the logging on going forward.

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Rick Bryce [:rbryce] from comment #7)
>  > Ack. I'm not sure how helpful that will be. We don't have STR for the bugs
> > I'm investigating :(. I'll certainly try to repro, but is it possible to
> > leave logging on until we hit the errors again? (Could be days/weeks.)
> 
> Logging is enabled on the generic-http-dev VIP on the phx zeus cluster
> 
> We can leave it on as long as you need.  I didn't think about it before, but
> this is dev.  I can't image a huge amount of logs.  We can probably leave
> the logging on going forward.

Oh, I was looking for production logs. It looks like I gave you the wrong path to the application logs in comment #0, maybe that's why we didn't find anything there in the first place. Can sort through this quickly on IRC today?

Comment 9

[[e-mail] (:bhearsum)]

We just hit this error again today. Before the logs disappear, can you have a look for app + zeus logs between 07:26 PT and 07:29 PT?

Comment 10

[Rick Bryce [:rbryce]]

(In reply to Ben Hearsum [:bhearsum] from comment #9)
> We just hit this error again today. Before the logs disappear, can you have
> a look for app + zeus logs between 07:26 PT and 07:29 PT?

Nothing in the application log as before.  Looking through the generic cluster logs, I didnt see anything specific to "ship-it", but I may not have had the LB log verbose enough.  I changed to log output to show more data.  

For reference. ship-it.m.o access log can be found on pp-zlb[1-2]

Comment 11

[[e-mail] (:bhearsum)]

I've got access to the production servers now, I think I can pull any logs additional log entries I need on my own. Thanks for your help!