488762 - Figure out how to figure out distribution (git/svn) problems

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations: Projects, task)

Description

[Jeremy Orem [:oremj]]

Need to improve speed and solve locking problems.

Comment 3

[Jeremy Orem [:oremj]]

Although not all issues are completely fixed updates should now be much quicker.

Comment 4

[Wil Clouser [:clouserw]]

what happened?

Comment 5

[matthew zeier [:mrz]]

I snagged demo Fusion-IO cards (http://www.fusionio.com/Products.aspx) and we're testing with those.  If anything, this will buy us some time.

Comment 6

[Michael Morgan [:morgamic]]

Any updates?

Comment 7

[matthew zeier [:mrz]]

Nothing more than comment #5.  Card is working great and I won't let them take it back.

Comment 8

[Jeremy Orem [:oremj]]

I think comment 3 and comment 5 might have resolved this bug.

Comment 9

[John J. Barton]

At comment #1 the getfirebug.com update time was duped to this bug. The update time has not improved as far as I can tell.

Comment 10

[Jeremy Orem [:oremj]]

Getfirebug shouldn't take longer than 30 minutes to sync out. You might be seeing the netscaler cache. If you append a random query string i.e., ?cache-buster12, so you see the new content?

Comment 11

[John J. Barton]

I guess I don't understand the concept: why does it take 30 minutes to update? It makes working on the site extremely painful.

Comment 12

[Jeremy Orem [:oremj]]

The svn cron runs every 15 minutes. The data syncs out to the US webheads within 10 min, but the NL/CN webheads have to wait for all US data to sync overseas and then sync to the those webheads.

Comment 13

[John J. Barton]

So what if we got rid of svn? Or: what can we do to get staging service with updates in the seconds range?

Comment 14

[matthew zeier [:mrz]]

(In reply to comment #13)
> So what if we got rid of svn? Or: what can we do to get staging service with
> updates in the seconds range?

svn might very well run very fast but it only updates 4 times an hour and the web servers only update 6 times an hour.  Amsterdam takes a bit longer.

Staging sites are only in the US and probably sync at a different clock interval.

Comment 15

[Michael Morgan [:morgamic]]

John - we try to avoid doing rapid iterative changes on production sites and do updates in batches on Tuesday or Thursday nights if possible.  It reduces the risk of having regressions and ensures quality for users.  Updates during business hours are possible but updating during off-peak hours has less chance of affecting users in a negative way.

Our environments are usually laid out like this:
1) dev - hacking/testing/development, instant testing/verification (sometimes someone's laptop)
2) staging - group testing, QA on production hardware, auto-pull from SVN every 5-15 minutes
3) production - updated by IT only, less frequent

Comment 16

[Michael Morgan [:morgamic]]

(In reply to comment #13)
> So what if we got rid of svn? Or: what can we do to get staging service with
> updates in the seconds range?
It'd be much faster to do development in an environment where you can instantly test your changes -- you should be doing this before changes are committed.

Comment 17

[John J. Barton]

(In reply to comment #15)
> John - we try to avoid doing rapid iterative changes on production sites and do
> updates in batches on Tuesday or Thursday nights if possible.  It reduces the
> risk of having regressions and ensures quality for users.  Updates during
> business hours are possible but updating during off-peak hours has less chance
> of affecting users in a negative way.

Our experience is exactly the opposite. getfirebug.com has bad quality because it is so slow to update we give up. If we do make updates and they are wrong its takes forever to fix it.

> 
> Our environments are usually laid out like this:
> 1) dev - hacking/testing/development, instant testing/verification (sometimes
> someone's laptop)

This is the one I want. How to get it.

> 2) staging - group testing, QA on production hardware, auto-pull from SVN every
> 5-15 minutes

This is fine for the final result.

> 3) production - updated by IT only, less frequent

We don't have the resources to work this way.

Comment 18

[John J. Barton]

(In reply to comment #9)
> At comment #1 the getfirebug.com update time was duped to this bug. The update
> time has not improved as far as I can tell.

Ok I did an update and it was much faster thanks, more like the 30 minutes you mentioned.  For builds this is ok.

Comment 19

[Michael Morgan [:morgamic]]

John - that's the reason why we don't make updates in production all the time, and only update when we have tested on staging.  It's not good to constantly fix regressions on a user-facing app on the fly for the same reasons you wouldn't publish an RC build of an extension without testing it.

For dev, could use a VM or just set up a LAMP stack/install on your desktop or laptop and that'd do the trick?  Possibly set up a VM hosted by IT that you could have access to, but it's much faster to just edit, save, refresh on your local machine.

I'd be willing to help get the site set up on your local machine if you need help with that.

Comment 20

[John J. Barton]

(In reply to comment #19)
> John - that's the reason why we don't make updates in production all the time,
> and only update when we have tested on staging.  It's not good to constantly
> fix regressions on a user-facing app on the fly for the same reasons you
> wouldn't publish an RC build of an extension without testing it.

Sure I understand all of this, but every other staging server I have used updates in seconds not 30 minutes.  I'll set up to test locally.

Comment 21

[Michael Morgan [:morgamic]]

I think we have pretty different definitions of staging.