Closed Bug 1281638 Opened 4 years ago Closed 4 years ago

Migrate aus4.mozilla.org as a proxy to aus5.mozilla.org

Categories

(Release Engineering Graveyard :: Applications: Balrog (backend), defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mostlygeek, Unassigned)

References

Details

In bug 1264801 aus3.mozilla.org was migrated as a proxy to aus5.mozilla.org. 

aus4.mozilla.org must also be a proxy since it has a different certificate pin than aus5. (bug 1281588). 

How to do this: 

- Add an ELB for aus4.mozilla.org to the proxy stack
- Same backend proxy servers will be used for both aus3 and aus4
- Upgrade the instance type of proxy to c4.large

Migration: 

- Get new stack up
- Migrate aus3.mozilla.org to new stack
- Alias balrog-proxy-aus4.r53-2.services.com -> new AUS4 ELB's DNS
- Inform webops of the change to aus4
- Change aus4.mozilla.org CNAME balrog-proxy-aus4.r53-2.services.mozilla.com
- Deployed a new proxy stack to prod 
- Migrated aus3.mozilla.org DNS to new AUS3 ELB 
- opted to use c3.large instead of c4.large. 
  The proxy servers have a 60s nginx cache that writes to ephemeral storage.

For the AUS4 migration: 

- notify webops/moc of the migration 
- update the R53 record in AWS
- should see an uptick in traffic on the servers and ELB
Migrated this morning. Traffic looks good to proxy servers. 
About 90% of requests are served from openresty's cache. 

There is a small percentage of traffic that looks strange. These appear in the logs after the aus4 migration. Getting about 200/minute:

1467052390.365 "-" "GET /update/4/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/%MOZ_VERSION%/update.xml HTTP/1.1" 400 166 "-" "-" 0.001 - "-" -
1467052392.159 "-" "GET /update/4/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/%MOZ_VERSION%/update.xml HTTP/1.1" 400 166 "-" "-" 0.000 - "-" -

What's strange is that the ELB or Openresty doesn't log the x-forwarded-for value for them. Otherwise the service is operating normally.
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
(In reply to Benson Wong [:mostlygeek] from comment #2)
> Migrated this morning. Traffic looks good to proxy servers. 
> About 90% of requests are served from openresty's cache. 
> 
> There is a small percentage of traffic that looks strange. These appear in
> the logs after the aus4 migration. Getting about 200/minute:
> 
> 1467052390.365 "-" "GET
> /update/4/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/
> %OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/%MOZ_VERSION%/update.xml
> HTTP/1.1" 400 166 "-" "-" 0.001 - "-" -
> 1467052392.159 "-" "GET
> /update/4/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/
> %OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/%MOZ_VERSION%/update.xml
> HTTP/1.1" 400 166 "-" "-" 0.000 - "-" -
> 
> What's strange is that the ELB or Openresty doesn't log the x-forwarded-for
> value for them. Otherwise the service is operating normally.

For posterity, these aren't new, and probably caused by a bad client change around Firefox for Android 37.0. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1282569, because Balrog can deal with these in a better way.
Product: Release Engineering → Release Engineering Graveyard
You need to log in before you can comment on or make changes to this bug.