Closed Bug 1344639 Opened 8 years ago Closed 8 years ago

EPEL mirror not available

Categories

(Cloud Services :: Operations: Miscellaneous, task)

Product:
Cloud Services
Component:
Operations: Miscellaneous
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rvitillo, Assigned: oremj)

Details

(Whiteboard: [SvcOps])

User Story

All our nightly ETL jobs have failed this morning because our EPEL mirror wasn't available and some libraries couldn't be installed (e.g. snappy-devel).

https://s3-us-west-2.amazonaws.com/net-mozaws-prod-us-west-2-ops-rpmrepo-mirror/epel/6/x86_64/debug/repodata/5f4997786493c9803ccb1f11da488180c557a507de98bf36a4fadf8b968243aa-primary.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"

https://rvitillo.pastebin.mozilla.org/8981141
No description provided.
Whiteboard: [SvcOps]
User Story: (updated)
The mirror is back online. Jason, any idea what happened?
Flags: needinfo?(jthomas)
We currently sync our EPEL mirror every 4 hours and it looks like the job that ran at 12AM only synced over repomd.xml[1]. This is problematic because if the job doesn't also sync the objects it references then we will see the failures above. It eventually synced the full set at the next run at 4AM [2]. :oremj can you take a look? [1] https://ops-master.jenkinsv2.prod.mozaws.net/job/yummirrors/job/epel/209/console [2] https://ops-master.jenkinsv2.prod.mozaws.net/job/yummirrors/job/epel/210/console
Assignee: nobody → oremj
Flags: needinfo?(jthomas) → needinfo?(oremj)
Severity: blocker → normal
Component: Metrics: Pipeline → Operations
This is the script that is used to sync the repo: mkdir -p epel/{6,7}/ rsync -av --delete rsync://mirrors.kernel.org/fedora-epel/6/x86_64/ ./epel/6/x86_64/ rsync -av --delete rsync://mirrors.kernel.org/fedora-epel/7/x86_64/ ./epel/7/x86_64/ aws --region us-west-2 s3 sync --exclude "*/repomd.xml" ./epel/ s3://net-mozaws-prod-us-west-2-ops-rpmrepo-mirror/epel/ aws --region us-west-2 s3 cp ./epel/7/x86_64/repodata/repomd.xml s3://net-mozaws-prod-us-west-2-ops-rpmrepo-mirror/epel/7/x86_64/repodata/repomd.xml aws --region us-west-2 s3 cp ./epel/6/x86_64/repodata/repomd.xml s3://net-mozaws-prod-us-west-2-ops-rpmrepo-mirror/epel/6/x86_64/repodata/repomd.xml aws --region us-west-2 s3 sync --delete ./epel/ s3://net-mozaws-prod-us-west-2-ops-rpmrepo-mirror/epel/ From the output of the console, it looks like nothing changed since the rsync didn't transfer anything in run 209. Looks like this failed using the debug epel repo and I see that the debug repo is not excluding repomd.xml and then copying it over after, like the x86_64 repos.
Actually, I think this broke in https://ops-master.jenkinsv2.prod.mozaws.net/job/yummirrors/job/epel/211/console: delete: s3://net-mozaws-prod-us-west-2-ops-rpmrepo-mirror/epel/6/x86_64/debug/repodata/5f4997786493c9803ccb1f11da488180c557a507de98bf36a4fadf8b968243aa-primary.sqlite.bz2 Completed 5 file(s) with ~2 file(s) remaining (calculating...) delete: s3://net-mozaws-prod-us-west-2-ops-rpmrepo-mirror/epel/6/x86_64/debug/repodata/d710f46fad1009bf6d3fd5ca72a05bbdb60b2f8631f02d81961e79b2d980bc84-filelists.xml.gz Completed 6 file(s) with ~1 file(s) remaining (calculating...) delete: s3://net-mozaws-prod-us-west-2-ops-rpmrepo-mirror/epel/6/x86_64/debug/repodata/fbc978b3125f6a9d0c1a3d36a381af8dbc0a38e1aa20964af9b898c8e71fadbc-filelists.sqlite.bz2 Completed 7 file(s) with ~0 file(s) remaining (calculating...) Completed 3.0 KiB/~3.0 KiB (3.4 KiB/s) with ~1 file(s) remaining (calculating...) upload: epel/6/x86_64/debug/repodata/repomd.xml to s3://net-mozaws-prod-us-west-2-ops-rpmrepo-mirror/epel/6/x86_64/debug/repodata/repomd.xml 5f49977 was deleted before repomd.xml was updated, so there was a small window when the old debug/.../repomd.xml was in place with its dependencies removed.
https://github.com/mozilla-services/cloudops-deployment/commit/d39aa6d03d567a831cbea451fc287e19dbc4e952 should fix the issue.
Status: NEW → RESOLVED
Closed: 8 years ago
Flags: needinfo?(oremj)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.