Closed Bug 1661605 Opened 4 years ago Closed 4 years ago

move minidump-stackwalk to separate repo

Categories

(Socorro :: General, task, P2)

Product:
Socorro
Component:
General
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: willkg, Assigned: willkg)

References

Details

Attachments

(6 files, 1 obsolete file)

pr 1: bug 1661605: add circle ci configuration
61 bytes, text/x-github-pull-request
Details | Review
pr 3: bug 1661605: fix docker user/pass env vars
61 bytes, text/x-github-pull-request
Details | Review
pr 4: bug 1661605: fix image name in docker-compose.yml
61 bytes, text/x-github-pull-request
Details | Review
pr 5: bug 1661605: redo hub pushing to only push on tags
61 bytes, text/x-github-pull-request
Details | Review
pr 6: bug 1661605: add release script
61 bytes, text/x-github-pull-request
Details | Review
pr 5535: bug 1661605: move minidump-stackwalk to separate repository
53 bytes, text/x-github-pull-request
Details | Review

Right now, Socorro builds minidump-stackwalk binaries as part of its build process. However, they rarely change. Further, because they're tangled up in the Socorro dev environment, it's tricky to make minidump-stackwalk changes or debug it--I did some crazy workarounds, but they're awkward and wonky at best.

To make it easier to maintain minidump-stackwalk and also reduce the time it takes to build Socorro, it makes sense to go back to what we used to do where we built minidump-stackwalk and stored the results in a publicly available location and then pulled that down for Socorro builds. Further, to make it easier to maintain and debug, it makes sense to split it off into a separate repository with its own CI and all that.

This bug covers extracting minidump-stackwalk code from the Socorro repository and put it in a separate repository. CI for that repository should build a docker image with the binaries. Once we have that working, we can change Socorro's build process to pull down that image to get the binaries.

Process

There are a few steps to do this:

  1. build a repo that builds a Docker image
  2. move that to mozilla-services
  3. set up CI to upload images to dockerhub on new tags
  4. change socorro to download that image and get the binaries

Tags

I was thinking we can continue using the YYYY.MM.DD tags we use for Socorro, Antenna, and Tecken. It doesn't show the breakpad sha, but it's straightforward to find that from a tag.

Image size

We want to be thoughtful about the image size, but I'd like the image to have the source used, the binaries, and be debuggable. We're currently using python:3.8.5-slim which is based on debian:buster-slim. I think we should base the minidump-stackwalk image on debian:buster-slim and see how that goes.

I did the bulk of step 1 last night: https://github.com/willkg/minidump-stackwalk

It uses debian:buster-slim and generates an image that's in the 760mb range.

I'll talk with Brian about moving that to mozilla-services and then setting up CI to build and push docker images to docker hub on tag.

Assignee: nobody → willkg
Status: NEW → ASSIGNED
Depends on: 1661702

I got admin access, so that's all set.

Now I'm setting it up in Circle CI.

Depends on: 1662132

Did a test tag 2020.08.31:

tag 2020.08.31
Tagger: Will Kahn-Greene <willkg@mozilla.com>
Date:   Mon Aug 31 19:12:24 2020 -0400

`8833ec1`: bug 1661605: add release script (#6) (willkg)
`b449250`: bug 1661605: redo hub pushing to only push on tags (#5) (willkg)
`ef1b1ad`: bug 1661605: fix image name in docker-compose.yml (#4) (willkg)
`1b63ac6`: bug 1661605: fix docker user/pass env vars (#3) (willkg)
`04d8865`: bug 1661605: change from underscores to hyphens (willkg)
`55275a0`: bug 1661605: rename image to socorro_minidump_stackwalk (willkg)
`612d71f`: bug 1661605: add circle ci configuration (willkg)
`30c5ba1`: Fix building and reduce resulting image size (willkg)
`93ede87`: Initial commit (willkg)
-----BEGIN PGP SIGNATURE-----

iQEzBAABCgAdFiEEzY3I6/E2Vh8o9nPoGL3RpvBm7BEFAl9NhCcACgkQGL3RpvBm
7BGqAQgAmvdRLN3LmV3+JFRgNgc09TiwTuifOjGy85oBvO0rRZF+qOd9fnkakZof
1oveCu7aqJJ6WxIhAVvtCLX8PDv58Djygr5Al1hWkycoGlMxrqI/BAd/BwmWV8g/
1xUzQCt5AQ6Ddz0efBgoNSq/s0PG33sn3CGpvkrq1PK8lTBfo9fOA+hw5mc9BW+S
4d86ihJ2avtesfCxkE00JY2uIE7az54Tsuul0H3zfBpYDLXOh2dBJTj7+m+Pl4MU
EEzAGwFNM2M9IyDvtt6QGnC7GbtwBJjIHpliTXliHWcAUBp31jH3io03VU8u4+s6
Vs4ER4QVgbUg/umFILPdhJpjOBxoMg==
=/iR2
-----END PGP SIGNATURE-----

willkg merged PR #5535: "bug 1661605: move minidump-stackwalk to separate repository" in ff953d2.

This is the last step in moving minidump-stackwalk into a separate repository. This does a few big things for us:

  1. makes it easier for people to work on minidump-stackwalk without the complexity of the Socorro environment bits
  2. potentially makes minidump-stackwalk usable by others
  3. Socorro builds go back to not building stackwalker binaries which will speed up Circle CI builds; a long time ago we did this in taskcluster, but then I nixed that because I couldn't maintain it, and now we're back to something like that again
  4. moves us one step closer to switching to a different stackwalker

TADA!

This was pushed to prod in bug #1664250. Marking as FIXED.

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: