943078 - Create a new revision of the vagrant image that has a minimal BMO database dump (or separate image)

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Developer Box, enhancement)

Description

[David Lawrence [:dkl]]

To create a much smaller image for people to test with, we could update sanitizeme.pl to strip out all current bugs, comments, attachment data, etc. but leave the products, components, users, etc. This would significantly shrink the size of the image and still allow developers to create accurate patches for use on BMO.

dkl

Comment 1

[Mark Côté [:mcote]]

As discussed on IRC, I think for the next revision of the dev box, it should by default include this minimal db.  Anyone who wants the full db (probably a much smaller subsection of users) can grab it and load it into their dev box.

So in sum, I think we should provide

* minimal db
* full (sanitized) db
* dev box with minimal db

Perhaps there's somewhere better than people.m.c to host this... fubar do you know?

Comment 2

[Kendall Libby [:fubar] (he/him)]

How large are they, and how often will they be updated? Is this for BMO specifically, or bugzilla in general?

Comment 3

[Mark Côté [:mcote]]

This would be for BMO.  The intention is to provide data for researchers and to provide a quick way to get started for contributors, the interests of whom don't particularly overlap.

The full, sanitized db dump with attachments removed is about 7 GB.  The minimal db dump will be much smaller; I don't know exactly, but I would guess no more than a few MB.  The dev box will be only very slightly larger than the minimal db.  I don't see any reason to store older dumps, since the data will be contained in the newer dump.  So all told, to be safe, I'd say just 10-15 GB would do it.

Comment 4

[Mark Côté [:mcote]]

CCing mhoye, since he won't have to host these on his people account anymore (assuming he doesn't mind :).

Comment 5

[David Lawrence [:dkl]]

(In reply to Mark Côté ( :mcote ) from comment #3)
> This would be for BMO.  The intention is to provide data for researchers and
> to provide a quick way to get started for contributors, the interests of
> whom don't particularly overlap.
> 
> The full, sanitized db dump with attachments removed is about 7 GB.  The
> minimal db dump will be much smaller; I don't know exactly, but I would
> guess no more than a few MB.  The dev box will be only very slightly larger
> than the minimal db.  I don't see any reason to store older dumps, since the
> data will be contained in the newer dump.  So all told, to be safe, I'd say
> just 10-15 GB would do it.

The sanitized SQL dump is ~2G in size

http://people.mozilla.org/~mhoye/bugzilla/

The vagrant developer image is almost 7G in size but that will be reduced with the new minimal database design.

http://people.mozilla.org/~dkl/

So if we are to only host one of either at any given time and no need to retain previous dumps, then 10-15G should do it.

dkl

Comment 6

[Kendall Libby [:fubar] (he/him)]

ftp.m.o is probably the right place for something that size

Comment 7

[David Lawrence [:dkl]]

(In reply to Kendall Libby [:fubar] from comment #6)
> ftp.m.o is probably the right place for something that size

Do I have to request an account on ftp.m.o to upload the file? I have access to staging.mozilla.org:/pub/mozilla.org/webtools where I put upstream bugzilla.org tarballs for download. That uses ssh keys to get in. Is that the same place you are referring to?

dkl

Comment 8

[Kendall Libby [:fubar] (he/him)]

That's the upload side, yep.

Comment 9

[David Lawrence [:dkl]]

With the new smaller vagrant box size with the minimal db, I think this is fine to leave on people.mozilla.org/~dkl for now. Users can download the full sanitized dump from mhoye's page
and import if they desire.

dkl