945343 - Set up a home for Fossology

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Lonnen :lonnen]

To support the FFOS License Review we'd like to set up Fossology (http://www.fossology.org/), an existing open source tool for this purpose. We'll need a box provisioned, and Gerv is happy to administrate it once a machine is in place. There is no PII, so this may be a prime candidate for an AWS instance. Here are the rough requirements:

  * 8 GB of RAM
  * 1 TB of disk
  * Postgres
  * Apache
  * PHP
  * SVN
  * GIT
  * The server should be behind LDAP (basic auth or VPN)

What other information is needed? What are the next steps?

Comment 1

[Lonnen :lonnen]

Talked with Gerv today and we don't think this is a great candidate for paas because the file system dependency would require a bunch of modification. Open to discussion about that though.

Comment 2

[Chris Turra [:cturra]]

:lonnen - these resource requirements definitely don't make this app a paas requirement. since the disk/memory requirements are quite high, i'd like to get a better understanding why they're needed and what all this server is going to be used for? we'll have to justify that to the storage/virtualization folks.

we're happy to help here, but this is not something we're going to be able to get complete before the christmas break due to outstanding work on q4 tasks. it's going to need to be pushed into q1 for our group.

Comment 3

[Lonnen :lonnen]

Hey Gerv -- how much would it use on disk if the database was hosted somewhere else? We have a postgres cluster we could put the DB on, and that should bring these requirements down a lot.

Comment 4

[Gervase Markham [:gerv]]

(In reply to Chris Turra [:cturra] from comment #2)
> :lonnen - these resource requirements definitely don't make this app a paas
> requirement. 

Did you miss a "don't" in what lonnen said? If not, I'm not sure what you are saying here.

lonnen was suggesting that this app is *not* a good fit for the PaaS. Do you agree?

> since the disk/memory requirements are quite high, i'd like to
> get a better understanding why they're needed and what all this server is
> going to be used for? we'll have to justify that to the
> storage/virtualization folks.

The way this software works is that you upload an archive of your entire source tree. For Firefox OS, that is certainly hundreds of MB, possibly a GB. It then unpacks all the archives and places each file in the filesystem. After that, it runs "agents" over the data to gather information about it, and places the result in the database.

Therefore, the database is (I suspect) quite small, and the filesystem requirements are large. So moving the DB elsewhere probably doesn't have much effect on the requirements. Of course, if admining the system is easier with the DB on an existing Postgres cluster, then that's fine.

The Fossology page on their system requirements is here:
http://www.fossology.org/projects/fossology/wiki/SysConfig

"It is recommended that the area you choose to keep the repository in,
be a separate mount point with at least 10x the size of the unpacked
data you intend to scan. For a small system intended to just scan a
few small personal projects this might mean gigabytes, but for systems
intended for scanning large collections of software including Linux
distributions, this probably means hundreds of gigabytes to terabytes."

We aren't scanning a complete Linux distro, but we are scanning a whole OS - and probably multiple versions of it over time. So perhaps 4GB of RAM and 200GB of disk? If the disk is expandable later, we can certainly start smaller.

Gerv

Comment 5

[Lonnen :lonnen]

(In reply to Gervase Markham [:gerv] from comment #4)
> (In reply to Chris Turra [:cturra] from comment #2)
> > :lonnen - these resource requirements definitely don't make this app a paas
> > requirement. 
> 
> Did you miss a "don't" in what lonnen said? If not, I'm not sure what you
> are saying here.

Cturra is agreeing with my assessment that the resource requirements are too high for PAAS. In particular, the hard disk requirement. We couldn't figure out what was taking so much space -- I assumed it was the database storing a bunch of metadata about every line of source code. Given your most recent comments, that assumption is probably wrong.

Comment 6

[Gervase Markham [:gerv]]

Let me know if you need anything more from me here.

Gerv

Comment 7

[Chris Turra [:cturra]]

our group currently has limited resources available for new project work. to help us prioritize this request against existing tasks in our queue, can you please provide some insight into the following questions?

 - which top level goal(s) does this support?
 - what is the impact of this service to our products and/or users?
 - what is the expected outcome of this request?

Comment 8

[Gervase Markham [:gerv]]

Top-level goal: Firefox OS (the info from this project is strongly desired by potential partners)
Impact: not sure how to answer this
Expected outcome: server provided on which to install Fossology

Jishnu and Denelle can probably give you a sense of the relative urgency.

Gerv

Comment 9

[Lonnen :lonnen]

Impact is direct enough: without it there's increased risk with FFOS adoption, which will have a negative impact on partnerships and ultimately harm platform adoption

Comment 10

[Jishnu Menon :jishnu]

Hi-

This is a high priority project and it will take a while to do. The goal of the project is to reduce legal risk and partner strain over the open source licenses in our Firefox OS product and is critical path for us to get scale right this year for Firefox OS. 

I'd love some creative thought about how we get to what's necessary.

I've also pinged Sylvie about this bug to see if she can help.

Thanks
Jishnu

Comment 11

[Gervase Markham [:gerv]]

cturra: ping? This is labelled [2014q1], and we are now beyond Q1. Are you or is anyone else able to tell us when we might get this server?

Thanks,

Gerv

Comment 12

[Corey Shields [:cshields]]

Sorry, I'm late to the thread here.  Please help me out with some details:

What product and company goal is this directly supporting (I'm assuming FFOS from context, but just to be sure)?

What is the impact of having this service, who will be using it, and for how long will the service be needed?

What are the alternatives, and what is the impact of -not- having this service?

thanks!

Comment 13

[Gervase Markham [:gerv]]

(In reply to Corey Shields [:cshields] from comment #12)
> Sorry, I'm late to the thread here.  Please help me out with some details:
> 
> What product and company goal is this directly supporting (I'm assuming FFOS
> from context, but just to be sure)?

Answered in comment 8.

> What is the impact of having this service, who will be using it, and for how
> long will the service be needed?

Impact: we can do the work to reassure our partners that Firefox OS is license-clean.

Users: me to begin with (although I hope not to be stuck with it forever), some lawyers (certainly gpiper, probably others), hopefully other helpful community members in the future but that doesn't need to be done in v1

How long: ongoing. We may need to reference our historical decisions in the future, and the analysis of vX.Y will inform the analysis of VX.Y+1. But if it's behind some sort of login, and so not internet-facing, Heartbleed-like things won't require a firedrill.

> What are the alternatives, and what is the impact of -not- having this
> service?

One alternative would be for me to run a copy of Fossology locally, and be unable to communicate with Real Lawyers to collaborate on the work except by email. That would be pretty terrible. TBH, if this bug doesn't get fixed, I'd next look into spinning up my own Amazon AWS instance and expensing it. But given that the data stored may in part consist of legal analysis and conclusions which Mozilla might want to reference in the future, that doesn't sound awesome either.

Let me know if you need more info.

Gerv

Comment 14

[Gervase Markham [:gerv]]

Corey asked on IRC: "has a secreview been done on the app yet?"

No. I was hoping that if we put the app behind some kind of auth, a secreview could be avoided. This is a fairly large codebase which Mozilla has not secreviewed before and, while there are public installs of it, I genuinely don't know how much it has been designed with security in mind. Having to secreview it, and get changes made, may well slow this project down even more. If an auth requirement means we can avoid secreview, I'd like to go that route.

Can we go with auth instead?

Gerv

Comment 15

[Jason Crowe [:jd]]

Lets let the security folks weigh in on weather or not auth is sufficient for this.

Comment 16

[Jishnu Menon :jishnu]

Hi - can we please get this up and running this week?

Comment 17

[Yvan Boily [:ygjb][:yvan]]

Provided:
* This is behind authentication (Basic, LDAP, or VPN)
* Uses SSL for that authentication (if it is HTTP or Form based)
* Does not store PII
* Does not share a domain name with any other services

There shouldn't be a need for a secreview.  If any of these assumptions are incorrect, please ping me on IRC so we can keep this moving forward.

Comment 18

[Gervase Markham [:gerv]]

(In reply to Yvan Boily [:ygjb][:yvan] from comment #17)
> Provided:
> * This is behind authentication (Basic, LDAP, or VPN)
> * Uses SSL for that authentication (if it is HTTP or Form based)

That is what we are requesting.

> * Does not store PII

All it stores is source code (which may contain PII such as email addresses, but it's already public information if the code is open source) and licensing notes from us.

It's possible that in the future it may store code which is not, or not yet, open source. (I can't imagine a concrete instance coming up, but it's not impossible.) But if people think that source code contains PII, we can cross that bridge when we come to it.

> * Does not share a domain name with any other services

That's no problem for us. fossology.mozilla.org would be cool, but anything will do.

> There shouldn't be a need for a secreview. 

Wonderful :-)

Gerv

Comment 19

[Chris Turra [:cturra]]

:gerv and i had a project kick off meeting for fossology today. our plan is to move forward with this in aws. grabbing this bug and will provide updates as we progress.

Comment 20

[Sheeri Cabral [:sheeri]]

Great! let me know what my team can help out with, we know databases but we also know a bit about AWS and can be another warm body if needed for sysadmin-y type work.

Comment 21

[Chris Turra [:cturra]]

:gerv - as discussed in my email earlier today, we're ready to start testing the first pass at the new fossolgy service. all of our requirements have now been fulfilled: ssl, ldap auth, config management and database backups[1].

the service is accessible (behind ldap/basic auth) at: 

  https://fossology.allizom.org


[1] i have set a 7 day retention period on database backups, which are stored on the large ebs volume we're using for fossology data.

Comment 22

[Gervase Markham [:gerv]]

Both a production and dev service are now up and running. Thanks to cturra :-)

Gerv