1053339 - fossology - second pass updates

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task)

Description

[Chris Turra [:cturra]]

after :gerv spent some time with fossology, he identified a couple requirements that were not previously identified that are now blockers due to the way the software works.

this morning we had a meeting to discuss these and have the following action items for our second pass:

 o system access needed:
   - update basic auth ldap (moco/mofo) *or* file (community)
   - shell mozilla only - backed ssh auth keys file
     - priv access outside of mozilla will need a committer agreement

 o installed from dpkg:
   - starting in dev...
     - cturra will removed the packages/fossology
     - gerv will then install from the repo he's working from
   - once complete, lets do this in prod!

Comment 1

[Chris Turra [:cturra]]

(In reply to Chris Turra [:cturra] from comment #0)
>  o installed from dpkg:
>    - starting in dev...
>      - cturra will removed the packages/fossology

this is now complete. :gerv - when you return from your pto, go ahead and apply your fossology repo install to the fossology-dev host. as part of this package removal, psql server was also deleted since it's a dependency of the fossology dpkg. since you have `sudo` access on this host, go ahead and `apt-get install` any deps you may need during the install process.

Comment 2

[Chris Turra [:cturra]]

(In reply to Chris Turra [:cturra] from comment #0)
>  o system access needed:
>    - update basic auth ldap (moco/mofo) *or* file (community)
>    - shell mozilla only - backed ssh auth keys file
>      - priv access outside of mozilla will need a committer agreement

this is now also complete. either LDAP or htpasswd basic auth can access the fossology prod web interface. note, htpasswd (file based) auth should only be used with community members who do *not* have LDAP accounts.

Comment 3

[Chris Turra [:cturra]]

(In reply to Chris Turra [:cturra] from comment #0)
> ...
>      - gerv will then install from the repo he's working from

^ as discussed in my last email, our next step here for :gerv to get fossology installed on the aws dev server from his repo.

Comment 4

[Gervase Markham [:gerv]]

Yes, sorry, I did do this, but forgot to paste the instructions here. This is what I did to get Fossology installed on the dev box:

sudo apt-get install git
git clone https://github.com/gerv/fossology

# Following http://www.fossology.org/projects/fossology/wiki/SysConfigSrc :

cd fossology
sudo su
utils/fo-installdeps

# Solve problem with Apache install, perhaps due to the fact Apache had been installed previously 
# - may or may not be necessary on another system.
# See http://www.ctheroux.com/2014/05/preinst-line-118-a2query-command-not-found/
dpkg --fsys-tarfile /var/cache/apt/archives/apache2_2.4.7-1ubuntu4.1_amd64.deb | tar xOf - ./usr/sbin/a2query > /usr/sbin/a2query
chmod 755 /usr/sbin/a2query
apt-get -f install
utils/fo-installdeps

make
make install

# Partly following http://www.fossology.org/projects/fossology/wiki/SysConfig :

# Create a shm.sh using the code from the above wiki page and run it to get the XXX values
sysctl -w kernel.shmmax=XXX
sysctl -w kernel.shmall=XXX
./shm.sh >> /etc/sysctl.conf

nano /etc/postgresql/9.3/main/postgresql.conf
# Make edits as the wiki page tells you under "Preparing Postgresql"
# But not the listen_addresses change - that sounds dangerous
# Ignore the "Database authentication" section
/etc/init.d/postgresql restart

/usr/local/lib/fossology/fo-postinstall

nano /etc/php5/apache2/php.ini
# Make edits as the wiki page tells you under "Configuring PHP"

nano /etc/apache2/sites-enabled/fossology.conf
# Change to make it appear on the root rather than at /repo/
# Change /usr/share to /usr/local/share in 2 places

nano /etc/apache2/apache2.conf
# Find the <Directory> block relating to /usr/share and make another identical one for 
# /usr/local/share

apache2ctl configtest
apache2ctl graceful

The Fossology front page should now be visible at the root of the domain.

/usr/local/etc/fossology/mods-enabled/scheduler/agent/fo_scheduler -t
/etc/init.d/fossology start


However, it seems it's not quite working right. The logs from an upload show that ununpack, the universal unpacker, is having trouble invoking dpkg.
http://fossology-dev.allizom.org/?mod=showjobs&upload=1&show=job&job=1

2014-09-09 10:42:07 ununpack [0] :: JOB[1].ununpack[3896.localhost]: "dpkg: warning: failed to open configuration file '/root/.dpkg.cfg' for reading: Permission denied"

I'm not sure what's going on there. But it doesn't seem to be able to unpack archives inside archives, which is a bit of a problem.

Gerv

Comment 5

[Chris Turra [:cturra]]

thanks for the update. glad to hear you made progress here.

(In reply to Gervase Markham [:gerv] from comment #4)
> 2014-09-09 10:42:07 ununpack [0] :: JOB[1].ununpack[3896.localhost]: "dpkg:
> warning: failed to open configuration file '/root/.dpkg.cfg' for reading:
> Permission denied"

i don't know anything about the application, but have a couple comments on this error. 1) it shouldn't be doing any work in the /root directory on this server and (2) the file it's requesting is not found there.

 root@ip-10-229-31-242:/usr/local/share/fossology# ls /root/.dpkg.cfg
 ls: cannot access /root/.dpkg.cfg: No such file or directory


default dpkg configs can be found in /etc/dpkg/, but those might not be appropriate if the application is looking for its own.

is this something you can reach out to the fossology community to get clarification on?

Comment 6

[Gervase Markham [:gerv]]

I'm going to pick this up again soon; busy at the moment.

Gerv

Comment 7

[Chris Turra [:cturra]]

now that some time has passed i just wanted to ping on this.

Comment 8

[Gervase Markham [:gerv]]

Hi Chris,

I appreciate your continued engagement. I'm currently working on another license-related issue and when that's done I'll return to this one. It may be that we don't use Fossology in the end, due to regular code WTFs and also (to be honest) the difficulty of this whole process (which I am not blaming on you).

The dev box still isn't quite working correctly, in the way outlined above, although I no longer see an error. So it's rather hard to debug. I had a bit of a go but couldn't get anywhere.

Gerv

Comment 9

[Chris Turra [:cturra]]

in light of this information, i am going to mark this bug as r/fixed. pls let me know if you need additional assistance here.

additionally, if you do decide not to proceed with fossoligy, please let me know so i can decomm the aws infrastructure i created for this project.