381420 - clarify usage of AUS2 update system for Sunbird 0.7 release

Status: VERIFIED FIXED

(Release Engineering :: General, defect)

Description

[Stefan Sitter [:ssitter]]

The following topic has been raised by J. Paul Reed (preed) and needs to be discussed before Sunbird 0.5 release:

I wanted to ask what the plan was for AUS2 usage for Sunbird. I guess Sunbird has been using aus2.m.o for nightly updates, but release updates are done differently (using patcher2), and we currently have no partitioning on the server for release snippets, so we'd have to post Sunbird's snippets, which puts an bottleneck on your release process.

There were discussions of standing up a community aus2 server, that would sit on the same network/infrastructure as the aus2 server, but
have different access protocols for it. 

What do you think? Will that pose problems?

let us know,
preed
--
J. Paul Reed
Build/Release Engineer - The Mozilla Corporation

Comment 1

[Stefan Sitter [:ssitter]]

Sunbird currently uses the following URL for app.update.url:

    https://aus2.mozilla.org/update/2/%PRODUCT%/%VERSION%/%BUILD_ID%/
    %BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/update.xml

How should the URL look like if there would be a different community server with a different access protocol?

Comment 2

[cmtalbert]

(In reply to comment #1)
> Sunbird currently uses the following URL for app.update.url:
> 
>     https://aus2.mozilla.org/update/2/%PRODUCT%/%VERSION%/%BUILD_ID%/
>     %BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/update.xml
> 
> How should the URL look like if there would be a different community server
> with a different access protocol?
> 
From what I understand, we will use the same access protocol with a different URL.  The aus server we'd be talking to would be specific community aus box, but it would still be an aus box.  Paul Reed, can you comment about what we need to change?

Comment 3

[cmtalbert]

Daniel, I think this should block 0.5. (adding him to CC).

Comment 4

[J. Paul Reed [:preed]]

(In reply to comment #2)

> From what I understand, we will use the same access protocol with a different
> URL.  The aus server we'd be talking to would be specific community aus box,
> but it would still be an aus box.  Paul Reed, can you comment about what we
> need to change?

The protocol would be the same; in fact, everything would be the same, except for the server name.

And since we brought this up, I don't want to block/hinder the Sunbird 0.5 release over it.

There are a couple of ways I can think of to deal with this:

1. Keep everything as is; for Sunbird 0.6, we'll post the appropriate update snippets on aus2.m.o, and those updates will contain a change to the default config, so 0.6 users aus2-sunbird.m.o or something like that.

Pro: nothing changes for 0.5
Con: I'm not 100% sure that making the change to the aus2 server in an update will work. I can't imagine any reason it wouldn't, but there are some settings that are persisted across prefs/installs when it comes to update, and we'd want to confirm that this should work.

2. Create a CNAME for aus2-sunbird (or aus2-community, or...) in DNS, and point the builds at that. For now, that would redirect to aus2.m.o.

Pros: Gets us moving on this change, and should require no re-work for 0.6
Cons: Not sure the DNS redirect would work with SSL; it might, but would have to confirm with IT. Also, this changes something pretty fundamental right before a release, so I'm not sure you want to deal with it.

Given that we're preoccupied with the aus2 move, we may want to lean towards solution 1.

Comment 5

[Stefan Sitter [:ssitter]]

I don't know what solution would work better. After all you are the Mozilla Build/Release Engineer here :-)

I _think_ your proposal 1 should work. As I understand it there is no change to aus2 server during update. It's just that during the update a new default
config file containing the new app.update.url is created that is active after the update is completed and Sunbird is restarted.

Once the new aus2 server is set up it should be possible to test this using nightly builds, right? In worst case users would have to manually download the next release I guess.

Comment 6

[Daniel Boelzle [:dbo]]

Proposal 1 sounds most sensible to me for now. Even if we run into trouble then (which I doubt from reading ssitter's comment), we could comment that for 0.7. After all, it's still only a 0.5, not 4.0...

Comment 7

[Matthew (lilmatt) Willis]

+1 for proposal 1.

Proposal 2, while "future-proof", also commits us to supporting that special-case CNAME in DNS from not until whenever we decide to kill it.  My ex-sysadmin side hates that.

Comment 8

[Stefan Sitter [:ssitter]]

For Sunbird 0.7 we should have a decision on this topic.

Comment 9

[Seth Bindernagel]

> For Sunbird 0.7 we should have a decision on this topic.
> 

If we would like to purchase a server, I can investigate partitioning a piece of the server I purchase and placed in Amsterdam.  I will need to learn in better detail the exact resources that will be needed and used if we were to partition the localization server in Amsterdam.  Stefan, can you help there?(In reply to comment #8)

Comment 10

[Simon Paquet [:sipaq]]

definitely a must-have for 0.7

Comment 11

[Seth Bindernagel]

(In reply to comment #10)
> definitely a must-have for 0.7
> 

On July 10, I will be hosting a community proposal evaluation meeting to look at requests from the community on how we can support.  If we were to partition the l10n server in Amsterdam to help with this, I will need to fill out a formal proposal.  Can Simon, lilmatt, or Dan Boelzle help me on this?  Here is the template for the proposal:

http://wiki.mozilla.org/Community:CommunityProgram/SelectionProcess/ProposalTemplate

Presently, the template asks questions to elicit information about the request from an *individual*.  This is obviously something for the entire community.  Can someone look at the questions and then provide detailed responses, thinking of each question as it relates to the community request?

Please email me if you have questions.

Comment 12

[Robert Kaiser]

When you're thinking about getting a server in place to serve that for the community, could you look if it can be done in a way that also can serve the SeaMonkey project? We would like to get the update system working as well at least in time for our 2.0 release, and if we can, we should probably set up a system that can serve both Sunbird and SeaMonkey.

Comment 13

[Seth Bindernagel]

The VM for Sunbird is approved by Community Giving.  I have copied Matt Zeier on the email.  He is Mozilla IT who will help set up the VM. 

(In reply to comment #12)
> When you're thinking about getting a server in place to serve that for the
> community, could you look if it can be done in a way that also can serve the
> SeaMonkey project? We would like to get the update system working as well at
> least in time for our 2.0 release, and if we can, we should probably set up a
> system that can serve both Sunbird and SeaMonkey.
> 

KaiRo:  We will look at SeaMonkey ASAP.  I assume that a similar set up is being requested?

Comment 14

[Robert Kaiser]

(In reply to comment #13)
> KaiRo:  We will look at SeaMonkey ASAP.  I assume that a similar set up is
> being requested?

Yes, though it might be possible to share one machine/setup for both Sunbird and SeaMonkey - I guess those familiar with AUS know this better than me, though.

Comment 15

[J. Paul Reed [:preed]]

(In reply to comment #14)
> (In reply to comment #13)
> > KaiRo:  We will look at SeaMonkey ASAP.  I assume that a similar set up is
> > being requested?
> 
> Yes, though it might be possible to share one machine/setup for both Sunbird
> and SeaMonkey - I guess those familiar with AUS know this better than me,
> though.

Yeup... 1 VM should be able to handle all the community projects.

Comment 16

[cmtalbert]

So, what is the status on this bug?  I know the VM is set up, but updates are not working in the Sunbird 0.7pre nightlies.  Is bug 371816 the reason for this, or is there something we need to change on the Sunbird side?  I'm not well-versed in how AUS works, so I'm not even certain what details you need in order to determine the cause of the problem.  But, let me know and I'll attach the details here (or in a new bug).

Comment 17

[Alfred Peng]

Paul, since 1 VM can serve for all the community projects, any chance that Firefox/Thunderbird on Solaris can get online update support with that?

The current situation for Firefox/Thunderbird on Solaris is that the users have to download the tar/pkg file from Mozilla and untar/install it by themselves. It must be a good news for Solaris users to have this update feature with the community's help.

Comment 18

[Robert Kaiser]

Firefox/Thunderbird are not the "community projects" we are talking about here.
Let's get Sunbird going in this bug, figure out SeaMonkey in a followup once this is done - and make all other additional efforts to get any other software into such a system in different followup bugs, please.

Comment 19

[Simon Paquet [:sipaq]]

Matthew, anything we can do to get more traction on this bug?

Comment 20

[matthew zeier [:mrz]]

I'm waiting on a short window from rhelmer to clone the existing ausstage01 VM.  

Comment 21

[J. Paul Reed [:preed]]

mrz: I see cb-ausstage01 running on the community server, but do we have a URL for access this server yet? It'll need to have https, and I'm thinking would be something like aus2-community.mozilla.org or some such?

I'm open to other suggestions, of course.

Comment 22

[matthew zeier [:mrz]]

Are you asking for a specific hostname other than cb-ausstage01?  

Comment 23

[J. Paul Reed [:preed]]

(In reply to comment #22)
> Are you asking for a specific hostname other than cb-ausstage01?  

Well, the machine needs to be externally accessible via https with a name that's something like aus2-community.m.o or something like that.

I actually can't seem to get to that machine externally at all right now (it's behind the jumphost, yes? That's probably why?) We'll need that part, too, so clients can use this.

Comment 24

[J. Paul Reed [:preed]]

I just talked to mrz about this; to get this completed, we'll need:

1. A host-specific SSL cert for aus2-community.m.o (or whatever we call it; let me confirm that name)
2. https access enabled to external clients (I believe apache is already installed/configured, but might need to be reconfigured)

I *think* that's it...

Comment 25

[Simon Paquet [:sipaq]]

Martin, preed, 
what is the current status on this bug? We are nearing the 0.7 release and need to get this resolved.

Comment 26

[Martin Schröder [:mschroeder]]

1. We have to use non-community AUS2 server for 0.5 -> 0.7 update.
2. We need the new server address to configure the 0.7 release to use the community AUS2 server for 0.7 -> 0.9 update. (see http://mxr.mozilla.org/seamonkey/source/calendar/sunbird/app/profile/sunbird.js#107)

Comment 27

[cmtalbert]

https://bugzilla.mozilla.org/show_bug.cgi?id=381420#c24

Guys,  this got lost by being in the wrong component and assignee.  We need to get the items in comment 24 (see above link) resolved.  Then we should be able to move forward with using the community AUS2 server for the 0.7 Calendar release.

Reassigning to proper component and default assignees.

Comment 28

[matthew zeier [:mrz]]

This stalled after comment #24 until the actual hostname was determined.  After someone decides on that, I need a CSR so I can go get a certificate.

If I can get those, I can get you a host specific cert.

Comment 29

[J. Paul Reed [:preed]]

(In reply to comment #28)
> This stalled after comment #24 until the actual hostname was determined.  After
> someone decides on that, I need a CSR so I can go get a certificate.
> 
> If I can get those, I can get you a host specific cert.

Ok, cool; I'll ask about it at tomorrow's Build Team meeting.

Comment 30

[Martin Schröder [:mschroeder]]

(In reply to comment #29)
> Ok, cool; I'll ask about it at tomorrow's Build Team meeting.

Any news on the hostname decision?

Comment 31

[J. Paul Reed [:preed]]

(In reply to comment #30)
> (In reply to comment #29)
> > Ok, cool; I'll ask about it at tomorrow's Build Team meeting.
> 
> Any news on the hostname decision?

I put it on our agenda, but we didn't get to it. It should get carried over to next week's agenda.

If we need to move quicker on this, you might mark it a blocker, and reassign to build?

(Not quite sure what the new protocol is to get things like this addressed quicker.)

Comment 32

[J. Paul Reed [:preed]]

Talked about this in our build team meeting today; aus2-community.mozilla.org seems to be the consensus, so please get a cert with that hostname.

Comment 33

[Martin Schröder [:mschroeder]]

(In reply to comment #28)
> This stalled after comment #24 until the actual hostname was determined.  After
> someone decides on that, I need a CSR so I can go get a certificate.
> 
> If I can get those, I can get you a host specific cert.
> 

Matthew, how long does it take to get the host specific cert after the build team decided in favor of aus2-community.mozilla.org?

Comment 34

[cmtalbert]

I spoke with mrz.  We need to generate the CSR from the private key held on the webserver.  Daniel, can you ask Ause to do this?

We need to finalize this before 0.7 ships.

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

The VM setup for this is called cb-ausstage01, but I don't know if anyone from the Calendar team has access to this box. 

It may need to be updated for OS fixes; the VI reports that VMware tools are old.

Comment 36

[cmtalbert]

mrz et al:  When I try to access this machine, it doesn't appear to be up.  Our build guy reported the same when he attempted to log on to generate the CSR.  Could ya'll take a look?

Ping results:

h-144:~/Sites clint$ ssh aus2-community.mozilla.org
ssh: aus2-community.mozilla.org: No address associated with nodename
h-144:~/Sites clint$ ping aus2-community.mozilla.org 
ping: cannot resolve aus2-community.mozilla.org: Unknown host

Comment 37

[J. Paul Reed [:preed]]

The VM is up at cb-ausstage01.sj.mozilla.com.

(I'm guessing aus2-community.mozilla.org will likely be a CNAME when it's all said and done.)

Comment 38

[cmtalbert]

(In reply to comment #37)
> The VM is up at cb-ausstage01.sj.mozilla.com.
So, in order to log in and generate this CSR certificate that we need to ssh into cb-ausstage01.sh.mozilla.com?  Because I can't ping that either...

Comment 39

[matthew zeier [:mrz]]

icmp is probably disallowed.  Host is at 63.245.210.21 - never made it into external DNS.

You can generate a CSR from any host running openssl too.

Comment 40

[ause]

i played with openssl trying to create a CSR. i used as much defaults as possible, knowing nothing about any specific requirements that may exist.

ok, this leaves me with a couple of questions on how to proceed:
- i'm i supposed to create the private key myself and transfer it to the webserver?
- anything special that i should know about the CSR to be created?
- the machine seems to be up and running - but ssh access is denied for me (ause and calbld). so either way, using an existing key or creating a new one, how to get access?
- what about all those fancy inputs (country/state/challenge pw/etc.) when creating the CSR?

and lost again...

Comment 41

[matthew zeier [:mrz]]

(In reply to comment #40)
> i played with openssl trying to create a CSR. i used as much defaults as
> possible, knowing nothing about any specific requirements that may exist.
> 
> ok, this leaves me with a couple of questions on how to proceed:
> - i'm i supposed to create the private key myself and transfer it to the
> webserver?

Yes.

> - anything special that i should know about the CSR to be created?
> - the machine seems to be up and running - but ssh access is denied for me
> (ause and calbld). so either way, using an existing key or creating a new one,
> how to get access?

I have a vague understanding of who owns this host but if you can attach your ssh key I can get you added.

> - what about all those fancy inputs (country/state/challenge pw/etc.) when
> creating the CSR?

These are somewhat arbitrary fields but should reflect reality.  You probably don't want a signed key - if you do, then Apache won't start on its own until someone enters the passcode. 

Comment 42

[ause]

> > - anything special that i should know about the CSR to be created?
> > - the machine seems to be up and running - but ssh access is denied for me
> > (ause and calbld). so either way, using an existing key or creating a new one,
> > how to get access?
> 
> I have a vague understanding of who owns this host but if you can attach your
> ssh key I can get you added.
see Bug 386736

> 
> > - what about all those fancy inputs (country/state/challenge pw/etc.) when
> > creating the CSR?
> 
> These are somewhat arbitrary fields but should reflect reality.  You probably
> don't want a signed key - if you do, then Apache won't start on its own until
> someone enters the passcode. 
reality depends on the point of view ;)
as i understand this certificate, it's meant rather for the webserver than for me. so the data should be valid for the server, where my knowledge ends with the yet to come CNAME. if that's enough, fine.

Comment 43

[matthew zeier [:mrz]]

Server is physically located in San Jose, California, USA.  Mozilla's offices are in Mountain View, California, USA - either of those would be sufficient for your CSR.

Contact address should be someone who's responsible for the box and if noone can claim it, I'd use [e-mail].

The hostname you already know - aus2-community.mozilla.org.  I've added that CNAME but you can generate the CSR regardless. 

Comment 44

[Robert Kaiser]

Hmm, that process sounds a bit awkward to me... We're setting up a system for the whole community to use and a member of one of the communities needs to do all the hard work and put up a certificate that he privately created for a server that will serve the whole community?

Or will every community project hosting updates on this server need to create their own cert to run there?
Will everybody updating any community project get a big window that says "Unknown certificate" with a cert organization being something like "Sun" or "Sunbird" or such? Might also look strange...

Comment 45

[matthew zeier [:mrz]]

We setup the community giving program such that we donate hardware/network and related resources but don't "own" the host or have root on them.  Instead, a member of the community has root access and is responsible for OS patching, firewall filtering and other server side setups. It is a community resource not managed by Mozilla Corp but by the community.

What's missing is a CSR I can take to get a valid SSL certificate - you won't get any "unknown certificate" warnings since it'll be a valid SSL certificate signed by some existing root authority. 

From my perspective, what's missing is an owner for this machine or who has access to it or who needs access to it (it was a clone of an existing VM IIRC). Doesn't look like anyone's logged into it other than root. 

Who needs access (send me your key)?  Who's the "owner"?

Comment 46

[Robert Kaiser]

OK, if ause does set up the things in a way that works for all community projects, I guess that's fine, then.

I don't know yet what kind of access to what I need of he does this setup stuff (Thanks a lot!) and we are just using the server for SeaMonkey as well, but if/when I need shell access to the machine, my ssh key should be on l10n/web/mozilla cvs and even on stage, I'm not sure if it's on any bugzilla reports (yet).

Comment 47

[ause]

i need access. here is the key
https://bugzilla.mozilla.org/attachment.cgi?id=270740&action=edit

Comment 48

[Robert Kaiser]

I think I should have access at least, ause and me didn't yet decide on who should officially be the owner, but I guess us two will the the ones on whom this probably will fall back.
My dsa key is in attachment 228698 [details]

Comment 49

[matthew zeier [:mrz]]

(In reply to comment #47)
> i need access. here is the key
> https://bugzilla.mozilla.org/attachment.cgi?id=270740&action=edit
> 

Added your key. 

Comment 50

[matthew zeier [:mrz]]

(In reply to comment #48)
> My dsa key is in attachment 228698 [details]

Added yours too.  

I'd like if you could generate the CSR on your own but if you need help with that let me know.

Comment 51

[matthew zeier [:mrz]]

I'd like to get this bug resolved - have you guys been able to generate a CSR or do you need help with that?

Comment 52

[matthew zeier [:mrz]]

Closing.  Re-open or open a new bug when you have the CSR.

Comment 53

[ause]

find the CSR on that machine in ~root/mkkey

looks like aus2-community.mozilla.org isn't yet reachable from outside. this is fine while setting up the server but needs to be changed at some point.

Comment 54

[matthew zeier [:mrz]]

Added the following rules:

! aus2-community
permit tcp any host 63.245.210.21 eq 80
permit tcp any host 63.245.210.21 eq 443

Comment 55

[ause]

hmm, machine is reachable now but no word about the cert. anything wrong with the CSR i pointed you at?
should i reopen? ;)

Comment 56

[Stefan Sitter [:ssitter]]

I get the following error messages when checking for updates:

    aus2-community.mozilla.org:443 uses an invalid security certificate.
    The certificate is not trusted because it is self signed.
    The certificate is only valid for name cb-ausstage01.mozilla.org.
    (Error code: sec_error_untrusted_issuer)

and
    AUS: Update XML File Malformed (200)

What steps are missing to get aus2-community.mozilla.org working?

Comment 57

[matthew zeier [:mrz]]

In the process of setting ourselves up with GeoTrust so I can 

Comment 58

[matthew zeier [:mrz]]

Er, submitted too soon - attaching CSR and passing to Justin.

-----BEGIN CERTIFICATE REQUEST-----
MIIBpzCCARACAQAwZzELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
FjAUBgNVBAcTDU1vdW50YWluIFZpZXcxKzApBgNVBAMUImF1czItY29tbXVuaXR5
Lm1vemlsbGEub3JnX2tleS5wZW0wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGB
AK80NOhbzH+VtHgAdj6MNyyS77Tqi6ShdAIPSanUsCTQVAJviIwTA61W6SmC+ChW
jTgpwuda66mliXatB0etl3evTjTf72ox7qSu9HXRdhy4AejLvonhrWxb9JvA/qqW
Ma2TCbKhOQuQL4lb7fJY2DekkTxtogGD33EoinBIWzbvAgMBAAGgADANBgkqhkiG
9w0BAQQFAAOBgQANdokMeyFaGKkl3VeS7RSWlk1IxZs/FzjG2SdRtetxXEgB6Lsd
Tk4sr/B6RqszJctmwyM8eIDStIt86hG10SWcRPOsd5lUC4fyW9YdEFnc0XGw4UEr
l4MNG6Rj0CZBJhxjQkcD2TE3eR9z3RRi8DT9Lv87VLG3Iv/uwP13P3rDEw==
-----END CERTIFICATE REQUEST-----

Comment 59

[ause]

beside the cert stuff, there is still some scripting missing to make the nightly update snippets pushed by the tinderboxes available to the server process.
the webservice itself already looks fine. you may try manually with the following url which uses some snippets i inserted manually for testing:

https://aus2-community.mozilla.org/update/2/sunbird/0.5/2007061403/Darwin_Universal-gcc3/en-US/release/update.xml

Comment 60

[J. Paul Reed [:preed]]

(In reply to comment #59)
> beside the cert stuff, there is still some scripting missing to make the
> nightly update snippets pushed by the tinderboxes available to the server
> process.
> the webservice itself already looks fine. you may try manually with the
> following url which uses some snippets i inserted manually for testing:

We've been discussing this on IRC, but just to keep everyone in the loop: the nightly update script that Firefox and Thunderbird use probably isn't going to be very useful to anyone else.

It includes a bunch of dead/not-very-useful code when the design of AUS was... different. Instead of using it for Calendar, I'm thinking it would be more useful to make this part of the nightly build process, if we can, instead of having a separate (external) service that handles partial generation. I think this makes more sense for community projects for a number of reasons (not having to monitor/deal with an external service, less dependencies, etc.)

Comment 61

[Robert Kaiser]

Yes, making it part of the nightly build process (tinderbox?) sounds like a very good idea, also for SeaMonkey.

Comment 62

[ause]

makes sense - in general...
is there any chance to get approval for a this kind of change on the MOZILLA_1_8_BRANCH?

Comment 63

[Robert Kaiser]

I don't believe we need to patch the branch itself - the branch tinderboxen are running trunk tinderbox code.

Comment 64

[Justin Fitzhugh]

The csr is invalid - error I get back is "Common Name does not contain fully qualified domain name."

Comment 65

[ause]

regenerated csr with hopefully correct CN field

-----BEGIN CERTIFICATE REQUEST-----
MIIBnzCCAQgCAQAwXzELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
FjAUBgNVBAcTDU1vdW50YWluIFZpZXcxIzAhBgNVBAMTGmF1czItY29tbXVuaXR5
Lm1vemlsbGEub3JnMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvNDToW8x/
lbR4AHY+jDcsku+06oukoXQCD0mp1LAk0FQCb4iMEwOtVukpgvgoVo04KcLnWuup
pYl2rQdHrZd3r0403+9qMe6krvR10XYcuAHoy76J4a1sW/SbwP6qljGtkwmyoTkL
kC+JW+3yWNg3pJE8baIBg99xKIpwSFs27wIDAQABoAAwDQYJKoZIhvcNAQEEBQAD
gYEAQz0uaFQzxUm1lC696jwN8p1vkAU2+QpwqdsK0MJKaqjnyLpJl82tPnwmbVVM
ycxbD+0s0+vEFyfCB1SsvnWgo+1OLkvH3v/TYaemvntNCBJ1GB/prGU6XLjhKIWH
yJqgT21nvvmIvaDT0oLioMAEoUyHtmWJRCNAXKZB+uLvcOw=
-----END CERTIFICATE REQUEST-----

Comment 66

[Justin Fitzhugh]

Here is the cert.

-----BEGIN CERTIFICATE-----
MIIDYTCCAsqgAwIBAgIDBx8RMA0GCSqGSIb3DQEBBAUAMFoxCzAJBgNVBAYTAlVT
MRwwGgYDVQQKExNFcXVpZmF4IFNlY3VyZSBJbmMuMS0wKwYDVQQDEyRFcXVpZmF4
IFNlY3VyZSBHbG9iYWwgZUJ1c2luZXNzIENBLTEwHhcNMDcxMTA5MDAzNTI5WhcN
MDgxMTA5MDAzNTI5WjCB0DELMAkGA1UEBhMCVVMxIzAhBgNVBAoTGmF1czItY29t
bXVuaXR5Lm1vemlsbGEub3JnMRMwEQYDVQQLEwpHVDE3ODE1MzUwMTEwLwYDVQQL
EyhTZWUgd3d3Lmdlb3RydXN0LmNvbS9yZXNvdXJjZXMvY3BzIChjKTA3MS8wLQYD
VQQLEyZEb21haW4gQ29udHJvbCBWYWxpZGF0ZWQgLSBRdWlja1NTTChSKTEjMCEG
A1UEAxMaYXVzMi1jb21tdW5pdHkubW96aWxsYS5vcmcwgZ8wDQYJKoZIhvcNAQEB
BQADgY0AMIGJAoGBAK80NOhbzH+VtHgAdj6MNyyS77Tqi6ShdAIPSanUsCTQVAJv
iIwTA61W6SmC+ChWjTgpwuda66mliXatB0etl3evTjTf72ox7qSu9HXRdhy4AejL
vonhrWxb9JvA/qqWMa2TCbKhOQuQL4lb7fJY2DekkTxtogGD33EoinBIWzbvAgMB
AAGjgb0wgbowDgYDVR0PAQH/BAQDAgTwMB0GA1UdDgQWBBSjfY+WvZUreJZxofuo
eSYL7f+GXzA7BgNVHR8ENDAyMDCgLqAshipodHRwOi8vY3JsLmdlb3RydXN0LmNv
bS9jcmxzL2dsb2JhbGNhMS5jcmwwHwYDVR0jBBgwFoAUvqigdHJQa0S3ySPY+6j/
s1draGwwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQC
MAAwDQYJKoZIhvcNAQEEBQADgYEACal6KhIVxW2C1SNZdwnsz9+DNKndOc04kLiT
hSctrOFEetFslbs1sFF7DZqAIIJAHa7Nr8pWiIBOFjsQhWrrq+LrdN7SRFOrz2TF
PYswqUjc40vQKw7Jp5jLY4AM0Usj0zXEZfUDv5HQCY76/3qmJZLXPDr0/vJEIjDR
3YvINec=
-----END CERTIFICATE-----

Comment 67

[Justin Fitzhugh]

Cert is done, think this goes back to build/release for aus2 setup - let me know if I am wrong.

Comment 68

[ause]

cert is integrated and seems to work fine. i think it's time to finally close this bug.

Comment 69

[Damian Szczepanik]

it's ok for nightly builds 
mark as verified