Recreate MDN virtualenv on every deploy and install compiled dependencies

RESOLVED FIXED

Status

RESOLVED FIXED
3 years ago
2 months ago

People

(Reporter: jezdez, Assigned: cliang)

Tracking

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/2439] )

(Reporter)

Description

3 years ago
We've recently switched to peep for installing dependencies and would profit heavily from being able to recreate the virtualenv on every deploy, as well as install compiled extensions via peep.

As of pip 7 wheel files (zip files basically) are created automatically for every dependency that doesn't provide one from PyPI and are cached for the next time an install command is run. Since wheel files are zip based, recreating a virtualenv is essentially unzipping files and drastically faster once the wheel cache is primed.

More info: https://pip.pypa.io/en/stable/reference/pip_install/#wheel-cache

I've prepared a code change to make our chief deploy script remove the virtualenv before creating a new one on every deploy: https://github.com/mozilla/kuma/pull/3738

:cyliang You said you wanted to time the virtualenv run, please let me know if you need anything from me to push this out.

Updated

3 years ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/2439]
(Reporter)

Updated

3 years ago
Flags: needinfo?(cliang)
(Reporter)

Updated

3 years ago
Blocks: 1238703

Updated

3 years ago
Assignee: server-ops-webops → cliang
(Assignee)

Comment 1

3 years ago
To relay from IRC: the time it takes to rebuild the virtualenv can be inferred from the chief files.  So, as long as we know which chief logs belong to "do not rebuild virtualenv" runs versus "rebuild virtualenv runs", we should have the information we need to see if this will have add unacceptable lag to the deployment process.
Flags: needinfo?(cliang)
(Reporter)

Comment 2

3 years ago
I did a test run yesterday and found that peep is ignoring pip's native ability to cache files by implementing an own download mechanism. This means until we can switch to the soon to be released pip 8 release that has peep-like hashing abilities we can't merge https://github.com/mozilla/kuma/pull/3738. This is effectively on hold.
(Reporter)

Comment 3

3 years ago
This has been deployed and works well.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED

Updated

2 months ago
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.