Closed Bug 1238740 Opened 8 years ago Closed 8 years ago

Recreate MDN virtualenv on every deploy and install compiled dependencies

Categories

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jezdez, Assigned: cliang)

References

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/2439] )

We've recently switched to peep for installing dependencies and would profit heavily from being able to recreate the virtualenv on every deploy, as well as install compiled extensions via peep.

As of pip 7 wheel files (zip files basically) are created automatically for every dependency that doesn't provide one from PyPI and are cached for the next time an install command is run. Since wheel files are zip based, recreating a virtualenv is essentially unzipping files and drastically faster once the wheel cache is primed.

More info: https://pip.pypa.io/en/stable/reference/pip_install/#wheel-cache

I've prepared a code change to make our chief deploy script remove the virtualenv before creating a new one on every deploy: https://github.com/mozilla/kuma/pull/3738

:cyliang You said you wanted to time the virtualenv run, please let me know if you need anything from me to push this out.
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/2439]
Flags: needinfo?(cliang)
Blocks: 1238703
Assignee: server-ops-webops → cliang
To relay from IRC: the time it takes to rebuild the virtualenv can be inferred from the chief files.  So, as long as we know which chief logs belong to "do not rebuild virtualenv" runs versus "rebuild virtualenv runs", we should have the information we need to see if this will have add unacceptable lag to the deployment process.
Flags: needinfo?(cliang)
I did a test run yesterday and found that peep is ignoring pip's native ability to cache files by implementing an own download mechanism. This means until we can switch to the soon to be released pip 8 release that has peep-like hashing abilities we can't merge https://github.com/mozilla/kuma/pull/3738. This is effectively on hold.
This has been deployed and works well.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.