1599094 - Removing performance signatures takes very long

Status: NEW

(Tree Management :: Treeherder: Infrastructure, defect)

Description

[Armen [:armenzg]]

We can remove over hundreds of thousands of perf datums entries within a couple of minutes yet removing perf signatures [1] can take over half and hour.

I believe we can simply delete all signatures older than a year and make this simpler:

DELETE FROM treeherder.performance_signature WHERE last_updated < '2018-11-26 00:00:00.000000';

Works for you?

[1]

[2019-11-25 13:44:41,300] WARNING [treeherder.model.management.commands.cycle_data:153] Removing performance signatures with missing jobs...
WARNING:Removing performance signatures with missing jobs...

Comment 1

[Armen [:armenzg]]

Here's the current code:

            # also remove any signatures which are (no longer) associated with
            # a job
            logger.warning('Removing performance signatures with missing jobs...')
            for signature in PerformanceSignature.objects.all():
                self._maybe_quit(started_at, max_overall_runtime)

                if not self.filter(
                        repository_id=signature.repository_id,  # leverages (repository, signature) compound index
                        signature_id=signature.id).exists():
                    signature.delete()

I'm OK if by some chance some Perf signatures were left behind that point to missing jobs.
The PerformanceSignature.objects.all() is extremely slow.
Querying signatures older than a year is so fast.