1126922 - Repeated needles in ES

Status: RESOLVED FIXED

(Webtools Graveyard :: DXR, defect)

Description

[Erik Rose [:erik][:erikrose]]

For example, look at these search results for http://33.33.33.77:8000/mozilla-central/search?q=%2Bmember%3Amozilla%3A%3Aa11y%3A%3AAccessibleWrap:

  u'c_var': [{u'end': 23,
              u'name': u'mAtkObject',
              u'qualname': u'mozilla::a11y::AccessibleWrap::mAtkObject',
              u'start': 13},
             {u'end': 23,
              u'name': u'mAtkObject',
              u'qualname': u'mozilla::a11y::AccessibleWrap::mAtkObject',
              u'start': 13},
             {u'end': 23,
              u'name': u'mAtkObject',
              u'qualname': u'mozilla::a11y::AccessibleWrap::mAtkObject',
              u'start': 13},
             {u'end': 23,
              u'name': u'mAtkObject',
              u'qualname': u'mozilla::a11y::AccessibleWrap::mAtkObject',
              u'start': 13},
             {u'end': 23,
              u'name': u'mAtkObject',
              u'qualname': u'mozilla::a11y::AccessibleWrap::mAtkObject',
              u'start': 13},
             {u'end': 23,
              u'name': u'mAtkObject',
              u'qualname': u'mozilla::a11y::AccessibleWrap::mAtkObject',
              u'start': 13},
             {u'end': 23,
              u'name': u'mAtkObject',
              u'qualname': u'mozilla::a11y::AccessibleWrap::mAtkObject',
              u'start': 13}],
  u'content': [u'  AtkObject *mAtkObject;'],
  u'ext': [u'h'],
  u'number': [83],
  u'path': [u'accessible/atk/AccessibleWrap.h']},
 {u'c_member': [{u'end': 27,
                 u'name': u'AccessibleWrap',
                 u'qualname': u'mozilla::a11y::AccessibleWrap',
                 u'start': 7},
                {u'end': 27,
                 u'name': u'AccessibleWrap',
                 u'qualname': u'mozilla::a11y::AccessibleWrap',
                 u'start': 7},
                {u'end': 27,
                 u'name': u'AccessibleWrap',
                 u'qualname': u'mozilla::a11y::AccessibleWrap',
                 u'start': 7},
                {u'end': 27,
                 u'name': u'AccessibleWrap',
                 u'qualname': u'mozilla::a11y::AccessibleWrap',
                 u'start': 7},
                {u'end': 27,
                 u'name': u'AccessibleWrap',
                 u'qualname': u'mozilla::a11y::AccessibleWrap',
                 u'start': 7},
                {u'end': 27,
                 u'name': u'AccessibleWrap',
                 u'qualname': u'mozilla::a11y::AccessibleWrap',
                 u'start': 7},
                {u'end': 27,
                 u'name': u'AccessibleWrap',
                 u'qualname': u'mozilla::a11y::AccessibleWrap',
                 u'start': 7}],
  u'c_type': [{u'end': 27,
               u'name': u'EAvailableAtkSignals',
               u'qualname': u'mozilla::a11y::AccessibleWrap::EAvailableAtkSignals',
               u'start': 7},
              {u'end': 27,
               u'name': u'EAvailableAtkSignals',
               u'qualname': u'mozilla::a11y::AccessibleWrap::EAvailableAtkSignals',
               u'start': 7},
              {u'end': 27,
               u'name': u'EAvailableAtkSignals',
               u'qualname': u'mozilla::a11y::AccessibleWrap::EAvailableAtkSignals',
               u'start': 7},
              {u'end': 27,
               u'name': u'EAvailableAtkSignals',
               u'qualname': u'mozilla::a11y::AccessibleWrap::EAvailableAtkSignals',
               u'start': 7},
              {u'end': 27,
               u'name': u'EAvailableAtkSignals',
               u'qualname': u'mozilla::a11y::AccessibleWrap::EAvailableAtkSignals',
               u'start': 7},
              {u'end': 27,
               u'name': u'EAvailableAtkSignals',
               u'qualname': u'mozilla::a11y::AccessibleWrap::EAvailableAtkSignals',
               u'start': 7},
              {u'end': 27,
               u'name': u'EAvailableAtkSignals',
               u'qualname': u'mozilla::a11y::AccessibleWrap::EAvailableAtkSignals',
               u'start': 7}],
  u'content': [u'  enum EAvailableAtkSignals {'],
  u'ext': [u'h'],
  u'number': [92],
  u'path': [u'accessible/atk/AccessibleWrap.h']}

Aside from wasting space, this probably makes the highlighter output awkward tag nestings.

Comment 1

[Erik Rose [:erik][:erikrose]]

FWIW, the moz-central ES index is 7.73GB with the repeats in.

Comment 2

[Erik Rose [:erik][:erikrose]]

7.31GB, I mean.

Comment 3

[Erik Rose [:erik][:erikrose]]

In the case of the first occurrence, it's a matter of the same answer being arrived at for various incarnations of a source file:

$ grep '83:13' 46798b74eaa00700609a7f20d57b2353049c7d04*
46798b74eaa00700609a7f20d57b2353049c7d04.1e9783d0dc9f085852ece242d3c84384cfef3f64.csv:variable,name,"mAtkObject",qualname,"mozilla::a11y::AccessibleWrap::mAtkObject",loc,"accessible/atk/AccessibleWrap.h:83:13",locend,"accessible/atk/AccessibleWrap.h:83:23",type,"AtkObject *",scopename,"AccessibleWrap",scopequalname,"mozilla::a11y::AccessibleWrap"
46798b74eaa00700609a7f20d57b2353049c7d04.9dd0694ffd7190f7389a3e0d2dace2f5151be3e9.csv:variable,name,"mAtkObject",qualname,"mozilla::a11y::AccessibleWrap::mAtkObject",loc,"accessible/atk/AccessibleWrap.h:83:13",locend,"accessible/atk/AccessibleWrap.h:83:23",type,"AtkObject *",scopename,"AccessibleWrap",scopequalname,"mozilla::a11y::AccessibleWrap"
46798b74eaa00700609a7f20d57b2353049c7d04.bf5508919eb69d1d095650309f4b9c3f6617f3b3.csv:variable,name,"mAtkObject",qualname,"mozilla::a11y::AccessibleWrap::mAtkObject",loc,"accessible/atk/AccessibleWrap.h:83:13",locend,"accessible/atk/AccessibleWrap.h:83:23",type,"AtkObject *",scopename,"AccessibleWrap",scopequalname,"mozilla::a11y::AccessibleWrap"
46798b74eaa00700609a7f20d57b2353049c7d04.c928d75cfb445923a6f1dc622af301e3d94451e4.csv:variable,name,"mAtkObject",qualname,"mozilla::a11y::AccessibleWrap::mAtkObject",loc,"accessible/atk/AccessibleWrap.h:83:13",locend,"accessible/atk/AccessibleWrap.h:83:23",type,"AtkObject *",scopename,"AccessibleWrap",scopequalname,"mozilla::a11y::AccessibleWrap"
46798b74eaa00700609a7f20d57b2353049c7d04.da141787fcf8ed7b90e77e6cd0826034a728a50d.csv:variable,name,"mAtkObject",qualname,"mozilla::a11y::AccessibleWrap::mAtkObject",loc,"accessible/atk/AccessibleWrap.h:83:13",locend,"accessible/atk/AccessibleWrap.h:83:23",type,"AtkObject *",scopename,"AccessibleWrap",scopequalname,"mozilla::a11y::AccessibleWrap"
46798b74eaa00700609a7f20d57b2353049c7d04.eb6f270cd7912cb602fd4ca0dcad55d5ae450ccd.csv:variable,name,"mAtkObject",qualname,"mozilla::a11y::AccessibleWrap::mAtkObject",loc,"accessible/atk/AccessibleWrap.h:83:13",locend,"accessible/atk/AccessibleWrap.h:83:23",type,"AtkObject *",scopename,"AccessibleWrap",scopequalname,"mozilla::a11y::AccessibleWrap"
46798b74eaa00700609a7f20d57b2353049c7d04.ee2f2f5e21901c91898e807adf671a9877c1dc29.csv:variable,name,"mAtkObject",qualname,"mozilla::a11y::AccessibleWrap::mAtkObject",loc,"accessible/atk/AccessibleWrap.h:83:13",locend,"accessible/atk/AccessibleWrap.h:83:23",type,"AtkObject *",scopename,"AccessibleWrap",scopequalname,"mozilla::a11y::AccessibleWrap"

Comment 4

[[github robot]]

Commit pushed to es at https://github.com/mozilla/dxr

https://github.com/mozilla/dxr/commit/dc898508784b5b535acb1c3b1fb978e52f677100
Filter out duplicates of condensed lines. Fixes bug 1126922.

Comment 5

[[github robot]]

Commit pushed to master at https://github.com/mozilla/dxr

https://github.com/mozilla/dxr/commit/dc898508784b5b535acb1c3b1fb978e52f677100
Filter out duplicates of condensed lines. Fixes bug 1126922.