1224014 - user_agent column not long enough

Status: RESOLVED FIXED

(Input Graveyard :: Code Quality, defect)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We've seen a few instances of user agents that exceed the column size. We allow for 255 characters, but some are coming in longer than that.

We should:

1. increase the size of the user_agent field
2. add some code to truncate user agents that are too long before saving

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Here's a sample redacted one:

'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0; WUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; WTB=xxxxx; WUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; WTB=24086; WUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; WTB=xxxxx; WUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; WTB=24086; WUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; WTB=xxxxx) Gecko/20100101 Firefox/40.0'

I have no clue why that looks like that.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this since I've seen a couple.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Traceback:

Internal Server Error: /en-US/feedback/firefox/41.0.2/
Traceback (most recent call last):
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.38.0.31/newrelic/hooks/framework_django.py", line 497, in wrapper
    return wrapped(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/data/www/input.mozilla.org/input/fjord/feedback/views.py", line 383, in _persist_feedbackdev
    resp = fun(request, *args, **kwargs)
  File "/data/www/input.mozilla.org/input/fjord/feedback/views.py", line 482, in feedback_router
    *args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/utils/decorators.py", line 110, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/data/www/input.mozilla.org/input/fjord/feedback/views.py", line 305, in generic_feedback
    version, channel)
  File "/data/www/input.mozilla.org/input/fjord/base/utils.py", line 378, in _wrapped
    return fn(request, *args, **kwargs)
  File "/data/www/input.mozilla.org/input/fjord/base/utils.py", line 378, in _wrapped
    return fn(request, *args, **kwargs)
  File "/data/www/input.mozilla.org/input/fjord/feedback/views.py", line 229, in _handle_feedback_post
    opinion.save()
  File "/data/www/input.mozilla.org/input/fjord/feedback/models.py", line 322, in save
    super(Response, self).save(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/base.py", line 762, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/base.py", line 846, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/base.py", line 885, in _do_insert
    using=using, raw=raw)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/query.py", line 920, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/sql/compiler.py", line 974, in execute_sql
    cursor.execute(sql, params)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/utils.py", line 97, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.38.0.31/newrelic/hooks/database_dbapi2.py", line 22, in execute
    *args, **kwargs)
  File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 173, in execute
    self.errorhandler(self, exc, value)
  File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
DataError: (1406, "Data too long for column 'user_agent' at row 1")

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

The field is max_length=255 characters. I've only seen a few that were too long. This isn't a huge problem, but it'll be annoying if a sniffer ever decided to test that field out.

Questions:

1. can we assume that user agents longer than 255 are typically junk?
2. should we truncate the end? does that lose important bits for ones that are long but aren't junk?
3. what about removing the extra characters from the middle?

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

It looks like user agents from Windows IE (and possibly Edge, too) include lots of information about .NET and related bits which can explode user agent lengths.

I'm going to increase the size to 1000 characters and truncate just like we're now doing with urls.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

In a PR: https://github.com/mozilla/fjord/pull/705

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed in https://github.com/mozilla/fjord/commit/ecfeb5bfc271eddf47da35238ba26dd0cf7e1476

Pushed to prod just now. Marking as FIXED.