1484715 - Spark 2.3 failing on ATMO due to stale Pandas version

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P1)

Description

[Frank Bertsch [:frank]]

Full Error:

Traceback (most recent call last):
  File "usage_report/usage_report.py", line 234, in <module>
    main()
  File "/mnt/anaconda2/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/mnt/anaconda2/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/mnt/anaconda2/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/anaconda2/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "usage_report/usage_report.py", line 169, in main
    usage_df = usage.toPandas()
  File "/usr/lib/spark/python/pyspark/sql/dataframe.py", line 1933, in toPandas
    require_minimum_pandas_version()
  File "/usr/lib/spark/python/pyspark/sql/utils.py", line 129, in require_minimum_pandas_version
    "your version was %s." % (minimum_pandas_version, pandas.__version__))
ImportError: Pandas >= 0.19.2 must be installed; however, your version was 0.18.1.