1362088 - SparkSession does not have access to Hive Metastore tables

Status: RESOLVED WORKSFORME

(Data Platform and Tools :: General, enhancement, P2)

Description

[Frank Bertsch [:frank]]

The new way of using spark is using a SparkSession:
```
import org.apache.spark.sql.SparkSession

val spark = SparkSession
   .builder()
   .appName("my-app")
   .getOrCreate()
```

Unfortunately, doing it this way the Hive tables are not accessible. `spark.sql("SELECT * FROM main_summary LIMIT 1")` errors out.

Comment 1

[Wesley Dawson [:whd]]

(In reply to Frank Bertsch [:frank] from comment #0)
> Unfortunately, doing it this way the Hive tables are not accessible.
> `spark.sql("SELECT * FROM main_summary LIMIT 1")` errors out.

Can you be more specific about the environment you are using and the error you are seeing? I just tried this from an interactive atmo cluster (emr 5.2.1) using spark-shell and it worked for me. I also saw enableHiveSupport at https://databricks.com/blog/2016/08/15/how-to-use-sparksession-in-apache-spark-2-0.html but didn't need to add it to get your example to work.

Comment 2

[Frank Bertsch [:frank]]

Works for me now.