1448389 - Consider expanding `spark.databricks.queryWatchdog.maxQueryTasks` on shared_serverless

Status: RESOLVED DUPLICATE of bug 1522682

(Data Platform and Tools :: General, enhancement)

Description

[Frank Bertsch [:frank]]

Shared serverless is great because it reduces friction to using databricks. However, many people have been reporting this issue after launching large jobs.

We should see what the pros/cons are of raising this limit. The error is the following:

```
Error in SQL statement: SparkException: The query is not executed because it tries to launch 54794 tasks in a single stage, while the maximum allowed tasks one query can launch is 20000; this limit can be modified with configuration parameter "spark.databricks.queryWatchdog.maxQueryTasks".
```

Temporary workaround for users: Launch your own Spark cluster, and set it like the following in a notebook:

```
spark.conf.set("spark.databricks.queryWatchdog.maxQueryTasks", 100000)
```

Comment 1

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

I ran into this and discovered that I could just run "SET spark.databricks.queryWatchdog.maxQueryTasks = 100000" as Spark SQL to override the limit for the current session.