Closed
Bug 1287749
Opened 8 years ago
Closed 8 years ago
Increase spark.kryoserializer.buffer.max value
Categories
(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)
Cloud Services Graveyard
Metrics: Pipeline
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: rvitillo, Assigned: rvitillo)
References
Details
(Whiteboard: [SvcOps])
User Story
We have seen some serialization errors in the wild, see below for a partial trace. at py4j.Gateway.invoke(Gateway.java:259) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:209) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.spark.SparkException: Kryo serialisation failed: Buffer overflow. Available: 0, required: 4232227. To avoid this, increase spark.kryoserializer.buffer.max value. at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:299) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:239) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Attachments
(1 file)
No description provided.
Assignee | ||
Updated•8 years ago
|
Whiteboard: [SvcOps]
Updated•8 years ago
|
Points: --- → 1
Priority: -- → P2
Assignee | ||
Updated•8 years ago
|
Assignee: nobody → rvitillo
Priority: P2 → P1
Assignee | ||
Comment 1•8 years ago
|
||
Attachment #8842364 -
Flags: review?(jthomas)
Updated•8 years ago
|
Attachment #8842364 -
Flags: review?(jthomas) → review+
Assignee | ||
Updated•8 years ago
|
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Updated•6 years ago
|
Product: Cloud Services → Cloud Services Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•