trait ShuffleExecutorComponents extends AnyRef
:: Private :: An interface for building shuffle support for Executors.
- Annotations
- @Private()
- Source
- ShuffleExecutorComponents.java
- Since
3.0.0
- Alphabetic
- By Inheritance
- ShuffleExecutorComponents
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def createMapOutputWriter(shuffleId: Int, mapTaskId: Long, numPartitions: Int): ShuffleMapOutputWriter
Called once per map task to create a writer that will be responsible for persisting all the partitioned bytes written by that map task.
Called once per map task to create a writer that will be responsible for persisting all the partitioned bytes written by that map task.
- shuffleId
Unique identifier for the shuffle the map task is a part of
- mapTaskId
An ID of the map task. The ID is unique within this Spark application.
- numPartitions
The number of partitions that will be written by the map task. Some of these partitions may be empty.
- abstract def initializeExecutor(appId: String, execId: String, extraConfigs: Map[String, String]): Unit
Called once per executor to bootstrap this module with state that is specific to that executor, specifically the application ID and executor ID.
Called once per executor to bootstrap this module with state that is specific to that executor, specifically the application ID and executor ID.
- appId
The Spark application id
- execId
The unique identifier of the executor being initialized
- extraConfigs
Extra configs that were returned by
ShuffleDriverComponents#initializeApplication()
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def createSingleFileMapOutputWriter(shuffleId: Int, mapId: Long): Optional[SingleSpillShuffleMapOutputWriter]
An optional extension for creating a map output writer that can optimize the transfer of a single partition file, as the entire result of a map task, to the backing store.
An optional extension for creating a map output writer that can optimize the transfer of a single partition file, as the entire result of a map task, to the backing store.
Most implementations should return the default
Optional#empty()to indicate that they do not support this optimization. This primarily is for backwards-compatibility in preserving an optimization in the local disk shuffle storage implementation.- shuffleId
Unique identifier for the shuffle the map task is a part of
- mapId
An ID of the map task. The ID is unique within this Spark application.
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)