Type Members

1.  trait ShuffleDataIO extends AnyRef

   :: Private :: An interface for plugging in modules for storing and reading temporary shuffle data.

   :: Private :: An interface for plugging in modules for storing and reading temporary shuffle data.

   This is the root of a plugin system for storing shuffle bytes to arbitrary storage backends in the sort-based shuffle algorithm implemented by the org.apache.spark.shuffle.sort.SortShuffleManager. If another shuffle algorithm is needed instead of sort-based shuffle, one should implement org.apache.spark.shuffle.ShuffleManager instead.

   A single instance of this module is loaded per process in the Spark application. The default implementation reads and writes shuffle data from the local disks of the executor, and is the implementation of shuffle file storage that has remained consistent throughout most of Spark's history.

   Alternative implementations of shuffle data storage can be loaded via setting spark.shuffle.sort.io.plugin.class.

   Annotations

   @Private()

   Since

   3.0.0

2.  trait ShuffleDriverComponents extends AnyRef

   :: Private :: An interface for building shuffle support modules for the Driver.

   :: Private :: An interface for building shuffle support modules for the Driver.

   Annotations

   @Private()

3.  trait ShuffleExecutorComponents extends AnyRef

   :: Private :: An interface for building shuffle support for Executors.

   :: Private :: An interface for building shuffle support for Executors.

   Annotations

   @Private()

   Since

   3.0.0

4.  trait ShuffleMapOutputWriter extends AnyRef

   :: Private :: A top-level writer that returns child writers for persisting the output of a map task, and then commits all of the writes as one atomic operation.

   :: Private :: A top-level writer that returns child writers for persisting the output of a map task, and then commits all of the writes as one atomic operation.

   Annotations

   @Private()

   Since

   3.0.0

5.  trait ShufflePartitionWriter extends AnyRef

   :: Private :: An interface for opening streams to persist partition bytes to a backing data store.

   :: Private :: An interface for opening streams to persist partition bytes to a backing data store.

   This writer stores bytes for one (mapper, reducer) pair, corresponding to one shuffle block.

   Annotations

   @Private()

   Since

   3.0.0

6.  trait SingleSpillShuffleMapOutputWriter extends AnyRef

   Optional extension for partition writing that is optimized for transferring a single file to the backing store.

   Optional extension for partition writing that is optimized for transferring a single file to the backing store.

   Annotations

   @Private()

7.  trait WritableByteChannelWrapper extends Closeable

   :: Private :: A thin wrapper around a WritableByteChannel.

   :: Private :: A thin wrapper around a WritableByteChannel.

   This is primarily provided for the local disk shuffle implementation to provide a java.nio.channels.FileChannel that keeps the channel open across partition writes.

   Annotations

   @Private()

   Since

   3.0.0