class StatsReportListener extends SparkListener with Logging
:: DeveloperApi :: Simple SparkListener that logs a few summary statistics when each stage completes.
- Annotations
- @DeveloperApi()
- Source
- StatsReportListener.scala
- Alphabetic
- By Inheritance
- StatsReportListener
- Logging
- SparkListener
- SparkListenerInterface
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new StatsReportListener()
Type Members
- implicit class LogStringContext extends AnyRef
- Definition Classes
- Logging
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit
Called when the application ends
Called when the application ends
- Definition Classes
- SparkListener → SparkListenerInterface
- def onApplicationStart(applicationStart: SparkListenerApplicationStart): Unit
Called when the application starts
Called when the application starts
- Definition Classes
- SparkListener → SparkListenerInterface
- def onBlockManagerAdded(blockManagerAdded: SparkListenerBlockManagerAdded): Unit
Called when a new block manager has joined
Called when a new block manager has joined
- Definition Classes
- SparkListener → SparkListenerInterface
- def onBlockManagerRemoved(blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit
Called when an existing block manager has been removed
Called when an existing block manager has been removed
- Definition Classes
- SparkListener → SparkListenerInterface
- def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated): Unit
Called when the driver receives a block update info.
Called when the driver receives a block update info.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onEnvironmentUpdate(environmentUpdate: SparkListenerEnvironmentUpdate): Unit
Called when environment properties have been updated
Called when environment properties have been updated
- Definition Classes
- SparkListener → SparkListenerInterface
- def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit
Called when the driver registers a new executor.
Called when the driver registers a new executor.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onExecutorBlacklisted(executorBlacklisted: SparkListenerExecutorBlacklisted): Unit
Called when the driver excludes an executor for a Spark application.
Called when the driver excludes an executor for a Spark application.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onExecutorBlacklistedForStage(executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit
Called when the driver excludes an executor for a stage.
Called when the driver excludes an executor for a stage.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onExecutorExcluded(executorExcluded: SparkListenerExecutorExcluded): Unit
Called when the driver excludes an executor for a Spark application.
Called when the driver excludes an executor for a Spark application.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onExecutorExcludedForStage(executorExcludedForStage: SparkListenerExecutorExcludedForStage): Unit
Called when the driver excludes an executor for a stage.
Called when the driver excludes an executor for a stage.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onExecutorMetricsUpdate(executorMetricsUpdate: SparkListenerExecutorMetricsUpdate): Unit
Called when the driver receives task metrics from an executor in a heartbeat.
Called when the driver receives task metrics from an executor in a heartbeat.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit
Called when the driver removes an executor.
Called when the driver removes an executor.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onExecutorUnblacklisted(executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit
Called when the driver re-enables a previously excluded executor.
Called when the driver re-enables a previously excluded executor.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onExecutorUnexcluded(executorUnexcluded: SparkListenerExecutorUnexcluded): Unit
Called when the driver re-enables a previously excluded executor.
Called when the driver re-enables a previously excluded executor.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onJobEnd(jobEnd: SparkListenerJobEnd): Unit
Called when a job ends
Called when a job ends
- Definition Classes
- SparkListener → SparkListenerInterface
- def onJobStart(jobStart: SparkListenerJobStart): Unit
Called when a job starts
Called when a job starts
- Definition Classes
- SparkListener → SparkListenerInterface
- def onNodeBlacklisted(nodeBlacklisted: SparkListenerNodeBlacklisted): Unit
Called when the driver excludes a node for a Spark application.
Called when the driver excludes a node for a Spark application.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onNodeBlacklistedForStage(nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit
Called when the driver excludes a node for a stage.
Called when the driver excludes a node for a stage.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onNodeExcluded(nodeExcluded: SparkListenerNodeExcluded): Unit
Called when the driver excludes a node for a Spark application.
Called when the driver excludes a node for a Spark application.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onNodeExcludedForStage(nodeExcludedForStage: SparkListenerNodeExcludedForStage): Unit
Called when the driver excludes a node for a stage.
Called when the driver excludes a node for a stage.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onNodeUnblacklisted(nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit
Called when the driver re-enables a previously excluded node.
Called when the driver re-enables a previously excluded node.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onNodeUnexcluded(nodeUnexcluded: SparkListenerNodeUnexcluded): Unit
Called when the driver re-enables a previously excluded node.
Called when the driver re-enables a previously excluded node.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onOtherEvent(event: SparkListenerEvent): Unit
Called when other events like SQL-specific events are posted.
Called when other events like SQL-specific events are posted.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onResourceProfileAdded(event: SparkListenerResourceProfileAdded): Unit
Called when a Resource Profile is added to the manager.
Called when a Resource Profile is added to the manager.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onSpeculativeTaskSubmitted(speculativeTask: SparkListenerSpeculativeTaskSubmitted): Unit
Called when a speculative task is submitted
Called when a speculative task is submitted
- Definition Classes
- SparkListener → SparkListenerInterface
- def onStageCompleted(stageCompleted: SparkListenerStageCompleted): Unit
Called when a stage completes successfully or fails, with information on the completed stage.
Called when a stage completes successfully or fails, with information on the completed stage.
- Definition Classes
- StatsReportListener → SparkListener → SparkListenerInterface
- def onStageExecutorMetrics(executorMetrics: SparkListenerStageExecutorMetrics): Unit
Called with the peak memory metrics for a given (executor, stage) combination.
Called with the peak memory metrics for a given (executor, stage) combination. Note that this is only present when reading from the event log (as in the history server), and is never called in a live application.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted): Unit
Called when a stage is submitted
Called when a stage is submitted
- Definition Classes
- SparkListener → SparkListenerInterface
- def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit
Called when a task ends
Called when a task ends
- Definition Classes
- StatsReportListener → SparkListener → SparkListenerInterface
- def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult): Unit
Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).
Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).
- Definition Classes
- SparkListener → SparkListenerInterface
- def onTaskStart(taskStart: SparkListenerTaskStart): Unit
Called when a task starts
Called when a task starts
- Definition Classes
- SparkListener → SparkListenerInterface
- def onUnpersistRDD(unpersistRDD: SparkListenerUnpersistRDD): Unit
Called when an RDD is manually unpersisted by the application
Called when an RDD is manually unpersisted by the application
- Definition Classes
- SparkListener → SparkListenerInterface
- def onUnschedulableTaskSetAdded(unschedulableTaskSetAdded: SparkListenerUnschedulableTaskSetAdded): Unit
Called when a taskset becomes unschedulable due to exludeOnFailure and dynamic allocation is enabled.
Called when a taskset becomes unschedulable due to exludeOnFailure and dynamic allocation is enabled.
- Definition Classes
- SparkListener → SparkListenerInterface
- def onUnschedulableTaskSetRemoved(unschedulableTaskSetRemoved: SparkListenerUnschedulableTaskSetRemoved): Unit
Called when an unschedulable taskset becomes schedulable and dynamic allocation is enabled.
Called when an unschedulable taskset becomes schedulable and dynamic allocation is enabled.
- Definition Classes
- SparkListener → SparkListenerInterface
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- def withLogContext(context: HashMap[String, String])(body: => Unit): Unit
- Attributes
- protected
- Definition Classes
- Logging
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)