Profile Documentation

Counters

Name Significance Unit Description
GcCount STABLE & LOW UNIT Per-Impalad Counter: The number of GC collections that have occurred in the Impala process over the duration of the query. Reported by JMX.
GcTimeMillis STABLE & LOW TIME_MS Per-Impalad Counter: The amount of time spent in GC in the Impala process over the duration of the query. Reported by JMX.
GcNumWarnThresholdExceeded STABLE & LOW UNIT Per-Impalad Counter: The number of JVM process pauses that occurred in this Impala process over the duration of the query. Tracks the number of pauses at the WARN threshold. See JvmPauseMonitor for details. Reported by the JvmPauseMonitor.
GcNumInfoThresholdExceeded STABLE & LOW UNIT Per-Impalad Counter: The number of JVM process pauses that occurred in this Impala process over the duration of the query. Tracks the number of pauses at the INFO threshold. See JvmPauseMonitor for details. Reported by the JvmPauseMonitor.
GcTotalExtraSleepTimeMillis STABLE & LOW TIME_MS Per-Impalad Counter: The amount of time the JVM process paused over the duration of the query. See JvmPauseMonitor for details. Reported by the JvmPauseMonitor.
ExchangeScanRatio STABLE & LOW DOUBLE_VALUE The ratio between TotalScanByteSent and TotalBytesRead, i.e. the selectivity over all fragment instances that had a scan node in their plan.
TotalBytesSent STABLE & LOW BYTES The total number of bytes sent (across the network) by this query in exchange nodes. Does not include remote reads, data written to disk, or data sent to the client.
FinalizationTimer STABLE & LOW TIME_NS Total time spent in finalization (typically 0 except for INSERT into hdfs tables).
RowsRead STABLE & HIGH UNIT Number of top-level rows/tuples read from the storage layer, including those discarded by predicate evaluation. Used for all types of scans.
TotalCpuTime STABLE & HIGH TIME_NS Total CPU time (user + system) consumed by a query.
TotalBytesRead STABLE & HIGH BYTES Total number of bytes read by a query.
TotalRawHdfsReadTime STABLE & LOW TIME_NS Aggregate wall clock time across all Disk I/O threads in HDFS read operations.
TotalRawHdfsOpenFileTime STABLE & LOW TIME_NS Aggregate wall clock time spent across all Disk I/O threads in HDFS open operations.
ScanRangesComplete STABLE & LOW UNIT Number of scan ranges that have been completed by a scan node.
CollectionItemsRead STABLE & LOW UNIT Total number of nested collection items read by the scan. Only created for scans (e.g. Parquet) that support nested types.
NumDisksAccessed STABLE & LOW UNIT Number of distinct disks accessed by HDFS scan. Each local disk is counted as a disk and each type of remote filesystem (e.g. HDFS remote reads, S3) is counted as a distinct disk.
InitialRangeIdealReservation DEBUG BYTES Tracks stats about the ideal reservation for initial scan ranges. Use this to determine if the scan got all of the reservation it wanted. Does not include subsequent reservation increases done by scanner implementation (e.g. for Parquet columns).
InitialRangeActualReservation DEBUG BYTES Tracks stats about the actual reservation for initial scan ranges. Use this to determine if the scan got all of the reservation it wanted. Does not include subsequent reservation increases done by scanner implementation (e.g. for Parquet columns).
BytesReadLocal STABLE & LOW BYTES The total number of bytes read locally
BytesReadShortCircuit STABLE & LOW BYTES The total number of bytes read via short circuit read
BytesReadDataNodeCache STABLE & HIGH BYTES The total number of bytes read from data node cache
RemoteScanRanges STABLE & HIGH UNIT The total number of remote scan ranges
BytesReadRemoteUnexpected STABLE & LOW BYTES The total number of bytes read remotely that were expected to be local
CachedFileHandlesHitCount STABLE & LOW UNIT Total number of file handle opens where the file handle was present in the cache
CachedFileHandlesMissCount STABLE & LOW UNIT Total number of file handle opens where the file handle was not in the cache
MaxCompressedTextFileLength STABLE & LOW BYTES The size of the largest compressed text file to be scanned. This is used to estimate scanner thread memory usage.
ScannerIoWaitTime STABLE & LOW TIME_NS Total amount of time scanner threads spent waiting for I/O. This value can be compared to the value of ScannerThreadsTotalWallClockTime of MT_DOP = 0 scan nodes or otherwise compared to the total time reported for MT_DOP > 0 scan nodes. High values show that scanner threads are spending significant time waiting for I/O instead of processing data. Note that this includes the time when the thread is runnable but not scheduled.
DataCacheHitCount STABLE & HIGH UNIT Total count of data cache hit
DataCachePartialHitCount STABLE & HIGH UNIT Total count of data cache partially hit
DataCacheMissCount STABLE & HIGH UNIT Total count of data cache miss
DataCacheHitBytes STABLE & HIGH BYTES Total bytes of data cache hit
DataCacheMissBytes STABLE & HIGH BYTES Total bytes of data cache miss
BytesRead STABLE & HIGH BYTES Total bytes read from disk by a scan node.
TotalReadThroughput STABLE & LOW BYTES_PER_SECOND BytesRead divided by the total wall clock time that this scan was executing (from Open() to Close()). This gives the aggregate data is scanned.
BytesReadSeries UNSTABLE BYTES Time series of BytesRead that samples the BytesRead counter.
MaterializeTupleTime UNSTABLE TIME_NS Wall clock time spent materializing tuples and evaluating predicates.
NumScannerThreadsStarted DEBUG UNIT NumScannerThreadsStarted - the number of scanner threads started for the duration of the ScanNode. A single scanner thread will likely process multiple scan ranges.Meanwhile, more than one scanner thread can be spun up to process data from a single scan range. This is *not* the same as peak scanner thread concurrency because the number of scanner threads can fluctuate during execution of the scan.
RowBatchesEnqueued STABLE & LOW UNIT Number of row batches enqueued in the scan node's output queue.
RowBatchBytesEnqueued STABLE & LOW BYTES Number of bytes enqueued in the scan node's output queue.
RowBatchQueueGetWaitTime UNSTABLE TIME_NS Wall clock time that the fragment execution thread spent blocked waiting for row batches to be added to the scan node's output queue.
RowBatchQueuePutWaitTime UNSTABLE TIME_NS Aggregate wall clock time across all scanner threads spent blocked waiting for space in the scan node's output queue when it is full.
RowBatchQueuePeakMemoryUsage DEBUG BYTES Peak memory consumption of row batches enqueued in the scan node's output queue.
PeakScannerThreadConcurrency STABLE & LOW UNIT Peak number of executing scanner threads.
NumExternalDataSourceGetNext DEBUG UNIT The total number of calls to ExternalDataSource::GetNext()
NumColumns STABLE & LOW UNIT Number of columns that need to be read.
NumScannersWithNoReads STABLE & LOW UNIT Number of scanners that end up doing no reads because their splits don't overlap with the midpoint of any row-group/stripe in the file.
FooterProcessingTime STABLE & LOW TIME_NS Average and min/max time spent processing the footer by each split.
IoReadSyncRequest DEBUG UNIT Number of stream read request done in synchronized manner.
IoReadAsyncRequest DEBUG UNIT Number of stream read request done in asynchronized manner.
IoReadTotalRequest DEBUG UNIT Total number of stream read request.
IoReadSyncBytes DEBUG BYTES The total number of bytes read from streams in synchronized manner.
IoReadAsyncBytes DEBUG BYTES The total number of bytes read from streams in asynchronized manner.
IoReadTotalBytes DEBUG BYTES The total number of bytes read from streams.
IoReadSkippedBytes DEBUG BYTES The total number of bytes skipped from streams.
NumFileMetadataRead DEBUG UNIT The total number of file metadata reads done in place of rows or row groups / stripe iteration.
TotalRawHBaseReadTime STABLE & HIGH TIME_NS Aggregate wall clock time spent reading from HBase.
NumBackends STABLE & HIGH UNIT Number of backends running this query.
FiltersReceived STABLE & LOW UNIT Total number of filter updates received (always 0 if filter mode is not GLOBAL). Excludes repeated broadcast filter updates.
NumFragments STABLE & HIGH UNIT Number of fragments in the plan of a query.
NumFragmentInstances STABLE & HIGH UNIT Number of fragment instances executed by a query.
TotalScanBytesSent STABLE & LOW BYTES The total number of bytes sent (across the network) by fragment instances that had a scan node in their plan.
TotalInnerBytesSent STABLE & LOW BYTES The total number of bytes sent (across the network) by fragment instances that did not have a scan node in their plan i.e. that received their input data from other instances through exchange node.
InnerNodeSelectivityRatio STABLE & LOW DOUBLE_VALUE The ratio between bytes sent by instances with a scan node in their plan and instances without a scan node in their plan. This indicates how well the inner nodes of the execution plan reduced the data volume.
NumCompletedBackends STABLE & HIGH UNIT The number of completed backends. Only valid after all backends have started executing. Does not count the number of CANCELLED Backends.
BytesAssigned STABLE & HIGH BYTES Total number of bytes of filesystem scan ranges assigned to this fragment instance.
CompletionTime STABLE & HIGH TIME_NS Completion time of this fragment instance
ExecutionRate STABLE & LOW BYTES_PER_SECOND Rate at which the fragment instance processed its input scan ranges.
PerReadThreadRawHdfsThroughput STABLE & LOW BYTES_PER_SECOND The read throughput in bytes/sec for each HDFS read thread while it is executing I/O operations on behalf of a scan.
AverageHdfsReadThreadConcurrency STABLE & LOW DOUBLE_VALUE The average number of HDFS read threads executing read operations on behalf of this scan. Higher values (i.e. close to the aggregate number of I/O threads across all disks accessed) show that this scan is using a larger proportion of the I/O capacity of the system. Lower values show that either this scan is not I/O bound or that it is getting a small share of the I/O capacity of the system.
ParquetUncompressedBytesReadPerColumn STABLE & LOW BYTES Stats about the number of uncompressed bytes read per column. Each sample in the counter is the size of a single column that is scanned by the scan node.
ParquetCompressedBytesReadPerColumn STABLE & LOW BYTES Stats about the number of compressed bytes read per column. Each sample in the counter is the size of a single column that is scanned by the scan node.
NumScannerThreadMemUnavailable STABLE & LOW UNIT Number of times scanner threads were not created because of memory not available.
AverageScannerThreadConcurrency STABLE & LOW DOUBLE_VALUE Average number of executing scanner threads.
ColumnarScannerIdealReservation DEBUG BYTES Tracks stats about the ideal reservation for a scanning a row group (parquet) or stripe (orc). The ideal reservation is calculated based on min and max buffer size.
ColumnarScannerActualReservation DEBUG BYTES Tracks stats about the actual reservation for a scanning a row group (parquet) or stripe (orc).

Appendix

Significance Description
STABLE & HIGH High level and stable counters - always useful for measuring query performance and status. Counters that everyone is interested. should rarely change and if it does we will make some effort to notify users.
STABLE & LOW Low level and stable counters - interesting counters to monitor and analyze by machine. It will probably be interesting under some circumstances for users.
UNSTABLE Unstable but useful - useful to understand query performance, but subject to change, particularly if the implementation changes. E.g. RowBatchQueuePutWaitTime, MaterializeTupleTimer
DEBUG Debugging counters - generally not useful to users of Impala, the main use case is low-level debugging. Can be hidden to reduce noise for most consumers of profiles.