read_state_metadata table-valued function

  • Article

Applies to: check marked yes Databricks Runtime 14.3 and above

Returns a table with rows that represent the metadata of a streaming query state.

A table valued function for reading metadata of streaming query stateful operators. It only supports running as a batch query.

Syntax

read_state_metadata(path)

Arguments

This function requires named parameter invocation for the option keys.

  • path: A STRING literal with the path of the streaming query checkpoint location.

Returns

A table of state metadata has the following schema. All columns are NOT NULL.

  • operatorId INT NOT NULL

    An integer id of the stateful streaming operator.

  • operatorName STRING NOT NULL

    Name of the stateful streaming operator.

  • stateStoreName STRING NOT NULL

    Name of the state store of the operator.

  • numPartitions INT NOT NULL

    Number of partitions of the state store.

  • minBatchId BIGINT NOT NULL

    The minimum batch ID available for querying state. The value could be invalid if the streaming query taking the checkpoint is running, as it would cause cleanup to run.

  • maxBatchId BIGINT NOT NULL

    The maximum batch ID available for querying state. The value could be invalid if the streaming query taking the checkpoint is running, as the query will commit further batches.

Examples

- Read state metadata from the checkpoint
> SELECT * FROM read_state_metadata('/checkpoint/path');