Kusto.Ingest 클라이언트 인터페이스 및 클래스
Kusto Ingest .NET 라이브러리의 기본 인터페이스 및 클래스는 다음과 같습니다.
- 인터페이스 IKustoIngestClient: 기본 수집 인터페이스
- Class ExtendedKustoIngestClient: 기본 수집 인터페이스에 대한 확장
- 클래스 KustoIngestFactory: 수집 클라이언트를 위한 기본 팩터리
- 클래스 KustoIngestionProperties: 일반적인 수집 속성을 제공하는 데 사용되는 클래스
- class SourceOptions: 원본 데이터 처리 옵션
- 클래스 IngestionMapping: 수집에 대한 데이터 매핑을 설명하는 데 사용되는 클래스입니다.
- Enum DataSourceFormat: 지원되는 데이터 원본 형식입니다. 예를 들어 CSV 또는 JSON입니다.
- 인터페이스 IKustoQueuedIngestClient: 큐에 대기 중인 수집에만 적용되는 작업을 설명하는 인터페이스
- 클래스 KustoQueuedIngestionProperties: 큐에 대기 중인 수집에만 적용되는 속성
참고
이 정보는 특히 .NET 클라이언트 라이브러리에 적용됩니다. 클라이언트 라이브러리마다 사용 가능한 인터페이스 및 클래스가 변형되어 있을 수 있습니다. 모든 클라이언트 라이브러리에서 Kusto 수집에 대한 자세한 내용은 Kusto 수집 개요를 참조하세요.
인터페이스 IKustoIngestClient
- IngestFromDataReaderAsync
- IngestFromStorageAsync
- IngestFromStreamAsync
public interface IKustoIngestClient : IDisposable
{
/// <summary>
/// Ingests data from <see cref="IDataReader"/>. <paramref name="dataReader"/> will be closed when the call completes.
/// </summary>
/// <param name="dataReader">The <see cref="IDataReader"/> data source to ingest. Only the first record set will be used</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <param name="sourceOptions">Options for the <see cref="IDataReader"/> ingestion source. This is an optional parameter</param>
/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>
Task<IKustoIngestionResult> IngestFromDataReaderAsync(IDataReader dataReader, KustoIngestionProperties ingestionProperties, DataReaderSourceOptions sourceOptions = null);
/// <summary>
/// Ingest data from one of the supported storage providers. Currently the supported providers are: File System, Azure Blob Storage.
/// </summary>
/// <param name="uri">The URI of the storage resource to be ingested. Note: This URI may include a storage account key or shared access signature (SAS).
/// See <see href="https://learn.microsoft.com/azure/kusto/api/connection-strings/storage"/> for the URI format options.</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <param name="sourceOptions">Options for the storage ingestion source. This is an optional parameter</param>
/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>
Task<IKustoIngestionResult> IngestFromStorageAsync(string uri, KustoIngestionProperties ingestionProperties, StorageSourceOptions sourceOptions = null);
/// <summary>
/// Ingest data from <see cref="Stream"/>.
/// </summary>
/// <param name="stream">The <see cref="Stream"/> data source to ingest</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <param name="sourceOptions">Options for the <see cref="Stream"/> ingestion source. This is an optional parameter</param>
/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>
Task<IKustoIngestionResult> IngestFromStreamAsync(Stream stream, KustoIngestionProperties ingestionProperties, StreamSourceOptions sourceOptions = null);
}
클래스 ExtendedKustoIngestClient
- IngestFromSingleBlob - 사용되지 않습니다. 대신
IKustoIngestClient.IngestFromStorageAsync
를 사용하세요. - IngestFromSingleBlobAsync - 사용되지 않습니다. 대신
IKustoIngestClient.IngestFromStorageAsync
를 사용하세요. - IngestFromDataReader - 사용되지 않습니다. 대신
IKustoIngestClient.IngestFromDataReaderAsync
를 사용하세요. - IngestFromDataReaderAsync
- IngestFromSingleFile - 사용되지 않습니다. 대신
IKustoIngestClient.IngestFromStorageAsync
를 사용하세요. - IngestFromSingleFileAsync - 사용되지 않습니다. 대신
IKustoIngestClient.IngestFromStorageAsync
를 사용하세요. - IngestFromStream - 사용되지 않습니다. 대신
IKustoIngestClient.IngestFromStreamAsync
를 사용하세요. - IngestFromStreamAsync
public static class ExtendedKustoIngestClient
{
/// <summary>
/// Ingest data from a single data blob
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="blobUri">The URI of the blob will be ingested</param>
/// <param name="deleteSourceOnSuccess">Indicates if the source blob should be deleted after a successful ingestion</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <param name="rawDataSize">The uncompressed raw data size</param>
/// <returns><see cref="IKustoIngestionResult"/></returns>
public static IKustoIngestionResult IngestFromSingleBlob(this IKustoIngestClient client, string blobUri, bool deleteSourceOnSuccess, KustoIngestionProperties ingestionProperties, long? rawDataSize = null);
/// <summary>
/// Ingest data from a single data blob asynchronously
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="blobUri">The URI of the blob will be ingested</param>
/// <param name="deleteSourceOnSuccess">Indicates if the source blob should be deleted after a successful ingestion</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <param name="rawDataSize">The uncompressed raw data size</param>
/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>
public static Task<IKustoIngestionResult> IngestFromSingleBlobAsync(this IKustoIngestClient client, string blobUri, bool deleteSourceOnSuccess, KustoIngestionProperties ingestionProperties, long? rawDataSize = null);
/// <summary>
/// Ingest data from a single data blob
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="blobDescription"><see cref="BlobDescription"/> representing the blobs that will be ingested</param>
/// <param name="deleteSourceOnSuccess">Indicates if the source blob should be deleted after a successful ingestion</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <param name="rawDataSize">The uncompressed raw data size</param>
/// <returns><see cref="IKustoIngestionResult"/></returns>
public static IKustoIngestionResult IngestFromSingleBlob(this IKustoIngestClient client, BlobDescription blobDescription, bool deleteSourceOnSuccess, KustoIngestionProperties ingestionProperties, long? rawDataSize = null);
/// <summary>
/// Ingest data from a single data blob asynchronously
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="blobDescription"><see cref="BlobDescription"/> representing the blobs that will be ingested</param>
/// <param name="deleteSourceOnSuccess">Indicates if the source blob should be deleted after a successful ingestion</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <param name="rawDataSize">The uncompressed raw data size</param>
/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>
public static Task<IKustoIngestionResult> IngestFromSingleBlobAsync(this IKustoIngestClient client, BlobDescription blobDescription, bool deleteSourceOnSuccess, KustoIngestionProperties ingestionProperties, long? rawDataSize = null);
/// <summary>
/// Ingest data from <see cref="IDataReader"/>, which is closed and disposed of upon call completion
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="dataReader">The data to ingest (only the first record set will be used)</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <returns><see cref="IKustoIngestionResult"/></returns>
public static IKustoIngestionResult IngestFromDataReader(this IKustoIngestClient client, IDataReader dataReader, KustoIngestionProperties ingestionProperties);
/// <summary>
/// Asynchronously ingest data from <see cref="IDataReader"/>, which is closed and disposed of upon call completion
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="dataReader">The data to ingest (only the first record set will be used)</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>
public static Task<IKustoIngestionResult> IngestFromDataReaderAsync(this IKustoIngestClient client, IDataReader dataReader, KustoIngestionProperties ingestionProperties);
/// <summary>
/// Ingest data from <see cref="IDataReader"/>, which is closed and disposed of upon call completion
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="dataReaderDescription"><see cref="DataReaderDescription"/>Represents the data to ingest (only the first record set will be used)</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <returns><see cref="IKustoIngestionResult"/></returns>
public static IKustoIngestionResult IngestFromDataReader(this IKustoIngestClient client, DataReaderDescription dataReaderDescription, KustoIngestionProperties ingestionProperties);
/// <summary>
/// Asynchronously ingest data from <see cref="IDataReader"/>, which is closed and disposed of upon call completion
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="dataReaderDescription"><see cref="DataReaderDescription"/>Represents the data to ingest (only the first record set will be used)</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>
public static Task<IKustoIngestionResult> IngestFromDataReaderAsync(this IKustoIngestClient client, DataReaderDescription dataReaderDescription, KustoIngestionProperties ingestionProperties);
/// <summary>
/// Ingest data from a single file
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="filePath">Absolute path of the source file to be ingested</param>
/// <param name="deleteSourceOnSuccess">Indicates if the source file should be deleted after a successful ingestion</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <returns><see cref="IKustoIngestionResult"/></returns>
public static IKustoIngestionResult IngestFromSingleFile(this IKustoIngestClient client, string filePath, bool deleteSourceOnSuccess, KustoIngestionProperties ingestionProperties);
/// <summary>
/// Ingest data from a single file asynchronously
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="filePath">Absolute path of the source file to be ingested</param>
/// <param name="deleteSourceOnSuccess">Indicates if the source file should be deleted after a successful ingestion</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>
public static Task<IKustoIngestionResult> IngestFromSingleFileAsync(this IKustoIngestClient client, string filePath, bool deleteSourceOnSuccess, KustoIngestionProperties ingestionProperties);
/// <summary>
/// Ingest data from a single file
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="fileDescription"><see cref="FileDescription"/> representing the file that will be ingested</param>
/// <param name="deleteSourceOnSuccess">Indicates if the source file should be deleted after a successful ingestion</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <returns><see cref="IKustoIngestionResult"/></returns>
public static IKustoIngestionResult IngestFromSingleFile(this IKustoIngestClient client, FileDescription fileDescription, bool deleteSourceOnSuccess, KustoIngestionProperties ingestionProperties);
/// <summary>
/// Ingest data from a single file asynchronously
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="fileDescription"><see cref="FileDescription"/> representing the file that will be ingested</param>
/// <param name="deleteSourceOnSuccess">Indicates if the source file should be deleted after a successful ingestion</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>
public static Task<IKustoIngestionResult> IngestFromSingleFileAsync(this IKustoIngestClient client, FileDescription fileDescription, bool deleteSourceOnSuccess, KustoIngestionProperties ingestionProperties);
/// <summary>
/// Ingest data from <see cref="Stream"/>
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="stream">The data to ingest</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <param name="leaveOpen">Optional. If set to 'false' (default value), <paramref name="stream"/> will be closed and disposed on call completion</param>
/// <returns><see cref="IKustoIngestionResult"/></returns>
public static IKustoIngestionResult IngestFromStream(this IKustoIngestClient client, Stream stream, KustoIngestionProperties ingestionProperties, bool leaveOpen = false);
/// <summary>
/// Ingest data from <see cref="Stream"/> asynchronously
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="stream">The data to ingest</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <param name="leaveOpen">Optional. If set to 'false' (default value), <paramref name="stream"/> will be closed and disposed on call completion</param>
/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>
public static Task<IKustoIngestionResult> IngestFromStreamAsync(this IKustoIngestClient client, Stream stream, KustoIngestionProperties ingestionProperties, bool leaveOpen = false);
/// <summary>
/// Ingest data from <see cref="Stream"/>
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="streamDescription"><see cref="StreamDescription"/>Represents the data to ingest</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <param name="leaveOpen">Optional. If set to 'false' (default value), streamDescription.Stream will be closed and disposed on call completion</param>
/// <returns><see cref="IKustoIngestionResult"/></returns>
public static IKustoIngestionResult IngestFromStream(this IKustoIngestClient client, StreamDescription streamDescription, KustoIngestionProperties ingestionProperties, bool leaveOpen = false);
/// <summary>
/// Ingest data from <see cref="Stream"/> asynchronously
/// </summary>
/// <param name="client">The ingest client that will execute the ingestions</param>
/// <param name="streamDescription"><see cref="StreamDescription"/>Represents the data to ingest</param>
/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>
/// <param name="leaveOpen">Optional. If set to 'false' (default value), streamDescription.Stream will be closed and disposed on call completion</param>
/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>
public static Task<IKustoIngestionResult> IngestFromStreamAsync(this IKustoIngestClient client, StreamDescription streamDescription, KustoIngestionProperties ingestionProperties, bool leaveOpen = false);
}
클래스 KustoIngestFactory
- CreateDirectIngestClient
- CreateQueuedIngestClient
- CreateManagedStreamingIngestClient
- CreateStreamingIngestClient
/// <summary>
/// Factory for creating Kusto ingestion objects.
/// </summary>
public static class KustoIngestFactory
{
/// <summary>
/// Creates an implementation of <see cref="IKustoIngestClient"/> that communicates
/// directly with the Kusto engine service.
/// </summary>
/// <param name="kcsb">Indicates the connection to the Kusto engine service.</param>
/// <returns>An implementation of <see cref="IKustoIngestClient"/> that communicates
/// directly with the Kusto engine service.</returns>
/// <remarks>In most cases, it is preferred that ingestion be done using the
/// queued implementation of <see cref="IKustoIngestClient"/>. See <see cref="CreateQueuedIngestClient(KustoConnectionStringBuilder)"/>.</remarks>
public static IKustoIngestClient CreateDirectIngestClient(KustoConnectionStringBuilder kcsb);
/// <summary>
/// Creates an implementation of <see cref="IKustoIngestClient"/> that communicates
/// directly with the Kusto engine service.
/// </summary>
/// <param name="connectionString">Indicates the connection to the Kusto engine service.</param>
/// <returns>An implementation of <see cref="IKustoIngestClient"/> that communicates
/// directly with the Kusto engine service.</returns>
/// <remarks>In most cases, it is preferred that ingestion be done using the
/// queued implementation of <see cref="IKustoIngestClient"/>. See <see cref="CreateQueuedIngestClient(string)"/>.</remarks>
public static IKustoIngestClient CreateDirectIngestClient(string connectionString);
/// <summary>
/// Creates an implementation of <see cref="IKustoQueuedIngestClient"/> that communicates
/// with the Kusto ingestion service using a reliable queue.
/// </summary>
/// <param name="kcsb">Indicates the connection to the Kusto ingestion service.
/// Note that the ingestion service generally has a "ingest-" prefix in the
/// DNS host name part.</param>
/// <returns>An implementation of <see cref="IKustoQueuedIngestClient"/> that communicates
/// with the Kusto ingestion service using a reliable queue.</returns>
public static IKustoQueuedIngestClient CreateQueuedIngestClient(KustoConnectionStringBuilder kcsb);
/// <summary>
/// Creates an implementation of <see cref="IKustoQueuedIngestClient"/> that communicates
/// with the Kusto ingestion service using a reliable queue.
/// </summary>
/// <param name="connectionString">Indicates the connection to the Kusto ingestion service.
/// Note that the ingestion service generally has a "ingest-" prefix in the
/// DNS host name part.</param>
/// <returns>An implementation of <see cref="IKustoQueuedIngestClient"/> that communicates with the Kusto ingestion service using a reliable queue.</returns>
public static IKustoQueuedIngestClient CreateQueuedIngestClient(string connectionString);
/// <summary>
/// Creates an implementation of <see cref="IKustoIngestClient"/> that performs managed streaming ingestion
/// </summary>
/// <param name="engineKcsb">Indicates the connection to the Kusto engine service.</param>
/// <param name="dmKcsb">Indicates the connection to the Kusto data management service.</param>
/// <returns>An implementation of <see cref="IKustoIngestClient"/> that performs managed streaming ingestion</returns>
/// <remarks>Streaming ingestion is performed directy intto Kusto enginge cluster
/// and is optimized for low-latency ingestion of relatively small chunks of data.
/// If the streaming ingset doesn't succeed after several retries, queued ingestion will be performed.</remarks>
public static IKustoIngestClient CreateManagedStreamingIngestClient(KustoConnectionStringBuilder engineKcsb, KustoConnectionStringBuilder dmKcsb);
/// <summary>
/// Creates an implementation of <see cref="IKustoIngestClient"/> that performs managed streaming ingestion
/// </summary>
/// <param name="engineConnectionString">Indicates the connection to the Kusto engine service.</param>
/// <param name="dmConnectionString">Indicates the connection to the Kusto data management service.</param>
/// <returns>An implementation of <see cref="IKustoIngestClient"/> that performs managed streaming ingestion</returns>
/// <remarks>Streaming ingestion is performed directy intto Kusto enginge cluster
/// and is optimized for low-latency ingestion of relatively small chunks of data.
/// If the streaming ingset doesn't succeed after several retries, queued ingestion will be performed.</remarks>
public static IKustoIngestClient CreateManagedStreamingIngestClient(string engineConnectionString, string dmConnectionString);
/// <summary>
/// Creates an implementation of <see cref="IKustoIngestClient"/> that performs streaming ingestion
/// </summary>
/// <param name="kcsb">Indicates the connection to the Kusto engine service.</param>
/// <returns>An implementation of <see cref="IKustoIngestClient"/> that performs streaming ingestion</returns>
/// <remarks>Streaming ingestion is performed directy intto Kusto enginge cluster
/// and is optimized for low-latency ingestion of relatively small chunks of data</remarks>
public static IKustoIngestClient CreateStreamingIngestClient(KustoConnectionStringBuilder kcsb);
/// <summary>
/// Creates an implementation of <see cref="IKustoIngestClient"/> that performs streaming ingestion
/// </summary>
/// <param name="connectionString">Indicates the connection to the Kusto engine service.</param>
/// <returns>An implementation of <see cref="IKustoIngestClient"/> that performs streaming ingestion</returns>
/// <remarks>Streaming ingestion is performed directy into Kusto enginge cluster
/// and is optimized for low-latency ingestion of relatively small chunks of data</remarks>
public static IKustoIngestClient CreateStreamingIngestClient(string connectionString);
}
클래스 KustoIngestionProperties
KustoIngestionProperties 클래스에는 수집 프로세스를 세부 제어하기 위한 기본 수집 속성이 포함되어 있습니다.
속성 | 의미 |
---|---|
DatabaseName | 수집할 데이터베이스의 이름 |
TableName | 수집할 테이블의 이름 |
DropByTags | 각 익스텐트에서 가질 태그입니다. DropByTags는 영구적이며 다음과 같이 사용할 수 있습니다. .show table T extents where tags has 'some tag' 또는 .drop extents <| .show table T extents where tags has 'some tag' |
IngestByTags | 익스텐트별로 작성된 태그입니다. 나중에 동일한 데이터를 두 번 수집하지 않도록 속성과 함께 IngestIfNotExists 사용할 수 있습니다. 자세한 내용은 ingest-by: 태그를 참조하세요. |
IngestionMapping | 종료 매핑 또는 열 매핑 목록에 대한 참조를 보유합니다. |
AdditionalTags | 필요에 따라 추가 태그 |
IngestIfNotExists | 다시 수집하지 않으려는 태그 목록(테이블당) |
ValidationPolicy | 데이터 유효성 검사 정의. |
서식 | 수집되는 데이터의 형식 |
AdditionalProperties | 수집 속성으로 수집 명령으로 전달되는 기타 속성 입니다. 모든 수집 속성이 이 클래스의 별도 멤버에 표시되지 않으므로 속성이 전달됩니다. |
public class KustoIngestionProperties
{
public string DatabaseName { get; set; }
public string TableName { get; set; }
public IEnumerable<string> DropByTags { get; set; }
public IEnumerable<string> IngestByTags { get; set; }
public IEnumerable<string> AdditionalTags { get; set; }
public IEnumerable<string> IngestIfNotExists { get; set; }
public IngestionMapping IngestionMapping { get; set; }
public ValidationPolicy ValidationPolicy { get; set; }
public DataSourceFormat? Format { get; set; }
public bool IgnoreSizeLimit { get; set; } // Determines whether the limit of 4GB per single ingestion source should be ignored. Defaults to false.
public IDictionary<string, string> AdditionalProperties { get; set; }
public KustoIngestionProperties(string databaseName, string tableName);
}
클래스 SourceOptions
SourceOptions 및 파생 클래스는 원본 데이터에 대한 추가 정보 및 처리 옵션을 캡슐화합니다. 지원되는 원본 간에 세부 사항이 다릅니다.
// Base class
public abstract class SourceOptions
{
public Guid SourceId { get; set; }; // Identifies the ingestion source
public bool Compress { get; set; }; // Determines whether data should be compressed before being uploaded.
}
// Represents a local file/blob/ADLSv2 file
public sealed class StorageSourceOptions : SourceOptions
{
public long? Size { get; set; }; // Uncompressed data size. Should be used to comunicate the file size to the service for efficient ingestion
public bool DeleteSourceOnSuccess { get; set; }; // Indicates whether the ingestion source should be deleted after successful ingestion. Defaults to 'false'. When set to 'true', will require the service to individually delete each blob, which could put extra pressure on the service.
public DataSourceCompressionType CompressionType { get; set; }; // Indicates compression used. Defaults to 'none'
}
클래스 IngestionMapping
기존 매핑 또는 열 매핑 목록에 대한 참조를 보유합니다.
속성 | 의미 |
---|---|
IngestionMappings | 대상 열 데이터 및 해당 원본을 설명하는 각 열 매핑 |
IngestionMappingKind | IngestionMappings 속성에 설명된 매핑 종류 - Csv, Json, Avro, Parquet, SStream, Orc, ApacheAvro 또는 W3CLogFile 중 하나 |
IngestionMappingReference | 미리 생성된 매핑 이름 |
public class IngestionMapping
{
public IEnumerable<ColumnMapping> IngestionMappings { get; set; }
public IngestionMappingKind IngestionMappingKind { get; set; }
public string IngestionMappingReference { get; set; }
public IngestionMapping()
public IngestionMapping(IngestionMapping ingestionMapping)
}
Enum DataSourceFormat
public enum DataSourceFormat
{
csv, // Data is in a CSV(-comma-separated values) format
tsv, // Data is in a TSV(-tab-separated values) format
scsv, // Data is in a SCSV(-semicolon-separated values) format
sohsv, // Data is in a SOHSV(-SOH (ASCII 1) separated values) format
psv, // Data is in a PSV (pipe-separated values) format
tsve, // Tab-separated value with '\' escaping character.
txt, // Each record is a line and has just one field
raw, // The entire stream/file/blob is a single record having a single field
json, // Data is in a JSON-line format (each line is record with a single JSON value)
multijson, // The data stream is a concatenation of JSON documents (property bags all)
avro, // Data is in a AVRO format
orc, // Data is in a ORC format
parquet, // Data is in a Parquet format
}
KustoIngestionProperties 정의의 예
var guid = new Guid().ToString();
var kustoIngestionProperties = new KustoIngestionProperties("TargetDatabase", "TargetTable")
{
DropByTags = new List<string> { DateTime.Today.ToString() },
IngestByTags = new List<string> { guid },
AdditionalTags = new List<string> { "some tags" },
IngestIfNotExists = new List<string> { guid },
IngestionMapping = new IngestionMapping() {
IngestionMappingKind = Data.Ingestion.IngestionMappingKind.Csv,
IngestionMappings = new ColumnMapping[] { new ColumnMapping() {
ColumnName = "stringColumn",
Properties = new Dictionary<string, string>() {
{ MappingConsts.Ordinal, "1"} }
} },
},
ValidationPolicy = new ValidationPolicy { ValidationImplications = ValidationImplications.Fail, ValidationOptions = ValidationOptions.ValidateCsvInputConstantColumns },
Format = DataSourceFormat.csv
};
인터페이스 IKustoQueuedIngestClient
IKustoQueuedIngestClient 인터페이스는 수집 작업 결과를 따르는 추적 메서드를 추가하고 수집 클라이언트에 대한 RetryPolicy를 노출합니다.
- PeekTopIngestionFailures
- GetAndDiscardTopIngestionFailures
- GetAndDiscardTopIngestionSuccesses
public interface IKustoQueuedIngestClient : IKustoIngestClient
{
/// <summary>
/// Peeks top (== oldest) ingestion failures
/// </summary>
/// <param name="messagesLimit">Maximum ingestion failures to peek. Default value peeks 32 messages.</param>
/// <returns>A task which its result contains IEnumerable of <see cref="IngestionFailure"/>. The received messages won't be discarded from the relevant azure queue.</returns>
Task<IEnumerable<IngestionFailure>> PeekTopIngestionFailures(int messagesLimit = -1);
/// <summary>
/// Returns and deletes top (== oldest) ingestion failure notifications
/// </summary>
/// <param name="messagesLimit">Maximum ingestion failure notifications to get. Default value peeks 32 messages.</param>
/// <returns>A task which its result contains IEnumerable of <see cref="IngestionFailure"/>. The received messages will be discarded from the relevant azure queue.</returns>
Task<IEnumerable<IngestionFailure>> GetAndDiscardTopIngestionFailures(int messagesLimit = -1);
/// <summary>
/// Returns and deletes top (== oldest) ingestion success notifications
/// </summary>
/// <param name="messagesLimit">Maximum ingestion success notifications to get. Default value peeks 32 messages.</param>
/// <returns>A task which its result contains IEnumerable of <see cref="IngestionSuccess"/>. The received messages will be discarded from the relevant azure queue.</returns>
Task<IEnumerable<IngestionSuccess>> GetAndDiscardTopIngestionSuccesses(int messagesLimit = -1);
/// <summary>
/// An implementation of IRetryPolicy that will be enforced on every ingest call,
/// which affects how the ingest client handles retrying on transient failures
/// </summary>
IRetryPolicy QueueRetryPolicy { get; set; }
}
클래스 KustoQueuedIngestionProperties
KustoQueuedIngestionProperties 클래스는 수집 동작을 미세 조정하는 데 사용할 수 있는 여러 컨트롤 노브를 사용하여 KustoIngestionProperties를 확장합니다.
속성 | 의미 |
---|---|
FlushImmediately | 기본값은 false 입니다. 로 true 설정하면 데이터 관리 서비스의 집계 메커니즘을 무시합니다. |
IngestionReportLevel | 수집 상태 보고 수준을 제어합니다(기본값은 FailuresOnly ). 성능 및 스토리지 사용량을 위해 IngestionReportLevel을 로 설정하지 않는 것이 좋습니다. FailuresAndSuccesses |
IngestionReportMethod | 수집 상태 보고의 대상을 제어합니다. 사용 가능한 옵션은 Azure Queue, Azure Table 또는 둘 다입니다. 기본값은 Queue 입니다. |
public class KustoQueuedIngestionProperties : KustoIngestionProperties
{
/// <summary>
/// Allows to stop the batching phase and will cause to an immediate ingestion.
/// Defaults to 'false'.
/// </summary>
public bool FlushImmediately { get; set; }
/// <summary>
/// Controls the ingestion status report level.
/// Defaults to 'FailuresOnly'.
/// </summary>
public IngestionReportLevel ReportLevel { get; set; }
/// <summary>
/// Controls the target of the ingestion status reporting. Available options are Azure Queue, Azure Table, or both.
/// Defaults to 'Queue'.
/// </summary>
public IngestionReportMethod ReportMethod { get; set; }
/// <summary>
/// Controls the target of the ingestion status reporting. Available options are Azure Queue, Azure Table, or both.
/// Defaults to 'Queue'.
/// </summary>
public IngestionReportMethod ReportMethod;
public KustoQueuedIngestionProperties(string databaseName, string tableName);
}
피드백
https://aka.ms/ContentUserFeedback
출시 예정: 2024년 내내 콘텐츠에 대한 피드백 메커니즘으로 GitHub 문제를 단계적으로 폐지하고 이를 새로운 피드백 시스템으로 바꿀 예정입니다. 자세한 내용은 다음을 참조하세요.다음에 대한 사용자 의견 제출 및 보기