Ingestion Jobs - Create

Reference

Service:: Azure AI Services

API Version:: 2023-10-01-preview

Starts a job for ingesting data into an Azure Search index to be used as a data source for completions. The status of the ingestion job will be kept for 48 hours after its completion and will be deleted afterwards.

PUT {endpoint}/openai/extensions/on-your-data/ingestion-jobs/{jobId}?api-version=2023-10-01-preview

URI Parameters

Name	In	Required	Type	Description
endpoint	path	True	string url	Supported Cognitive Services endpoints (protocol and hostname, for example: https://aoairesource.openai.azure.com. Replace "aoairesource" with your Azure OpenAI account name).
jobId	path	True	string	The id of the ingestion job.
api-version	query	True	string	The requested API version.

Request Header

Name	Required	Type	Description
api-key	True	string	Provide your Cognitive Services Azure OpenAI account key here.
searchServiceEndpoint	True	string	The endpoint to the search service resource where the ingestion assets will be created.
searchServiceAdminKey		string	The admin key to the search service resource if Managed Identity is not setup.
storageConnectionString	True	string	The connection string to the blob storage account hosting the data that will be ingested.
storageContainer	True	string	The name of the container hosting the data that will be ingested.
embeddingEndpoint		string	The endpoint of the embeddings model to be used for embeddings calculations if enabled.
embeddingKey		string	The key to use for accessing the embeddings model if enabled and if Managed Identity is not setup.
storageEndpoint		string	The blob storage endpoint hosting the data that will be ingested. Required for ResourceId format of storageConnectionString.
embeddingDeploymentName		string	The name of embedding deployment of the current Azure OpenAI resource to use for accessing the embeddings model.

Request Body

Name	Required	Type	Description
dataRefreshIntervalInMinutes	True	integer
chunkSize		integer	This number defines the maximum number of tokens in each chunk produced by the ingestion flow.
completionAction		IngestionJobCompletionAction	IngestionJobCompletionAction The action to take on ingestion job completion.
id		string
progress		IngestionJobProgress	IngestionJobProgress
warnings		string[]

Responses

Name	Type	Description
200 OK	IngestionJob	Success
Other Status Codes	ErrorResponse	An error occurred.

Security

api-key

Provide your Cognitive Services Azure OpenAI account key here.

Type: apiKey
In: header

Examples

Starting an ingestion job.

Sample Request

HTTP

PUT https://aoairesource.openai.azure.com/openai/extensions/on-your-data/ingestion-jobs/test?api-version=2023-10-01-preview






{
  "dataRefreshIntervalInMinutes": 60,
  "completionAction": "cleanUpAssets"
}

Sample Response

Status code:: 200

{
  "id": "test",
  "dataRefreshIntervalInMinutes": 60,
  "completionAction": "cleanUpAssets",
  "status": "notRunning",
  "warnings": [],
  "progress": {
    "stageProgress": [
      {
        "name": "Preprocessing",
        "totalItems": 0,
        "processedItems": 0
      },
      {
        "name": "Indexing",
        "totalItems": 0,
        "processedItems": 0
      }
    ]
  }
}

Definitions

Name	Description
Error	Error
ErrorCode	ErrorCode
ErrorResponse	ErrorResponse
IngestionJob	IngestionJob
IngestionJobCompletionAction	IngestionJobCompletionAction
IngestionJobProgress	IngestionJobProgress
InnerError	InnerError
InnerErrorCode	InnerErrorCode
OperationState	OperationState
StageProgress

Error

Name	Type	Description
code	ErrorCode	ErrorCode Error codes as defined in the Microsoft REST guidelines (https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses).
details	Error[]	The error details if available.
innererror	InnerError	InnerError Inner error as defined in the Microsoft REST guidelines (https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses).
message	string	The message of this error.
target	string	The location where the error happened if available.

ErrorCode

Name	Type	Description
conflict	string	The requested operation conflicts with the current resource state.
contentFilter	string	Image generation failed as a result of our safety system.
fileImportFailed	string	Import of file failed.
forbidden	string	The operation is forbidden for the current user/api key.
internalFailure	string	Internal error. Please retry.
invalidPayload	string	The request data is invalid for this operation.
itemDoesAlreadyExist	string	The item does already exist.
jsonlValidationFailed	string	Validation of jsonl data failed.
notFound	string	The resource is not found.
quotaExceeded	string	Quota exceeded.
serviceUnavailable	string	The service is currently not available.
tooManyRequests	string	Too many requests. Please retry later.
unauthorized	string	The current user/api key is not authorized for the operation.
unexpectedEntityState	string	The operation cannot be executed in the current resource's state.

ErrorResponse

Name	Type	Description
error	Error	Error Error content as defined in the Microsoft REST guidelines (https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses).

IngestionJob

Name	Type	Default Value	Description
chunkSize	integer	1024	This number defines the maximum number of tokens in each chunk produced by the ingestion flow.
completionAction	IngestionJobCompletionAction		IngestionJobCompletionAction The action to take on ingestion job completion.
dataRefreshIntervalInMinutes	integer
error	Error		Error Error content as defined in the Microsoft REST guidelines (https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses).
id	string
progress	IngestionJobProgress		IngestionJobProgress
status	OperationState		OperationState The state of a job or item.
warnings	string[]

IngestionJobCompletionAction

Name	Type	Description
cleanUpAssets	string	Will clean up intermediate assets created during the ingestion process.
keepAllAssets	string	Will not clean up any of the intermediate assets created during the ingestion process.

IngestionJobProgress

Name	Type	Description
stageProgress	StageProgress[]

InnerError

Name	Type	Description
code	InnerErrorCode	InnerErrorCode Inner error codes as defined in the Microsoft REST guidelines (https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses).
innererror	InnerError	InnerError Inner error as defined in the Microsoft REST guidelines (https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses).

InnerErrorCode

Name	Type	Description
invalidPayload	string	The request data is invalid for this operation.

OperationState

Name	Type	Description
failed	string	The operation has completed processing with a failure and cannot be further consumed.
notRunning	string	The operation is not yet queued to be processed in the future.
running	string	The operation has started to be processed.
succeeded	string	The operation has successfully be processed and is ready for consumption.

StageProgress

Name	Type	Description
name	string
processedItems	integer
totalItems	integer