Azure Functions with FastAPI and OpenAI Streaming

Question

How to send out the Azure Open AI response in real-time streaming through FastAPI and hosted on Azure Functions?

Background:

1. The goal is to deploy a OpenAI streaming response API with Azure Functions + FastAPI.
2. OpenAI streaming works perfect with FastAPI alone.
3. However since FastAPI is hosted by Function App, the response is blocked until streaming is done .

# openai router
from fastapi.responses import StreamingResponse
from fastapi import APIRouter
import openai
import os

OPENAI_API_TYPE = os.environ.get("OPENAI_API_TYPE", "azure")
OPENAI_API_VERSION = os.environ.get("OPENAI_API_VERSION", "2023-05-15")
OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY")
OPENAI_RESOURCE_ENDPOINT = os.environ.get("OPENAI_RESOURCE_ENDPOINT")

openai.api_type = OPENAI_API_TYPE
openai.api_key = OPENAI_API_KEY
openai.api_base = OPENAI_RESOURCE_ENDPOINT
openai.api_version = OPENAI_API_VERSION

router = APIRouter(prefix="/api/openai", tags=["openai"])

@router.post("/gpt")
async def ask_gpt(text: str):

  def get_chat_completion_stream(
      messages: list[dict],
      engine="gpt-35-turbo",
      temperature: Union[int, float] = 0,
  ) -> str:
    res = openai.ChatCompletion.create(
        messages=messages, temperature=temperature, engine=engine, stream=True
    )

    start_time = time.time()
    for event in res:
        if "content" in event["choices"][0].delta:
            current_response = event["choices"][0].delta.content
            res_time = time.time() - start_time

            print(f"response :{current_response}, time: {res_time}")
            yield "data: " + current_response + "\n\n"

 chat_prompts = [
                {"role": "user", "content": text},
            ]
 return StreamingResponse(
                    openai_utils.get_chat_completion_stream(chat_prompts),
                    media_type="text/event-stream",
                )

# __init__.py

import azure.functions as func

from main import app

def main(req: func.HttpRequest, context: func.Context) -> func.HttpResponse:
    return func.AsgiMiddleware(app).handle(req, context)

{
  "scriptFile": "__init__.py",
  "bindings": [
    {
      "authLevel": "Anonymous",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": ["get", "post", "patch", "delete", "put"],
      "route": "/api/{*route}"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "$return"
    }
  ]
}

Answer 1

Hello @Liu, Kevin 劉峻瑋 (598648) - This is likely because the Python worker is running behind the host process. I'll get a confirmation and update you here.

UPDATE 11/14/23:

The product team will be releasing a new version of the Python worker that'll support streaming req/res. Please track the following discussion: https://github.com/Azure/azure-functions-python-worker/discussions/1349#discussioncomment-7569591

In the meantime, it's reported that the SubApplication pattern works (as discussed in this thread) but I believe the timeouts will still be a potential issue.

Related thread: https://learn.microsoft.com/en-us/answers/questions/1402873/azure-function-streaming-response

---

Please "Accept Answer" if the answer is helpful so that others in the community may benefit from your experience.

Answer 2

{text

Answer 3

فعال سازی هوش مصنوعی فارسی زبان