叢集因程式庫衝突而取消執行 Python 命令 Cluster cancels Python command execution due to library conflict

問題Problem

叢集會 Cancelled 在 Python 筆記本中返回。The cluster returns Cancelled in a Python notebook. 所有其他語言中的筆記本都成功地在相同的叢集上執行。Notebooks in all other languages execute successfully on the same cluster.

原因Cause

當您將程式庫(例如、、、或)的衝突版本安裝 ipython ipywidgets 到時 numpy scipy pandas PYTHONPATH ,Python 複寫可能會中斷,使所有命令 Cancelled 在30秒後傳回。When you install a conflicting version of a library, such as ipython, ipywidgets, numpy, scipy, or pandas to the PYTHONPATH, then the Python REPL can break, causing all commands to return Cancelled after 30 seconds. 這也會中斷% sh,這是可讓您在 Python 筆記本儲存格中輸入 shell 腳本的筆記本宏。This also breaks %sh, the notebook macro that lets you enter shell scripts in Python notebook cells.

注意

解決方法Solution

若要解決此問題,請執行下列步驟:To solve this problem, do the following:

  1. 找出衝突的程式庫,並將它卸載。Identify the conflicting library and uninstall it.
  2. 在筆記本中安裝正確的程式庫版本,或使用叢集範圍的初始化腳本。Install the correct version of the library in a notebook or with a cluster-scoped init script.

識別衝突的程式庫Identify the conflicting library

  1. 一次卸載一個程式庫,並檢查 Python 複寫是否仍中斷。Uninstall each library one at a time, and check if the Python REPL still breaks.
  2. 如果複寫仍然中斷,請重新安裝您移除的媒體櫃,並移除下一個程式庫。If the REPL still breaks, reinstall the library you removed and remove the next one.
  3. 當您找到導致複寫中斷的程式庫時,請使用下列兩種方法的其中一種來安裝該程式庫的正確版本。When you find the library that causes the REPL to break, install the correct version of that library using one of the two methods below.

您也可以檢查叢集的驅動程式記錄檔( std.err )(位於 [叢集設定] 頁面上)來取得堆疊追蹤,以及可協助識別程式庫衝突的錯誤訊息。You can also inspect the driver log (std.err) for the cluster (on the Cluster Configuration page) for a stack trace and error message that can help identify the library conflict.

安裝正確的程式庫Install the correct library

執行下列其中一項動作。Do one of the following.

選項1:使用 pip3 在筆記本中安裝Option 1: Install in a notebook using pip3

%sh sudo apt-get -y install python3-pip
  pip3 install <library-name>

選項2:使用叢集範圍的初始化腳本進行安裝Option 2: Install using a cluster-scoped init script

請遵循下列步驟來建立叢集範圍的初始化腳本,以安裝正確的程式庫版本。Follow the steps below to create a cluster-scoped init script that installs the correct version of the library. <library-name>將範例中的取代為要安裝的程式庫檔案名。Replace <library-name> in the examples with the filename of the library to install.

  1. 如果 init 腳本還不存在,請建立基底目錄來儲存它:If the init script does not already exist, create a base directory to store it:

    dbutils.fs.mkdirs("dbfs:/databricks/<directory>/")
    
  2. 建立下列腳本:Create the following script:

    dbutils.fs.put("/databricks/init/cluster-name/<library-name>.sh","""
     #!/bin/bash
     sudo apt-get -y install python3-pip
     sudo pip3 install <library-name>
     """, True)
    
  3. 確認腳本存在:Confirm that the script exists:

    display(dbutils.fs.ls("dbfs:/databricks/<directory>/<library-name>.sh"))
    
  4. 移至 [叢集設定[] 頁面,然後按一下 Advanced Options ] 切換。Go to the cluster configuration page and click the Advanced Options toggle.

  5. 在頁面底部,按一下 [初始化腳本] 索引標籤:At the bottom of the page, click the Init Scripts tab:

    無替代文字no-alternative-text

  6. 在 [目的地] 下拉式選單中,選取 [ DBFS],提供腳本的檔案路徑,然後按一下 [新增]。In the Destination drop-down, select DBFS, provide the file path to the script, and click Add.

  7. 重新開機叢集。Restart the cluster.