適用於 ML 的 Databricks Runtime 7.5 （不支援）

Databricks 於 2020 年 12 月發行此映射。

Databricks Runtime 7.5 for 機器學習 根據 Databricks Runtime 7.5 （不支援）提供適用於機器學習和數據科學的現成環境。 Databricks Runtime ML 包含許多熱門的機器學習連結庫，包括 TensorFlow、PyTorch 和 XGBoost。 它也支援使用 Horovod 的分散式深度學習訓練。

如需詳細資訊，包括建立 Databricks Runtime ML 叢集的指示，請參閱 Databricks 上的 AI 和 機器學習。

新功能和重大變更

Databricks Runtime 7.5 ML 建置在 Databricks Runtime 7.5 之上。 如需 Databricks Runtime 7.5 新功能的相關信息，包括 Apache Spark MLlib 和 SparkR，請參閱 Databricks Runtime 7.5 （不支援） 版本資訊。

棄用項目

• 套件 gorilla 已被取代，將在即將推出的 Databricks Runtime ML 主要版本中移除。

Databricks Runtime ML Python 環境的重大變更

如需 Databricks Runtime Python 環境的重大變更，請參閱 Databricks Runtime 7.5 （不支援）。 如需已安裝 Python 套件及其版本的完整清單，請參閱 Python 連結庫。

已升級 Python 套件

• databricks-cli 0.13.0 -> 0.14.0
• hyperopt 0.2.4.db2 -> 0.2.5.db1
• koalas 1.2.0 -> 1.4.0
• mlflow 1.11.0 -> 1.12.1
• petastorm 0.9.6 -> 0.9.7
• plotly 4.10.0 -> 4.12.0
• pytorch 1.6.0 -> 1.7.0
• torchvision 0.7.0 -> 0.8.1
• xgboost 1.2.0 -> 1.2.1

系統環境

Databricks Runtime 7.5 ML 中的系統環境與 Databricks Runtime 7.5 不同，如下所示：

• DBUtils:D atabricks Runtime ML 不包含 連結庫公用程式 （dbutils.library） （legacy）。 您可以改用 %pip 和 %conda 命令。 請參閱 筆記本範圍的 Python 連結庫。
• 針對 GPU 叢集，Databricks Runtime ML 包含下列 NVIDIA GPU 連結庫：
  • CUDA 10.1 Update 2
  • cuDNN 7.6.5
  • NCCL 2.7.3
  • TensorRT 6.0.1

程式庫

下列各節列出 Databricks Runtime 7.5 ML 中包含的連結庫，與 Databricks Runtime 7.5 中包含的連結庫不同。

本節內容：

• 頂層連結庫
• Python 連結庫
• R 連結庫
• Java 和 Scala 連結庫 （Scala 2.12 叢集）

頂層連結庫

Databricks Runtime 7.5 ML 包含下列最上層 連結庫：

• GraphFrames
• Horovod 和 HorovodRunner
• MLflow
• PyTorch
• spark-tensorflow-connector
• TensorFlow
• TensorBoard

Python 程式庫

Databricks Runtime 7.5 ML 使用 Conda 進行 Python 套件管理，並包含許多熱門 ML 套件。

除了下列各節中 Conda 環境中指定的套件之外，Databricks Runtime 7.5 ML 也會安裝下列套件：

• hyperopt 0.2.5.db1
• sparkdl 2.1.0-db2

CPU 叢集上的 Python 連結庫

name: databricks-ml
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.9.0=py37_0
  - asn1crypto=1.3.0=py37_1
  - astor=0.8.0=py37_0
  - backcall=0.1.0=py37_0
  - backports=1.0=pyhd3eb1b0_2
  - bcrypt=3.2.0=py37h7b6447c_0
  - blas=1.0=mkl
  - blinker=1.4=py37_0
  - boto3=1.12.0=py_0
  - botocore=1.15.0=py_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2020.10.14=h06a4308_1 # (updated from 0 in May 26, 2021 maintenance update)
  - cachetools=4.1.1=py_0
  - certifi=2020.11.8=py37h06a4308_0
  - cffi=1.14.0=py37he30daa8_1 # (updated from py37h2e261b9_0 in May 26, 2021 maintenance update)
  - chardet=3.0.4=py37h06a4308_1003
  - click=7.0=py37_0
  - cloudpickle=1.4.1=py_0
  - configparser=3.7.4=py37_0
  - cpuonly=1.0=0
  - cryptography=2.8=py37h1ba5d50_0
  - cycler=0.10.0=py37_0
  - cython=0.29.15=py37he6710b0_0
  - decorator=4.4.1=py_0
  - dill=0.3.1.1=py37_1
  - docutils=0.15.2=py37_0
  - entrypoints=0.3=py37_0
  - flask=1.1.1=py_1
  - freetype=2.9.1=h8a8886c_1
  - future=0.18.2=py37_1
  - gast=0.3.3=py_0
  - gitdb=4.0.5=py_0
  - gitpython=3.1.0=py_0
  - google-auth=1.11.2=py_0
  - google-auth-oauthlib=0.4.1=py_2
  - google-pasta=0.2.0=py_0
  - grpcio=1.27.2=py37hf8bcb03_0
  - gunicorn=20.0.4=py37_0
  - h5py=2.10.0=py37h7918eee_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=he6710b0_3
  - idna=2.8=py37_0
  - intel-openmp=2020.0=166
  - ipykernel=5.1.4=py37h39e3cac_0
  - ipython=7.12.0=py37h5ca1d4c_0
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=py37_0
  - jedi=0.17.2=py37_0
  - jinja2=2.11.1=py_0
  - jmespath=0.10.0=py_0
  - joblib=0.14.1=py_0
  - jpeg=9b=h024ee3a_2
  - jupyter_client=5.3.4=py37_0
  - jupyter_core=4.6.1=py37_0
  - kiwisolver=1.1.0=py37he6710b0_0
  - krb5=1.17.1=h173b8e3_0 # (updated from 1.16.4 in May 26, 2021 maintenance update)
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20181209=hc058e9b_0
  - libffi=3.3=he6710b0_2 # (updated from 3.2.1 in May 26, 2021 maintenance update)
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libpng=1.6.37=hbc83047_0
  - libpq=12.2=h20c2e04_0 # (updated from 11.2 in May 26, 2021 maintenance update)
  - libprotobuf=3.11.4=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.1.0=h2733197_0
  - libuv=1.40.0=h7b6447c_0
  - lightgbm=2.3.0=py37he6710b0_0
  - lz4-c=1.8.1.2=h14c3975_0
  - mako=1.1.2=py_0
  - markdown=3.1.1=py37_0
  - markupsafe=1.1.1=py37h14c3975_1
  - matplotlib-base=3.1.3=py37hef1b27d_0
  - mkl=2020.0=166
  - mkl-service=2.3.0=py37he904b0f_0
  - mkl_fft=1.0.15=py37ha843d7b_0
  - mkl_random=1.1.0=py37hd6b4f25_0
  - ncurses=6.2=he6710b0_1
  - networkx=2.4=py_1
  - ninja=1.10.2=py37hff7bd54_0
  - nltk=3.4.5=py37_0
  - numpy=1.18.1=py37h4f9e942_0
  - numpy-base=1.18.1=py37hde5b4d6_1
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py37_0
  - openssl=1.1.1k=h27cfd23_0 # (updated from 1.1.1h in May 26, 2021 maintenance update)
  - packaging=20.1=py_0
  - pandas=1.0.1=py37h0573a6f_0
  - paramiko=2.7.1=py_0
  - parso=0.7.0=py_0
  - patsy=0.5.1=py37_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=py37_1001
  - pillow=7.0.0=py37hb39fc2d_0
  - pip=20.0.2=py37_3
  - plotly=4.12.0=pyhd3eb1b0_0
  - prompt_toolkit=3.0.3=py_0
  - protobuf=3.11.4=py37he6710b0_0
  - psutil=5.6.7=py37h7b6447c_0
  - psycopg2=2.8.6=py37h3c74f83_1 # (updated from 2.8.4 in May 26, 2021 maintenance update)
  - ptyprocess=0.6.0=pyhd3eb1b0_2
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.19=py37_0
  - pygments=2.5.2=py_0
  - pyjwt=1.7.1=py37_0
  - pynacl=1.3.0=py37h7b6447c_0
  - pyodbc=4.0.30=py37he6710b0_0
  - pyopenssl=19.1.0=pyhd3eb1b0_1
  - pyparsing=2.4.6=py_0
  - pysocks=1.7.1=py37_1
  - python=3.7.10=hdb3f193_0 # (updated from 3.7.6 in May 26, 2021 maintenance update)
  - python-dateutil=2.8.1=py_0
  - python-editor=1.0.4=py_0
  - pytorch=1.7.0=py3.7_cpu_0
  - pytz=2019.3=py_0
  - pyzmq=18.1.1=py37he6710b0_0
  - readline=8.1=h27cfd23_0 # (updated from 7.0 in May 26, 2021 maintenance update)
  - requests=2.22.0=py37_1
  - requests-oauthlib=1.3.0=py_0
  - retrying=1.3.3=py37_2
  - rsa=4.0=py_0
  - s3transfer=0.3.3=py37_1
  - scikit-learn=0.22.1=py37hd81dba3_0
  - scipy=1.4.1=py37h0b6359f_0
  - setuptools=45.2.0=py37_0
  - simplejson=3.17.0=py37h7b6447c_0
  - six=1.14.0=py37h06a4308_0
  - smmap=3.0.4=py_0
  - sqlite=3.35.4=hdfb4753_0 # (updated from 3.31.1 in May 26, 2021 maintenance update)
  - sqlparse=0.4.1=py_0
  - statsmodels=0.11.0=py37h7b6447c_0
  - tabulate=0.8.3=py37_0
  - tk=8.6.10=hbc83047_0 # (updated from 8.6.8 in May 26, 2021 maintenance update)
  - torchvision=0.8.1=py37_cpu
  - tornado=6.0.3=py37h7b6447c_3
  - tqdm=4.42.1=py_0
  - traitlets=4.3.3=py37_0
  - typing_extensions=3.7.4.3=py_0
  - unixodbc=2.3.7=h14c3975_0
  - urllib3=1.25.8=py37_0
  - wcwidth=0.1.8=py_0
  - websocket-client=0.56.0=py37_0
  - werkzeug=1.0.0=py_0
  - wheel=0.34.2=py37_0
  - wrapt=1.11.2=py37h7b6447c_0
  - xz=5.2.5=h7b6447c_0 # (updated from 5.2.4 in May 26, 2021 maintenance update)
  - zeromq=4.3.1=he6710b0_3
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.3.7=h0b5b093_0
  - pip:
    - astunparse==1.6.3
    - azure-core==1.9.0
    - azure-storage-blob==12.6.0
    - databricks-cli==0.14.0
    - diskcache==5.1.0
    - docker==4.4.0
    - gorilla==0.3.0
    - horovod==0.20.3
    - joblibspark==0.3.0
    - keras-preprocessing==1.1.2
    - koalas==1.4.0
    - mleap==0.16.1
    - mlflow==1.12.1
    - msrest==0.6.19
    - opt-einsum==3.3.0
    - petastorm==0.9.7
    - pyarrow==1.0.1
    - pyyaml==5.3.1
    - querystring-parser==1.2.4
    - seaborn==0.10.0
    - spark-tensorflow-distributor==0.1.0
    - tensorboard==2.3.0
    - tensorboard-plugin-wit==1.7.0
    - tensorflow-cpu==2.3.1
    - tensorflow-estimator==2.3.0
    - termcolor==1.1.0
    - xgboost==1.2.1
prefix: /databricks/conda/envs/databricks-ml

GPU 叢集上的 Python 連結庫

name: databricks-ml-gpu
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.9.0=py37_0
  - asn1crypto=1.3.0=py37_1
  - astor=0.8.0=py37_0
  - backcall=0.1.0=py37_0
  - backports=1.0=pyhd3eb1b0_2
  - bcrypt=3.2.0=py37h7b6447c_0
  - blas=1.0=mkl
  - blinker=1.4=py37_0
  - boto3=1.12.0=py_0
  - botocore=1.15.0=py_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2020.10.14=h06a4308_1 # (updated from 0 in May 26, 2021 maintenance update)
  - cachetools=4.1.1=py_0
  - certifi=2020.11.8=py37h06a4308_0
  - cffi=1.14.0=py37he30daa8_1 # (updated from py37h2e261b9_0 in May 26, 2021 maintenance update)
  - chardet=3.0.4=py37h06a4308_1003
  - click=7.0=py37_0
  - cloudpickle=1.4.1=py_0
  - configparser=3.7.4=py37_0
  - cryptography=2.8=py37h1ba5d50_0
  - cudatoolkit=10.1.243=h6bb024c_0
  - cycler=0.10.0=py37_0
  - cython=0.29.15=py37he6710b0_0
  - decorator=4.4.1=py_0
  - dill=0.3.1.1=py37_1
  - docutils=0.15.2=py37_0
  - entrypoints=0.3=py37_0
  - flask=1.1.1=py_1
  - freetype=2.9.1=h8a8886c_1
  - future=0.18.2=py37_1
  - gast=0.3.3=py_0
  - gitdb=4.0.5=py_0
  - gitpython=3.1.0=py_0
  - google-auth=1.11.2=py_0
  - google-auth-oauthlib=0.4.1=py_2
  - google-pasta=0.2.0=py_0
  - grpcio=1.27.2=py37hf8bcb03_0
  - gunicorn=20.0.4=py37_0
  - h5py=2.10.0=py37h7918eee_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=he6710b0_3
  - idna=2.8=py37_0
  - intel-openmp=2020.0=166
  - ipykernel=5.1.4=py37h39e3cac_0
  - ipython=7.12.0=py37h5ca1d4c_0
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=py37_0
  - jedi=0.17.2=py37_0
  - jinja2=2.11.1=py_0
  - jmespath=0.10.0=py_0
  - joblib=0.14.1=py_0
  - jpeg=9b=h024ee3a_2
  - jupyter_client=5.3.4=py37_0
  - jupyter_core=4.6.1=py37_0
  - kiwisolver=1.1.0=py37he6710b0_0
  - krb5=1.17.1=h173b8e3_0 # (updated from 1.16.4 in May 26, 2021 maintenance update)
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20181209=hc058e9b_0
  - libffi=3.3=he6710b0_2 # (updated from 3.2.1 in May 26, 2021 maintenance update)
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libpng=1.6.37=hbc83047_0
  - libpq=12.2=h20c2e04_0 # (updated from 11.2 in May 26, 2021 maintenance update)
  - libprotobuf=3.11.4=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.1.0=h2733197_0
  - libuv=1.40.0=h7b6447c_0
  - lightgbm=2.3.0=py37he6710b0_0
  - lz4-c=1.8.1.2=h14c3975_0
  - mako=1.1.2=py_0
  - markdown=3.1.1=py37_0
  - markupsafe=1.1.1=py37h14c3975_1
  - matplotlib-base=3.1.3=py37hef1b27d_0
  - mkl=2020.0=166
  - mkl-service=2.3.0=py37he904b0f_0
  - mkl_fft=1.0.15=py37ha843d7b_0
  - mkl_random=1.1.0=py37hd6b4f25_0
  - ncurses=6.2=he6710b0_1
  - networkx=2.4=py_1
  - ninja=1.10.2=py37hff7bd54_0
  - nltk=3.4.5=py37_0
  - numpy=1.18.1=py37h4f9e942_0
  - numpy-base=1.18.1=py37hde5b4d6_1
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py37_0
  - openssl=1.1.1k=h27cfd23_0 # (updated from 1.1.1h in May 26, 2021 maintenance update)
  - packaging=20.1=py_0
  - pandas=1.0.1=py37h0573a6f_0
  - paramiko=2.7.1=py_0
  - parso=0.7.0=py_0
  - patsy=0.5.1=py37_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=py37_1001
  - pillow=7.0.0=py37hb39fc2d_0
  - pip=20.0.2=py37_3
  - plotly=4.12.0=pyhd3eb1b0_0
  - prompt_toolkit=3.0.3=py_0
  - protobuf=3.11.4=py37he6710b0_0
  - psutil=5.6.7=py37h7b6447c_0
  - psycopg2=2.8.6=py37h3c74f83_1 # (updated from 2.8.4 in May 26, 2021 maintenance update)
  - ptyprocess=0.6.0=pyhd3eb1b0_2
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.19=py37_0
  - pygments=2.5.2=py_0
  - pyjwt=1.7.1=py37_0
  - pynacl=1.3.0=py37h7b6447c_0
  - pyodbc=4.0.30=py37he6710b0_0
  - pyopenssl=19.1.0=pyhd3eb1b0_1
  - pyparsing=2.4.6=py_0
  - pysocks=1.7.1=py37_1
  - python=3.7.10=hdb3f193_0 # (updated from 3.7.6 in May 26, 2021 maintenance update)
  - python-dateutil=2.8.1=py_0
  - python-editor=1.0.4=py_0
  - pytorch=1.7.0=py3.7_cuda10.1.243_cudnn7.6.3_0
  - pytz=2019.3=py_0
  - pyzmq=18.1.1=py37he6710b0_0
  - readline=8.1=h27cfd23_0 # (updated from 7.0 in May 26, 2021 maintenance update)
  - requests=2.22.0=py37_1
  - requests-oauthlib=1.3.0=py_0
  - retrying=1.3.3=py37_2
  - rsa=4.0=py_0
  - s3transfer=0.3.3=py37_1
  - scikit-learn=0.22.1=py37hd81dba3_0
  - scipy=1.4.1=py37h0b6359f_0
  - setuptools=45.2.0=py37_0
  - simplejson=3.17.0=py37h7b6447c_0
  - six=1.14.0=py37h06a4308_0
  - smmap=3.0.4=py_0
  - sqlite=3.35.4=hdfb4753_0 # (updated from 3.31.1 in May 26, 2021 maintenance update)
  - sqlparse=0.4.1=py_0
  - statsmodels=0.11.0=py37h7b6447c_0
  - tabulate=0.8.3=py37_0
  - tk=8.6.10=hbc83047_0 # (updated from 8.6.8 in May 26, 2021 maintenance update)
  - torchvision=0.8.1=py37_cu101
  - tornado=6.0.3=py37h7b6447c_3
  - tqdm=4.42.1=py_0
  - traitlets=4.3.3=py37_0
  - typing_extensions=3.7.4.3=py_0
  - unixodbc=2.3.7=h14c3975_0
  - urllib3=1.25.8=py37_0
  - wcwidth=0.1.8=py_0
  - websocket-client=0.56.0=py37_0
  - werkzeug=1.0.0=py_0
  - wheel=0.34.2=py37_0
  - wrapt=1.11.2=py37h7b6447c_0
  - xz=5.2.5=h7b6447c_0 # (updated from 5.2.4 in May 26, 2021 maintenance update)
  - zeromq=4.3.1=he6710b0_3
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.3.7=h0b5b093_0
  - pip:
    - astunparse==1.6.3
    - azure-core==1.9.0
    - azure-storage-blob==12.6.0
    - databricks-cli==0.14.0
    - diskcache==5.1.0
    - docker==4.4.0
    - gorilla==0.3.0
    - horovod==0.20.3
    - joblibspark==0.3.0
    - keras-preprocessing==1.1.2
    - koalas==1.4.0
    - mleap==0.16.1
    - mlflow==1.12.1
    - msrest==0.6.19
    - opt-einsum==3.3.0
    - petastorm==0.9.7
    - pyarrow==1.0.1
    - pyyaml==5.3.1
    - querystring-parser==1.2.4
    - seaborn==0.10.0
    - spark-tensorflow-distributor==0.1.0
    - tensorboard==2.3.0
    - tensorboard-plugin-wit==1.7.0
    - tensorflow==2.3.1
    - tensorflow-estimator==2.3.0
    - termcolor==1.1.0
    - xgboost==1.2.1
prefix: /databricks/conda/envs/databricks-ml-gpu

包含 Python 模組的 Spark 套件

Spark 套件	Python 模組	版本
graphframes	graphframes	0.8.1-db1-spark3.0

R 程式庫

R 連結庫與 Databricks Runtime 7.5 中的 R 連結庫 相同。

Java 和 Scala 連結庫 （Scala 2.12 叢集）

除了 Databricks Runtime 7.5 中的 Java 和 Scala 連結庫之外，Databricks Runtime 7.5 ML 還包含下列 JAR：

CPU 叢集

群組識別碼	成品標識碼	版本
com.typesafe.akka	akka-actor_2.12	2.5.23
ml.combust.mleap	mleap-databricks-runtime_2.12	0.17.3-4882dc3
ml.dmlc	xgboost4j-spark_2.12	1.2.0
ml.dmlc	xgboost4j_2.12	1.2.0
org.mlflow	mlflow-client	1.12.1
org.scala-lang.modules	scala-java8-compat_2.12	0.8.0
org.tensorflow	spark-tensorflow-connector_2.12	1.15.0

GPU 叢集

群組識別碼	成品標識碼	版本
com.typesafe.akka	akka-actor_2.12	2.5.23
ml.combust.mleap	mleap-databricks-runtime_2.12	0.17.3-4882dc3
ml.dmlc	xgboost4j-spark-gpu_2.12	1.2.0
ml.dmlc	xgboost4j-gpu_2.12	1.2.0
org.mlflow	mlflow-client	1.12.1
org.scala-lang.modules	scala-java8-compat_2.12	0.8.0
org.tensorflow	spark-tensorflow-connector_2.12	1.15.0