Databricks Runtime 5.5 LTS för Machine Learning (stöds inte)

  • Artikel
  • 10 minuter för att läsa

Databricks släppte den här avbildningen i juli 2019. Supporten upphörde den 27 juli 2021. Databricks Runtime 5.5 ML utökad support (stöds inte) utökar 5,5 ML support till och med december 2021. Den använder Ubuntu 18.04.5 LTS i stället för den föråldrade Ubuntu 16.04.6 LTS-distributionen som användes i den ursprungliga Databricks Runtime 5.5 ML LTS. Stöd för Ubuntu 16.04.6 LTS den 1 april 2021.

Databricks Runtime 5.5 LTS för Machine Learning är en färdig miljö för maskininlärning och datavetenskap baserad på Databricks Runtime 5.5 LTS (stöds inte). Databricks Runtime ML innehåller många populära maskininlärningsbibliotek, inklusive TensorFlow, PyTorch, Keras och XGBoost. Den stöder även distribuerad djupinlärningsträning med Horovod.

Mer information, inklusive instruktioner för att skapa ett Databricks Runtime ML kluster, finns i Databricks Runtime för Machine Learning.

Nya funktioner

Databricks Runtime 5.5 LTS för Machine Learning bygger på Databricks Runtime 5.5 LTS. Information om vad som är nytt i Databricks Runtime 5.5 LTS finns i viktig information Databricks Runtime 5.5 LTS (stöds inte).

Förutom biblioteksuppdateringar introducerarDatabricks Runtime 5.5 LTS för Machine Learning följande nya funktioner:

Förbättringar

  • Uppgraderade maskininlärningsbibliotek

    • Tensorflow har uppgraderats från 1.12.0 till 1.13.1
    • PyTorch har uppgraderats från 0.4.1 till 1.1.0
    • scikit-learn har uppgraderats från 0.19.1 till 0.20.3

  • Ennodsåtgärd för HorovodRunner

    HorovodRunner har aktiverats för att endast köras på drivrutinsnoden. Om du tidigare vill använda HorovodRunner skulle du behöva köra en drivrutin och minst en arbetsnod. Med den här ändringen kan du nu distribuera träning inom en enda nod (det vill säga en nod med flera GPU:er) och därmed använda beräkningsresurser mer effektivt.

Utfasning

I hyperopt-biblioteket har vi föraktat följande egenskaper för :

  • SparkTrials.successful_trials_count
  • SparkTrials.failed_trials_count
  • SparkTrials.cancelled_trials_count
  • SparkTrials.total_trials_count

och ersatte egenskaperna med följande funktioner:

  • SparkTrials.count_successful_trials()
  • SparkTrials.count_failed_trials()
  • SparkTrials.count_cancelled_trials()
  • SparkTrials.count_total_trials()

Systemmiljö

Systemmiljön i Databricks Runtime 5.5 LTS för Machine Learning skiljer sig från Databricks Runtime 5.5 enligt följande:

  • Python:3.6.5 för Python 3-kluster och 2.7.15 för Python 2-kluster.
  • DBUtils:Innehåller inte biblioteksverktyget (dbutils.library).
  • För GPU-kluster, följande NVIDIA GPU-bibliotek:
    • CUDA 10.0
    • CUDNN 7.6.0

Bibliotek

I följande avsnitt listas de bibliotek som ingår i Databricks Runtime 5.5 LTS för Machine Learning som skiljer sig från de som ingår i Databricks Runtime 5.5.

Bibliotek på den översta nivån

Databricks Runtime 5.5 LTS för Machine Learning innehåller följande bibliotek på den översta nivån:

Python-bibliotek

Databricks Runtime 5.5 LTS för Machine Learning använder Conda för Python-pakethantering. Därför finns det stora skillnader i installerade Python-bibliotek jämfört med Databricks Runtime. I följande avsnitt beskrivs Conda-miljöer för Databricks Runtime 5.5 LTS för Machine Learning-kluster som använder Python 2 eller 3 och datorer med CPU- eller GPU-aktiverat.

Python 3 på CPU-kluster

name: null
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=2.0=cpu_0
  - _tflow_select=2.3.0=mkl
  - absl-py=0.7.1=py36_0
  - asn1crypto=0.24.0=py36_0
  - astor=0.7.1=py36_0
  - backcall=0.1.0=py36_0
  - backports=1.0=py_2
  - bcrypt=3.1.6=py36h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py36_0
  - boto=2.48.0=py36_1
  - boto3=1.7.62=py36h28b3542_1
  - botocore=1.10.62=py36h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - cffi=1.11.5=py36he75722e_1
  - chardet=3.0.4=py36_1
  - click=7.0=py36_0
  - cloudpickle=0.8.0=py36_0
  - colorama=0.3.9=py36h489cec4_0
  - configparser=3.7.3=py36_1
  - cryptography=2.2.2=py36h14c3975_0
  - cycler=0.10.0=py36h93f1223_0
  - cython=0.28.2=py36h14c3975_0
  - decorator=4.3.0=py36_0
  - docutils=0.14=py36hb0f60f5_0
  - entrypoints=0.2.3=py36_2
  - et_xmlfile=1.0.1=py36hd6bccc3_0
  - flask=1.0.2=py36_1
  - freetype=2.8=hab7d2ae_1
  - gast=0.2.2=py36_0
  - gitdb2=2.0.5=py36_0
  - gitpython=2.1.11=py36_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py36hdbcaa40_0
  - gunicorn=19.9.0=py36_0
  - h5py=2.8.0=py36h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py36_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py36h82fb2a8_1
  - intel-openmp=2018.0.0=8
  - ipython=6.4.0=py36_1
  - ipython_genutils=0.2.0=py36_0
  - itsdangerous=0.24=py36_1
  - jdcal=1.4=py36_0
  - jedi=0.12.0=py36_1
  - jinja2=2.10=py36_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py36_0
  - jupyter_client=5.2.3=py36_0
  - jupyter_core=4.4.0=py36_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py36_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=he6710b0_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - llvmlite=0.23.1=py36hdbcaa40_0
  - lxml=4.2.1=py36h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py36_0
  - markupsafe=1.0=py36h14c3975_1
  - mistune=0.8.3=py36h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py36ha843d7b_0
  - mkl_random=1.0.2=py36hd81dba3_0
  - mock=3.0.5=py36_0
  - msgpack-python=0.5.6=py36h6bb024c_1
  - nbconvert=5.3.1=py36_0
  - nbformat=4.4.0=py36h31c9010_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py36hfd86e86_0
  - numba=0.38.0=py36h637b7d7_0
  - numpy=1.16.2=py36h7e9f1db_0
  - numpy-base=1.16.2=py36hde5b4d6_0
  - olefile=0.45.1=py36_0
  - openpyxl=2.5.3=py36_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py36h637b7d7_0
  - pandocfilters=1.4.2=py36_1
  - paramiko=2.4.2=py36_0
  - parso=0.2.0=py36_0
  - pathlib2=2.3.2=py36_0
  - patsy=0.5.0=py36_0
  - pexpect=4.5.0=py36_0
  - pickleshare=0.7.4=py36_0
  - pillow=5.1.0=py36h3deb7b8_0
  - pip=10.0.1=py36_0
  - ply=3.11=py36_0
  - prompt_toolkit=1.0.15=py36h17d85b1_0
  - protobuf=3.8.0=py36he6710b0_0
  - psycopg2=2.7.5=py36hb7f436b_0
  - ptyprocess=0.5.2=py36h69acd42_0
  - py-xgboost=0.90=py36he6710b0_0
  - py-xgboost-cpu=0.90=py36_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py36_1
  - pygments=2.2.0=py36_0
  - pynacl=1.3.0=py36h7b6447c_0
  - pyopenssl=18.0.0=py36_0
  - pyparsing=2.2.0=py36_1
  - pysocks=1.6.8=py36_0
  - python=3.6.5=hc3d631a_2
  - python-dateutil=2.7.3=py36_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py36_0
  - pyyaml=5.1=py36h7b6447c_0
  - pyzmq=17.0.0=py36h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py36he2e5f8d_1
  - s3transfer=0.1.13=py36_0
  - scikit-learn=0.20.3=py36hd81dba3_0
  - scipy=1.1.0=py36h7c811a0_2
  - setuptools=39.1.0=py36_0
  - simplegeneric=0.8.1=py36_2
  - simplejson=3.16.0=py36h14c3975_0
  - singledispatch=3.4.0.3=py36_0
  - six=1.11.0=py36_1
  - smmap2=2.0.5=py36_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py36h035aef0_0
  - tabulate=0.8.3=py36_0
  - tensorboard=1.13.1=py36hf484d3e_0
  - tensorflow=1.13.1=mkl_py36h27d456a_0
  - tensorflow-base=1.13.1=mkl_py36h7ce6ba3_0
  - tensorflow-estimator=1.13.0=py_0
  - tensorflow-mkl=1.13.1=h4fcabd2_0
  - termcolor=1.1.0=py36_1
  - testpath=0.3.1=py36h8cadb63_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py36h14c3975_0
  - traitlets=4.3.2=py36_0
  - urllib3=1.22=py36hbe7ace6_0
  - virtualenv=16.0.0=py36_0
  - wcwidth=0.1.7=py36hdf4376a_0
  - webencodings=0.5.1=py36_1
  - werkzeug=0.14.1=py36_0
  - wheel=0.31.1=py36_0
  - wrapt=1.11.1=py36h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch-cpu=1.1.0=py3.6_cpu_0
  - torchvision-cpu=0.3.0=py36_cuNone_1
  - pip:
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - future==0.17.1
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - tensorboardx==1.7
    - torchvision==0.3.0
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python3

Python 3 på GPU-kluster

name: null
channels:
  - pytorch
  - Databricks
  - defaults
dependencies:
  - tensorflow=1.13.1.db1=gpu_py36h2903d8e_0
  - tensorflow-base=1.13.1.db1=gpu_py36he292aa2_0
  - tensorflow-gpu=1.13.1.db1=h0d30ee6_0
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=1.0=gpu_0
  - _tflow_select=2.1.0=gpu
  - absl-py=0.7.1=py36_0
  - asn1crypto=0.24.0=py36_0
  - astor=0.7.1=py36_0
  - backcall=0.1.0=py36_0
  - backports=1.0=py_2
  - bcrypt=3.1.6=py36h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py36_0
  - boto=2.48.0=py36_1
  - boto3=1.7.62=py36h28b3542_1
  - botocore=1.10.62=py36h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - cffi=1.11.5=py36he75722e_1
  - chardet=3.0.4=py36_1
  - click=7.0=py36_0
  - cloudpickle=0.8.0=py36_0
  - colorama=0.3.9=py36h489cec4_0
  - configparser=3.7.3=py36_1
  - cryptography=2.2.2=py36h14c3975_0
  - cudnn=7.6.0=cuda10.0_0
  - cupti=10.0.130=0
  - cycler=0.10.0=py36_0
  - cython=0.28.2=py36h14c3975_0
  - decorator=4.3.0=py36_0
  - docutils=0.14=py36_0
  - entrypoints=0.2.3=py36_2
  - et_xmlfile=1.0.1=py36hd6bccc3_0
  - flask=1.0.2=py36_1
  - freetype=2.8=hab7d2ae_1
  - gast=0.2.2=py36_0
  - gitdb2=2.0.5=py36_0
  - gitpython=2.1.11=py36_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py36hdbcaa40_0
  - gunicorn=19.9.0=py36_0
  - h5py=2.8.0=py36h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py36_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py36h82fb2a8_1
  - intel-openmp=2018.0.0=8
  - ipython=6.4.0=py36_1
  - ipython_genutils=0.2.0=py36hb52b0d5_0
  - itsdangerous=0.24=py36_1
  - jdcal=1.4=py36_0
  - jedi=0.12.0=py36_1
  - jinja2=2.10=py36_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py36_0
  - jupyter_client=5.2.3=py36_0
  - jupyter_core=4.4.0=py36_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py36_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=h688424c_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - llvmlite=0.23.1=py36hdbcaa40_0
  - lxml=4.2.1=py36h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py36_0
  - markupsafe=1.0=py36h14c3975_1
  - mistune=0.8.3=py36h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py36ha843d7b_0
  - mkl_random=1.0.2=py36hd81dba3_0
  - mock=3.0.5=py36_0
  - msgpack-python=0.5.6=py36h6bb024c_1
  - nbconvert=5.3.1=py36_0
  - nbformat=4.4.0=py36h31c9010_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py36hfd86e86_0
  - numba=0.38.0=py36h637b7d7_0
  - numpy=1.16.2=py36h7e9f1db_0
  - numpy-base=1.16.2=py36hde5b4d6_0
  - olefile=0.45.1=py36_0
  - openpyxl=2.5.3=py36_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py36h637b7d7_0
  - pandocfilters=1.4.2=py36_1
  - paramiko=2.4.2=py36_0
  - parso=0.2.0=py36_0
  - pathlib2=2.3.2=py36_0
  - patsy=0.5.0=py36_0
  - pexpect=4.5.0=py36_0
  - pickleshare=0.7.4=py36h63277f8_0
  - pillow=5.1.0=py36h3deb7b8_0
  - pip=10.0.1=py36_0
  - ply=3.11=py36_0
  - prompt_toolkit=1.0.15=py36_0
  - protobuf=3.8.0=py36he6710b0_0
  - psycopg2=2.7.5=py36hb7f436b_0
  - ptyprocess=0.5.2=py36h69acd42_0
  - py-xgboost=0.90=py36h688424c_0
  - py-xgboost-gpu=0.90=py36h28bbb66_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py36_1
  - pygments=2.2.0=py36_0
  - pynacl=1.3.0=py36h7b6447c_0
  - pyopenssl=18.0.0=py36_0
  - pyparsing=2.2.0=py36_1
  - pysocks=1.6.8=py36_0
  - python=3.6.5=hc3d631a_2
  - python-dateutil=2.7.3=py36_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py36_0
  - pyyaml=5.1=py36h7b6447c_0
  - pyzmq=17.0.0=py36h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py36he2e5f8d_1
  - s3transfer=0.1.13=py36_0
  - scikit-learn=0.20.3=py36hd81dba3_0
  - scipy=1.1.0=py36h7c811a0_2
  - setuptools=39.1.0=py36_0
  - simplegeneric=0.8.1=py36_2
  - simplejson=3.16.0=py36h14c3975_0
  - singledispatch=3.4.0.3=py36h7a266c3_0
  - six=1.11.0=py36_1
  - smmap2=2.0.5=py36_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py36h035aef0_0
  - tabulate=0.8.3=py36_0
  - tensorboard=1.13.1=py36hf484d3e_0
  - tensorflow-estimator=1.13.0=py_0
  - termcolor=1.1.0=py36_1
  - testpath=0.3.1=py36_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py36h14c3975_0
  - traitlets=4.3.2=py36h674d592_0
  - urllib3=1.22=py36hbe7ace6_0
  - virtualenv=16.0.0=py36_0
  - wcwidth=0.1.7=py36hdf4376a_0
  - webencodings=0.5.1=py36_1
  - werkzeug=0.14.1=py36_0
  - wheel=0.31.1=py36_0
  - wrapt=1.11.1=py36h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch=1.1.0=py3.6_cuda10.0.130_cudnn7.5.1_0
  - torchvision=0.3.0=py36_cu10.0.130_1
  - pip:
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - future==0.17.1
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - tensorboardx==1.7
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python3

Python 2 på CPU-kluster

name: null
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=2.0=cpu_0
  - _tflow_select=2.3.0=mkl
  - absl-py=0.7.1=py27_0
  - asn1crypto=0.24.0=py27_0
  - astor=0.7.1=py27_0
  - backports=1.0=py_2
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports.weakref=1.0.post1=py_1
  - backports_abc=0.5=py_0
  - bcrypt=3.1.6=py27h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py27_0
  - boto=2.48.0=py27_1
  - boto3=1.7.62=py27h28b3542_1
  - botocore=1.10.62=py27h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py27_0
  - cffi=1.11.5=py27he75722e_1
  - chardet=3.0.4=py27_1
  - click=7.0=py27_0
  - cloudpickle=0.8.0=py27_0
  - colorama=0.3.9=py27h5cde069_0
  - configparser=3.7.3=py27_1
  - cryptography=2.2.2=py27h14c3975_0
  - cycler=0.10.0=py27hc7354d3_0
  - cython=0.28.2=py27h14c3975_0
  - decorator=4.3.0=py27_0
  - docutils=0.14=py27_0
  - entrypoints=0.2.3=py27_2
  - enum34=1.1.6=py27_1
  - et_xmlfile=1.0.1=py27_0
  - flask=1.0.2=py27_1
  - freetype=2.8=hab7d2ae_1
  - funcsigs=1.0.2=py27_0
  - functools32=3.2.3.2=py27_1
  - future=0.17.1=py27_0
  - futures=3.2.0=py27_0
  - gast=0.2.2=py27_0
  - gitdb2=2.0.5=py27_0
  - gitpython=2.1.11=py27_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py27hdbcaa40_0
  - gunicorn=19.9.0=py27_0
  - h5py=2.8.0=py27h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py27_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py27h5722d68_1
  - intel-openmp=2018.0.0=8
  - ipaddress=1.0.22=py27_0
  - ipython=5.7.0=py27_0
  - ipython_genutils=0.2.0=py27_0
  - itsdangerous=0.24=py27_1
  - jdcal=1.4=py27_0
  - jinja2=2.10=py27_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py27h7ed5aa4_0
  - jupyter_client=5.2.3=py27_0
  - jupyter_core=4.4.0=py27_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py27_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=he6710b0_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - linecache2=1.0.0=py27_0
  - llvmlite=0.23.1=py27hdbcaa40_0
  - lxml=4.2.1=py27h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py27_0
  - markupsafe=1.0=py27h14c3975_1
  - mistune=0.8.3=py27h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py27ha843d7b_0
  - mkl_random=1.0.2=py27hd81dba3_0
  - mock=3.0.5=py27_0
  - msgpack-python=0.5.6=py27h6bb024c_1
  - nbconvert=5.3.1=py27_0
  - nbformat=4.4.0=py27hed7f2b2_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py27hfd86e86_0
  - numba=0.38.0=py27h637b7d7_0
  - numpy=1.16.2=py27h7e9f1db_0
  - numpy-base=1.16.2=py27hde5b4d6_0
  - olefile=0.45.1=py27_0
  - openpyxl=2.5.3=py27_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py27h637b7d7_0
  - pandocfilters=1.4.2=py27_1
  - paramiko=2.4.2=py27_0
  - pathlib2=2.3.2=py27_0
  - patsy=0.5.0=py27_0
  - pexpect=4.5.0=py27_0
  - pickleshare=0.7.4=py27_0
  - pillow=5.1.0=py27h3deb7b8_0
  - pip=10.0.1=py27_0
  - ply=3.11=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - protobuf=3.8.0=py27he6710b0_0
  - psycopg2=2.7.5=py27hb7f436b_0
  - ptyprocess=0.5.2=py27h4ccb14c_0
  - py-xgboost=0.90=py27he6710b0_0
  - py-xgboost-cpu=0.90=py27_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py27_1
  - pygments=2.2.0=py27_0
  - pynacl=1.3.0=py27h7b6447c_0
  - pyopenssl=18.0.0=py27_0
  - pyparsing=2.2.0=py27_1
  - pysocks=1.6.8=py27_0
  - python=2.7.15=h1571d57_0
  - python-dateutil=2.7.3=py27_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py27_0
  - pyyaml=5.1=py27h7b6447c_0
  - pyzmq=17.0.0=py27h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py27hc5b0589_1
  - s3transfer=0.1.13=py27_0
  - scandir=1.7=py27h14c3975_0
  - scikit-learn=0.20.3=py27hd81dba3_0
  - scipy=1.1.0=py27h7c811a0_2
  - setuptools=39.1.0=py27_0
  - simplegeneric=0.8.1=py27_2
  - simplejson=3.16.0=py27h14c3975_0
  - singledispatch=3.4.0.3=py27_0
  - six=1.11.0=py27_1
  - smmap2=2.0.5=py27_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py27h035aef0_0
  - tabulate=0.8.3=py27_0
  - tensorboard=1.13.1=py27hf484d3e_0
  - tensorflow=1.13.1=mkl_py27h74ee40f_0
  - tensorflow-base=1.13.1=mkl_py27h7ce6ba3_0
  - tensorflow-estimator=1.13.0=py_0
  - tensorflow-mkl=1.13.1=h4fcabd2_0
  - termcolor=1.1.0=py27_1
  - testpath=0.3.1=py27hc38d2c4_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py27h14c3975_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27_0
  - unittest2=1.1.0=py27_0
  - urllib3=1.22=py27ha55213b_0
  - virtualenv=16.0.0=py27_0
  - wcwidth=0.1.7=py27h9e3e1ab_0
  - webencodings=0.5.1=py27_1
  - werkzeug=0.14.1=py27_0
  - wheel=0.31.1=py27_0
  - wrapt=1.11.1=py27h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch-cpu=1.1.0=py2.7_cpu_0
  - torchvision-cpu=0.3.0=py27_cuNone_1
  - pip:
    - backports.functools-lru-cache==1.5
    - backports.ssl-match-hostname==3.7.0.1
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - subprocess32==3.5.4
    - tensorboardx==1.7
    - torchvision==0.3.0
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python2

Python 2 på GPU-kluster

name: null
channels:
  - Databricks
  - pytorch
  - defaults
dependencies:
  - tensorflow=1.13.1.db1=gpu_py27h8e347d7_0
  - tensorflow-base=1.13.1.db1=gpu_py27he292aa2_0
  - tensorflow-gpu=1.13.1.db1=h0d30ee6_0
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=1.0=gpu_0
  - _tflow_select=2.1.0=gpu
  - absl-py=0.7.1=py27_0
  - asn1crypto=0.24.0=py27_0
  - astor=0.7.1=py27_0
  - backports=1.0=py_2
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports.weakref=1.0.post1=py_1
  - backports_abc=0.5=py_0
  - bcrypt=3.1.6=py27h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py27_0
  - boto=2.48.0=py27_1
  - boto3=1.7.62=py27h28b3542_1
  - botocore=1.10.62=py27h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py27_0
  - cffi=1.11.5=py27he75722e_1
  - chardet=3.0.4=py27_1
  - click=7.0=py27_0
  - cloudpickle=0.8.0=py27_0
  - colorama=0.3.9=py27_0
  - configparser=3.7.3=py27_1
  - cryptography=2.2.2=py27h14c3975_0
  - cudnn=7.6.0=cuda10.0_0
  - cupti=10.0.130=0
  - cycler=0.10.0=py27_0
  - cython=0.28.2=py27h14c3975_0
  - decorator=4.3.0=py27_0
  - docutils=0.14=py27hae222c1_0
  - entrypoints=0.2.3=py27_2
  - enum34=1.1.6=py27_1
  - et_xmlfile=1.0.1=py27h75840f5_0
  - flask=1.0.2=py27_1
  - freetype=2.8=hab7d2ae_1
  - funcsigs=1.0.2=py27_0
  - functools32=3.2.3.2=py27_1
  - future=0.17.1=py27_0
  - futures=3.2.0=py27_0
  - gast=0.2.2=py27_0
  - gitdb2=2.0.5=py27_0
  - gitpython=2.1.11=py27_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py27hdbcaa40_0
  - gunicorn=19.9.0=py27_0
  - h5py=2.8.0=py27h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py27_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py27h5722d68_1
  - intel-openmp=2018.0.0=8
  - ipaddress=1.0.22=py27_0
  - ipython=5.7.0=py27_0
  - ipython_genutils=0.2.0=py27h89fb69b_0
  - itsdangerous=0.24=py27_1
  - jdcal=1.4=py27_0
  - jinja2=2.10=py27_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py27h7ed5aa4_0
  - jupyter_client=5.2.3=py27_0
  - jupyter_core=4.4.0=py27_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py27_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=h688424c_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - linecache2=1.0.0=py27_0
  - llvmlite=0.23.1=py27hdbcaa40_0
  - lxml=4.2.1=py27h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py27_0
  - markupsafe=1.0=py27h14c3975_1
  - mistune=0.8.3=py27h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py27ha843d7b_0
  - mkl_random=1.0.2=py27hd81dba3_0
  - mock=3.0.5=py27_0
  - msgpack-python=0.5.6=py27h6bb024c_1
  - nbconvert=5.3.1=py27_0
  - nbformat=4.4.0=py27hed7f2b2_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py27hfd86e86_0
  - numba=0.38.0=py27h637b7d7_0
  - numpy=1.16.2=py27h7e9f1db_0
  - numpy-base=1.16.2=py27hde5b4d6_0
  - olefile=0.45.1=py27_0
  - openpyxl=2.5.3=py27_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py27h637b7d7_0
  - pandocfilters=1.4.2=py27_1
  - paramiko=2.4.2=py27_0
  - pathlib2=2.3.2=py27_0
  - patsy=0.5.0=py27_0
  - pexpect=4.5.0=py27_0
  - pickleshare=0.7.4=py27h09770e1_0
  - pillow=5.1.0=py27h3deb7b8_0
  - pip=10.0.1=py27_0
  - ply=3.11=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - protobuf=3.8.0=py27he6710b0_0
  - psycopg2=2.7.5=py27hb7f436b_0
  - ptyprocess=0.5.2=py27h4ccb14c_0
  - py-xgboost=0.90=py27h688424c_0
  - py-xgboost-gpu=0.90=py27h28bbb66_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py27_1
  - pygments=2.2.0=py27_0
  - pynacl=1.3.0=py27h7b6447c_0
  - pyopenssl=18.0.0=py27_0
  - pyparsing=2.2.0=py27_1
  - pysocks=1.6.8=py27_0
  - python=2.7.15=h1571d57_0
  - python-dateutil=2.7.3=py27_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py27_0
  - pyyaml=5.1=py27h7b6447c_0
  - pyzmq=17.0.0=py27h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py27hc5b0589_1
  - s3transfer=0.1.13=py27_0
  - scandir=1.7=py27h14c3975_0
  - scikit-learn=0.20.3=py27hd81dba3_0
  - scipy=1.1.0=py27h7c811a0_2
  - setuptools=39.1.0=py27_0
  - simplegeneric=0.8.1=py27_2
  - simplejson=3.16.0=py27h14c3975_0
  - singledispatch=3.4.0.3=py27h9bcb476_0
  - six=1.11.0=py27_1
  - smmap2=2.0.5=py27_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py27h035aef0_0
  - tabulate=0.8.3=py27_0
  - tensorboard=1.13.1=py27hf484d3e_0
  - tensorflow-estimator=1.13.0=py_0
  - termcolor=1.1.0=py27_1
  - testpath=0.3.1=py27_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py27h14c3975_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27hd6ce930_0
  - unittest2=1.1.0=py27_0
  - urllib3=1.22=py27ha55213b_0
  - virtualenv=16.0.0=py27_0
  - wcwidth=0.1.7=py27_0
  - webencodings=0.5.1=py27_1
  - werkzeug=0.14.1=py27_0
  - wheel=0.31.1=py27_0
  - wrapt=1.11.1=py27h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch=1.1.0=py2.7_cuda10.0.130_cudnn7.5.1_0
  - torchvision=0.3.0=py27_cu10.0.130_1
  - pip:
    - backports.functools-lru-cache==1.5
    - backports.ssl-match-hostname==3.7.0.1
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - subprocess32==3.5.4
    - tensorboardx==1.7
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python2

Spark-paket som innehåller Python-moduler

Spark-paket Python-modul Version
graphframes graphframes 0.7.0-db1-spark2.4
spark-deep-learning sparkdl 1.5.0-db4-spark2.4
tensorframes tensorframes 0.7.0-s_2.11

R-bibliotek

R-biblioteken är identiska med R-biblioteken i Databricks Runtime 5.5.

Java- och Scala-bibliotek (Scala 2.11-kluster)

Förutom Java- och Scala-biblioteken i Databricks Runtime 5.5 innehåller Databricks Runtime 5.5 LTS för Machine Learning följande JAR:er:

Grupp-ID Artefakt-ID Version
com.databricks spark-deep-learning 1.5.0-db4-spark2.4
com.typesafe.akka akka-actor_2.11 2.3.11
ml.combust.mleap mleap-databricks-runtime_2.11 0.13.0
ml.dmlc xgboost4j 0,90
ml.dmlc xgboost4j-spark 0,90
org.graphframes graphframes_2.11 0.7.0-db1-spark2.4
org.tensorflow libtensorflow 1.13.1
org.tensorflow libtensorflow_jni 1.13.1
org.tensorflow spark-tensorflow-connector_2.11 1.13.1
org.tensorflow tensorflow 1.13.1
org.tensorframes tensorframes 0.7.0-s_2.11