Databricks Runtime 5,5 LTS för Machine Learning

Databricks släpptes den här avbildningen i juli 2019.

Databricks Runtime 5,5 LTS för Machine Learning är en färdig miljö för maskin inlärning och data vetenskap baserat på Databricks Runtime 5,5 LTS. Databricks Runtime ML innehåller många populära maskin inlärnings bibliotek, inklusive TensorFlow, PyTorch, keras och XGBoost. Den har också stöd för distribuerad djup inlärnings utbildning med Horovod.

Mer information, inklusive anvisningar för hur du skapar ett Databricks Runtime ML-kluster, finns Databricks runtime för Machine Learning.

Nya funktioner

Databricks Runtime 5,5 LTS för Machine Learning skapas ovanpå Databricks Runtime 5,5 LTS. Information om vad som är nytt i Databricks Runtime 5,5-LTS finns i viktig information om Databricks Runtime 5,5 LTS .

Förutom biblioteks uppdateringarhar DATABRICKS runtime 5,5 LTS för Machine Learning introducerar följande nya funktioner:

  • MLflow 1,0 python-paketet har lagts till

Bättre

  • Uppgraderade Machine Learning-bibliotek

    • Tensorflow har uppgraderats från 1.12.0 till 1.13.1
    • PyTorch har uppgraderats från 0.4.1 till 1.1.0
    • scikit – lär dig uppgraderade från 0.19.1 till 0.20.3
  • Åtgärd med en nod för HorovodRunner

    Aktiverade HorovodRunner som bara körs på driv rutins-noden. Tidigare skulle du behöva köra en driv rutin och minst en arbetsnod för att använda HorovodRunner. Med den här ändringen kan du nu distribuera utbildning inom en enda nod (det vill säga en nod med flera GPU: n) och därmed använda beräknings resurserna på ett mer effektivt sätt.

Utfasning

I hyperopt -biblioteket har vi föråldrat följande egenskaper för hyperopt.SparkTrials :

  • SparkTrials.successful_trials_count
  • SparkTrials.failed_trials_count
  • SparkTrials.cancelled_trials_count
  • SparkTrials.total_trials_count

och ersatte egenskaperna med följande funktioner:

  • SparkTrials.count_successful_trials()
  • SparkTrials.count_failed_trials()
  • SparkTrials.count_cancelled_trials()
  • SparkTrials.count_total_trials()

Systemmiljö

System miljön i Databricks Runtime 5,5 LTS för Machine Learning skiljer sig från Databricks Runtime 5,5 på följande sätt:

  • Python: 3.6.5 för python 3-kluster och 2.7.15 för python 2-kluster.
  • DBUtils: innehåller inte biblioteks verktyg.
  • För GPU-kluster är följande NVIDIA GPU-bibliotek:
    • CUDA 10,0
    • CUDNN 7.6.0

Bibliotek

I följande avsnitt listas de bibliotek som ingår i Databricks Runtime 5,5-LTS för Machine Learning som skiljer sig från de som ingår i Databricks Runtime 5,5.

Bibliotek på översta nivån

Databricks Runtime 5,5-LTS för Machine Learning innehåller följande bibliotekpå den översta nivån:

Python-bibliotek

Databricks Runtime 5,5 LTS för Machine Learning använder Conda för hantering av python-paket. Därför finns det stora skillnader i installerade python-bibliotek jämfört med Databricks Runtime. I följande avsnitt beskrivs Conda-miljöer för Databricks Runtime 5,5-LTS för Machine Learning kluster med python 2 eller 3, och processor-eller GPU-aktiverade datorer.

Python 3 på CPU-kluster

name: null
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=2.0=cpu_0
  - _tflow_select=2.3.0=mkl
  - absl-py=0.7.1=py36_0
  - asn1crypto=0.24.0=py36_0
  - astor=0.7.1=py36_0
  - backcall=0.1.0=py36_0
  - backports=1.0=py_2
  - bcrypt=3.1.6=py36h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py36_0
  - boto=2.48.0=py36_1
  - boto3=1.7.62=py36h28b3542_1
  - botocore=1.10.62=py36h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - cffi=1.11.5=py36he75722e_1
  - chardet=3.0.4=py36_1
  - click=7.0=py36_0
  - cloudpickle=0.8.0=py36_0
  - colorama=0.3.9=py36h489cec4_0
  - configparser=3.7.3=py36_1
  - cryptography=2.2.2=py36h14c3975_0
  - cycler=0.10.0=py36h93f1223_0
  - cython=0.28.2=py36h14c3975_0
  - decorator=4.3.0=py36_0
  - docutils=0.14=py36hb0f60f5_0
  - entrypoints=0.2.3=py36_2
  - et_xmlfile=1.0.1=py36hd6bccc3_0
  - flask=1.0.2=py36_1
  - freetype=2.8=hab7d2ae_1
  - gast=0.2.2=py36_0
  - gitdb2=2.0.5=py36_0
  - gitpython=2.1.11=py36_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py36hdbcaa40_0
  - gunicorn=19.9.0=py36_0
  - h5py=2.8.0=py36h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py36_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py36h82fb2a8_1
  - intel-openmp=2018.0.0=8
  - ipython=6.4.0=py36_1
  - ipython_genutils=0.2.0=py36_0
  - itsdangerous=0.24=py36_1
  - jdcal=1.4=py36_0
  - jedi=0.12.0=py36_1
  - jinja2=2.10=py36_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py36_0
  - jupyter_client=5.2.3=py36_0
  - jupyter_core=4.4.0=py36_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py36_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=he6710b0_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - llvmlite=0.23.1=py36hdbcaa40_0
  - lxml=4.2.1=py36h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py36_0
  - markupsafe=1.0=py36h14c3975_1
  - mistune=0.8.3=py36h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py36ha843d7b_0
  - mkl_random=1.0.2=py36hd81dba3_0
  - mock=3.0.5=py36_0
  - msgpack-python=0.5.6=py36h6bb024c_1
  - nbconvert=5.3.1=py36_0
  - nbformat=4.4.0=py36h31c9010_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py36hfd86e86_0
  - numba=0.38.0=py36h637b7d7_0
  - numpy=1.16.2=py36h7e9f1db_0
  - numpy-base=1.16.2=py36hde5b4d6_0
  - olefile=0.45.1=py36_0
  - openpyxl=2.5.3=py36_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py36h637b7d7_0
  - pandocfilters=1.4.2=py36_1
  - paramiko=2.4.2=py36_0
  - parso=0.2.0=py36_0
  - pathlib2=2.3.2=py36_0
  - patsy=0.5.0=py36_0
  - pexpect=4.5.0=py36_0
  - pickleshare=0.7.4=py36_0
  - pillow=5.1.0=py36h3deb7b8_0
  - pip=10.0.1=py36_0
  - ply=3.11=py36_0
  - prompt_toolkit=1.0.15=py36h17d85b1_0
  - protobuf=3.8.0=py36he6710b0_0
  - psycopg2=2.7.5=py36hb7f436b_0
  - ptyprocess=0.5.2=py36h69acd42_0
  - py-xgboost=0.90=py36he6710b0_0
  - py-xgboost-cpu=0.90=py36_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py36_1
  - pygments=2.2.0=py36_0
  - pynacl=1.3.0=py36h7b6447c_0
  - pyopenssl=18.0.0=py36_0
  - pyparsing=2.2.0=py36_1
  - pysocks=1.6.8=py36_0
  - python=3.6.5=hc3d631a_2
  - python-dateutil=2.7.3=py36_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py36_0
  - pyyaml=5.1=py36h7b6447c_0
  - pyzmq=17.0.0=py36h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py36he2e5f8d_1
  - s3transfer=0.1.13=py36_0
  - scikit-learn=0.20.3=py36hd81dba3_0
  - scipy=1.1.0=py36h7c811a0_2
  - setuptools=39.1.0=py36_0
  - simplegeneric=0.8.1=py36_2
  - simplejson=3.16.0=py36h14c3975_0
  - singledispatch=3.4.0.3=py36_0
  - six=1.11.0=py36_1
  - smmap2=2.0.5=py36_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py36h035aef0_0
  - tabulate=0.8.3=py36_0
  - tensorboard=1.13.1=py36hf484d3e_0
  - tensorflow=1.13.1=mkl_py36h27d456a_0
  - tensorflow-base=1.13.1=mkl_py36h7ce6ba3_0
  - tensorflow-estimator=1.13.0=py_0
  - tensorflow-mkl=1.13.1=h4fcabd2_0
  - termcolor=1.1.0=py36_1
  - testpath=0.3.1=py36h8cadb63_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py36h14c3975_0
  - traitlets=4.3.2=py36_0
  - urllib3=1.22=py36hbe7ace6_0
  - virtualenv=16.0.0=py36_0
  - wcwidth=0.1.7=py36hdf4376a_0
  - webencodings=0.5.1=py36_1
  - werkzeug=0.14.1=py36_0
  - wheel=0.31.1=py36_0
  - wrapt=1.11.1=py36h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch-cpu=1.1.0=py3.6_cpu_0
  - torchvision-cpu=0.3.0=py36_cuNone_1
  - pip:
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - future==0.17.1
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - tensorboardx==1.7
    - torchvision==0.3.0
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python3

Python 3 på GPU-kluster

name: null
channels:
  - pytorch
  - Databricks
  - defaults
dependencies:
  - tensorflow=1.13.1.db1=gpu_py36h2903d8e_0
  - tensorflow-base=1.13.1.db1=gpu_py36he292aa2_0
  - tensorflow-gpu=1.13.1.db1=h0d30ee6_0
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=1.0=gpu_0
  - _tflow_select=2.1.0=gpu
  - absl-py=0.7.1=py36_0
  - asn1crypto=0.24.0=py36_0
  - astor=0.7.1=py36_0
  - backcall=0.1.0=py36_0
  - backports=1.0=py_2
  - bcrypt=3.1.6=py36h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py36_0
  - boto=2.48.0=py36_1
  - boto3=1.7.62=py36h28b3542_1
  - botocore=1.10.62=py36h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - cffi=1.11.5=py36he75722e_1
  - chardet=3.0.4=py36_1
  - click=7.0=py36_0
  - cloudpickle=0.8.0=py36_0
  - colorama=0.3.9=py36h489cec4_0
  - configparser=3.7.3=py36_1
  - cryptography=2.2.2=py36h14c3975_0
  - cudnn=7.6.0=cuda10.0_0
  - cupti=10.0.130=0
  - cycler=0.10.0=py36_0
  - cython=0.28.2=py36h14c3975_0
  - decorator=4.3.0=py36_0
  - docutils=0.14=py36_0
  - entrypoints=0.2.3=py36_2
  - et_xmlfile=1.0.1=py36hd6bccc3_0
  - flask=1.0.2=py36_1
  - freetype=2.8=hab7d2ae_1
  - gast=0.2.2=py36_0
  - gitdb2=2.0.5=py36_0
  - gitpython=2.1.11=py36_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py36hdbcaa40_0
  - gunicorn=19.9.0=py36_0
  - h5py=2.8.0=py36h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py36_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py36h82fb2a8_1
  - intel-openmp=2018.0.0=8
  - ipython=6.4.0=py36_1
  - ipython_genutils=0.2.0=py36hb52b0d5_0
  - itsdangerous=0.24=py36_1
  - jdcal=1.4=py36_0
  - jedi=0.12.0=py36_1
  - jinja2=2.10=py36_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py36_0
  - jupyter_client=5.2.3=py36_0
  - jupyter_core=4.4.0=py36_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py36_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=h688424c_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - llvmlite=0.23.1=py36hdbcaa40_0
  - lxml=4.2.1=py36h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py36_0
  - markupsafe=1.0=py36h14c3975_1
  - mistune=0.8.3=py36h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py36ha843d7b_0
  - mkl_random=1.0.2=py36hd81dba3_0
  - mock=3.0.5=py36_0
  - msgpack-python=0.5.6=py36h6bb024c_1
  - nbconvert=5.3.1=py36_0
  - nbformat=4.4.0=py36h31c9010_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py36hfd86e86_0
  - numba=0.38.0=py36h637b7d7_0
  - numpy=1.16.2=py36h7e9f1db_0
  - numpy-base=1.16.2=py36hde5b4d6_0
  - olefile=0.45.1=py36_0
  - openpyxl=2.5.3=py36_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py36h637b7d7_0
  - pandocfilters=1.4.2=py36_1
  - paramiko=2.4.2=py36_0
  - parso=0.2.0=py36_0
  - pathlib2=2.3.2=py36_0
  - patsy=0.5.0=py36_0
  - pexpect=4.5.0=py36_0
  - pickleshare=0.7.4=py36h63277f8_0
  - pillow=5.1.0=py36h3deb7b8_0
  - pip=10.0.1=py36_0
  - ply=3.11=py36_0
  - prompt_toolkit=1.0.15=py36_0
  - protobuf=3.8.0=py36he6710b0_0
  - psycopg2=2.7.5=py36hb7f436b_0
  - ptyprocess=0.5.2=py36h69acd42_0
  - py-xgboost=0.90=py36h688424c_0
  - py-xgboost-gpu=0.90=py36h28bbb66_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py36_1
  - pygments=2.2.0=py36_0
  - pynacl=1.3.0=py36h7b6447c_0
  - pyopenssl=18.0.0=py36_0
  - pyparsing=2.2.0=py36_1
  - pysocks=1.6.8=py36_0
  - python=3.6.5=hc3d631a_2
  - python-dateutil=2.7.3=py36_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py36_0
  - pyyaml=5.1=py36h7b6447c_0
  - pyzmq=17.0.0=py36h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py36he2e5f8d_1
  - s3transfer=0.1.13=py36_0
  - scikit-learn=0.20.3=py36hd81dba3_0
  - scipy=1.1.0=py36h7c811a0_2
  - setuptools=39.1.0=py36_0
  - simplegeneric=0.8.1=py36_2
  - simplejson=3.16.0=py36h14c3975_0
  - singledispatch=3.4.0.3=py36h7a266c3_0
  - six=1.11.0=py36_1
  - smmap2=2.0.5=py36_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py36h035aef0_0
  - tabulate=0.8.3=py36_0
  - tensorboard=1.13.1=py36hf484d3e_0
  - tensorflow-estimator=1.13.0=py_0
  - termcolor=1.1.0=py36_1
  - testpath=0.3.1=py36_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py36h14c3975_0
  - traitlets=4.3.2=py36h674d592_0
  - urllib3=1.22=py36hbe7ace6_0
  - virtualenv=16.0.0=py36_0
  - wcwidth=0.1.7=py36hdf4376a_0
  - webencodings=0.5.1=py36_1
  - werkzeug=0.14.1=py36_0
  - wheel=0.31.1=py36_0
  - wrapt=1.11.1=py36h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch=1.1.0=py3.6_cuda10.0.130_cudnn7.5.1_0
  - torchvision=0.3.0=py36_cu10.0.130_1
  - pip:
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - future==0.17.1
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - tensorboardx==1.7
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python3

Python 2 på CPU-kluster

name: null
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=2.0=cpu_0
  - _tflow_select=2.3.0=mkl
  - absl-py=0.7.1=py27_0
  - asn1crypto=0.24.0=py27_0
  - astor=0.7.1=py27_0
  - backports=1.0=py_2
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports.weakref=1.0.post1=py_1
  - backports_abc=0.5=py_0
  - bcrypt=3.1.6=py27h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py27_0
  - boto=2.48.0=py27_1
  - boto3=1.7.62=py27h28b3542_1
  - botocore=1.10.62=py27h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py27_0
  - cffi=1.11.5=py27he75722e_1
  - chardet=3.0.4=py27_1
  - click=7.0=py27_0
  - cloudpickle=0.8.0=py27_0
  - colorama=0.3.9=py27h5cde069_0
  - configparser=3.7.3=py27_1
  - cryptography=2.2.2=py27h14c3975_0
  - cycler=0.10.0=py27hc7354d3_0
  - cython=0.28.2=py27h14c3975_0
  - decorator=4.3.0=py27_0
  - docutils=0.14=py27_0
  - entrypoints=0.2.3=py27_2
  - enum34=1.1.6=py27_1
  - et_xmlfile=1.0.1=py27_0
  - flask=1.0.2=py27_1
  - freetype=2.8=hab7d2ae_1
  - funcsigs=1.0.2=py27_0
  - functools32=3.2.3.2=py27_1
  - future=0.17.1=py27_0
  - futures=3.2.0=py27_0
  - gast=0.2.2=py27_0
  - gitdb2=2.0.5=py27_0
  - gitpython=2.1.11=py27_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py27hdbcaa40_0
  - gunicorn=19.9.0=py27_0
  - h5py=2.8.0=py27h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py27_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py27h5722d68_1
  - intel-openmp=2018.0.0=8
  - ipaddress=1.0.22=py27_0
  - ipython=5.7.0=py27_0
  - ipython_genutils=0.2.0=py27_0
  - itsdangerous=0.24=py27_1
  - jdcal=1.4=py27_0
  - jinja2=2.10=py27_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py27h7ed5aa4_0
  - jupyter_client=5.2.3=py27_0
  - jupyter_core=4.4.0=py27_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py27_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=he6710b0_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - linecache2=1.0.0=py27_0
  - llvmlite=0.23.1=py27hdbcaa40_0
  - lxml=4.2.1=py27h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py27_0
  - markupsafe=1.0=py27h14c3975_1
  - mistune=0.8.3=py27h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py27ha843d7b_0
  - mkl_random=1.0.2=py27hd81dba3_0
  - mock=3.0.5=py27_0
  - msgpack-python=0.5.6=py27h6bb024c_1
  - nbconvert=5.3.1=py27_0
  - nbformat=4.4.0=py27hed7f2b2_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py27hfd86e86_0
  - numba=0.38.0=py27h637b7d7_0
  - numpy=1.16.2=py27h7e9f1db_0
  - numpy-base=1.16.2=py27hde5b4d6_0
  - olefile=0.45.1=py27_0
  - openpyxl=2.5.3=py27_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py27h637b7d7_0
  - pandocfilters=1.4.2=py27_1
  - paramiko=2.4.2=py27_0
  - pathlib2=2.3.2=py27_0
  - patsy=0.5.0=py27_0
  - pexpect=4.5.0=py27_0
  - pickleshare=0.7.4=py27_0
  - pillow=5.1.0=py27h3deb7b8_0
  - pip=10.0.1=py27_0
  - ply=3.11=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - protobuf=3.8.0=py27he6710b0_0
  - psycopg2=2.7.5=py27hb7f436b_0
  - ptyprocess=0.5.2=py27h4ccb14c_0
  - py-xgboost=0.90=py27he6710b0_0
  - py-xgboost-cpu=0.90=py27_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py27_1
  - pygments=2.2.0=py27_0
  - pynacl=1.3.0=py27h7b6447c_0
  - pyopenssl=18.0.0=py27_0
  - pyparsing=2.2.0=py27_1
  - pysocks=1.6.8=py27_0
  - python=2.7.15=h1571d57_0
  - python-dateutil=2.7.3=py27_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py27_0
  - pyyaml=5.1=py27h7b6447c_0
  - pyzmq=17.0.0=py27h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py27hc5b0589_1
  - s3transfer=0.1.13=py27_0
  - scandir=1.7=py27h14c3975_0
  - scikit-learn=0.20.3=py27hd81dba3_0
  - scipy=1.1.0=py27h7c811a0_2
  - setuptools=39.1.0=py27_0
  - simplegeneric=0.8.1=py27_2
  - simplejson=3.16.0=py27h14c3975_0
  - singledispatch=3.4.0.3=py27_0
  - six=1.11.0=py27_1
  - smmap2=2.0.5=py27_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py27h035aef0_0
  - tabulate=0.8.3=py27_0
  - tensorboard=1.13.1=py27hf484d3e_0
  - tensorflow=1.13.1=mkl_py27h74ee40f_0
  - tensorflow-base=1.13.1=mkl_py27h7ce6ba3_0
  - tensorflow-estimator=1.13.0=py_0
  - tensorflow-mkl=1.13.1=h4fcabd2_0
  - termcolor=1.1.0=py27_1
  - testpath=0.3.1=py27hc38d2c4_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py27h14c3975_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27_0
  - unittest2=1.1.0=py27_0
  - urllib3=1.22=py27ha55213b_0
  - virtualenv=16.0.0=py27_0
  - wcwidth=0.1.7=py27h9e3e1ab_0
  - webencodings=0.5.1=py27_1
  - werkzeug=0.14.1=py27_0
  - wheel=0.31.1=py27_0
  - wrapt=1.11.1=py27h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch-cpu=1.1.0=py2.7_cpu_0
  - torchvision-cpu=0.3.0=py27_cuNone_1
  - pip:
    - backports.functools-lru-cache==1.5
    - backports.ssl-match-hostname==3.7.0.1
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - subprocess32==3.5.4
    - tensorboardx==1.7
    - torchvision==0.3.0
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python2

Python 2 på GPU-kluster

name: null
channels:
  - Databricks
  - pytorch
  - defaults
dependencies:
  - tensorflow=1.13.1.db1=gpu_py27h8e347d7_0
  - tensorflow-base=1.13.1.db1=gpu_py27he292aa2_0
  - tensorflow-gpu=1.13.1.db1=h0d30ee6_0
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=1.0=gpu_0
  - _tflow_select=2.1.0=gpu
  - absl-py=0.7.1=py27_0
  - asn1crypto=0.24.0=py27_0
  - astor=0.7.1=py27_0
  - backports=1.0=py_2
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports.weakref=1.0.post1=py_1
  - backports_abc=0.5=py_0
  - bcrypt=3.1.6=py27h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py27_0
  - boto=2.48.0=py27_1
  - boto3=1.7.62=py27h28b3542_1
  - botocore=1.10.62=py27h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py27_0
  - cffi=1.11.5=py27he75722e_1
  - chardet=3.0.4=py27_1
  - click=7.0=py27_0
  - cloudpickle=0.8.0=py27_0
  - colorama=0.3.9=py27_0
  - configparser=3.7.3=py27_1
  - cryptography=2.2.2=py27h14c3975_0
  - cudnn=7.6.0=cuda10.0_0
  - cupti=10.0.130=0
  - cycler=0.10.0=py27_0
  - cython=0.28.2=py27h14c3975_0
  - decorator=4.3.0=py27_0
  - docutils=0.14=py27hae222c1_0
  - entrypoints=0.2.3=py27_2
  - enum34=1.1.6=py27_1
  - et_xmlfile=1.0.1=py27h75840f5_0
  - flask=1.0.2=py27_1
  - freetype=2.8=hab7d2ae_1
  - funcsigs=1.0.2=py27_0
  - functools32=3.2.3.2=py27_1
  - future=0.17.1=py27_0
  - futures=3.2.0=py27_0
  - gast=0.2.2=py27_0
  - gitdb2=2.0.5=py27_0
  - gitpython=2.1.11=py27_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py27hdbcaa40_0
  - gunicorn=19.9.0=py27_0
  - h5py=2.8.0=py27h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py27_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py27h5722d68_1
  - intel-openmp=2018.0.0=8
  - ipaddress=1.0.22=py27_0
  - ipython=5.7.0=py27_0
  - ipython_genutils=0.2.0=py27h89fb69b_0
  - itsdangerous=0.24=py27_1
  - jdcal=1.4=py27_0
  - jinja2=2.10=py27_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py27h7ed5aa4_0
  - jupyter_client=5.2.3=py27_0
  - jupyter_core=4.4.0=py27_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py27_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=h688424c_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - linecache2=1.0.0=py27_0
  - llvmlite=0.23.1=py27hdbcaa40_0
  - lxml=4.2.1=py27h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py27_0
  - markupsafe=1.0=py27h14c3975_1
  - mistune=0.8.3=py27h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py27ha843d7b_0
  - mkl_random=1.0.2=py27hd81dba3_0
  - mock=3.0.5=py27_0
  - msgpack-python=0.5.6=py27h6bb024c_1
  - nbconvert=5.3.1=py27_0
  - nbformat=4.4.0=py27hed7f2b2_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py27hfd86e86_0
  - numba=0.38.0=py27h637b7d7_0
  - numpy=1.16.2=py27h7e9f1db_0
  - numpy-base=1.16.2=py27hde5b4d6_0
  - olefile=0.45.1=py27_0
  - openpyxl=2.5.3=py27_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py27h637b7d7_0
  - pandocfilters=1.4.2=py27_1
  - paramiko=2.4.2=py27_0
  - pathlib2=2.3.2=py27_0
  - patsy=0.5.0=py27_0
  - pexpect=4.5.0=py27_0
  - pickleshare=0.7.4=py27h09770e1_0
  - pillow=5.1.0=py27h3deb7b8_0
  - pip=10.0.1=py27_0
  - ply=3.11=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - protobuf=3.8.0=py27he6710b0_0
  - psycopg2=2.7.5=py27hb7f436b_0
  - ptyprocess=0.5.2=py27h4ccb14c_0
  - py-xgboost=0.90=py27h688424c_0
  - py-xgboost-gpu=0.90=py27h28bbb66_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py27_1
  - pygments=2.2.0=py27_0
  - pynacl=1.3.0=py27h7b6447c_0
  - pyopenssl=18.0.0=py27_0
  - pyparsing=2.2.0=py27_1
  - pysocks=1.6.8=py27_0
  - python=2.7.15=h1571d57_0
  - python-dateutil=2.7.3=py27_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py27_0
  - pyyaml=5.1=py27h7b6447c_0
  - pyzmq=17.0.0=py27h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py27hc5b0589_1
  - s3transfer=0.1.13=py27_0
  - scandir=1.7=py27h14c3975_0
  - scikit-learn=0.20.3=py27hd81dba3_0
  - scipy=1.1.0=py27h7c811a0_2
  - setuptools=39.1.0=py27_0
  - simplegeneric=0.8.1=py27_2
  - simplejson=3.16.0=py27h14c3975_0
  - singledispatch=3.4.0.3=py27h9bcb476_0
  - six=1.11.0=py27_1
  - smmap2=2.0.5=py27_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py27h035aef0_0
  - tabulate=0.8.3=py27_0
  - tensorboard=1.13.1=py27hf484d3e_0
  - tensorflow-estimator=1.13.0=py_0
  - termcolor=1.1.0=py27_1
  - testpath=0.3.1=py27_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py27h14c3975_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27hd6ce930_0
  - unittest2=1.1.0=py27_0
  - urllib3=1.22=py27ha55213b_0
  - virtualenv=16.0.0=py27_0
  - wcwidth=0.1.7=py27_0
  - webencodings=0.5.1=py27_1
  - werkzeug=0.14.1=py27_0
  - wheel=0.31.1=py27_0
  - wrapt=1.11.1=py27h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch=1.1.0=py2.7_cuda10.0.130_cudnn7.5.1_0
  - torchvision=0.3.0=py27_cu10.0.130_1
  - pip:
    - backports.functools-lru-cache==1.5
    - backports.ssl-match-hostname==3.7.0.1
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - subprocess32==3.5.4
    - tensorboardx==1.7
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python2

Spark-paket som innehåller python-moduler

Spark-paket Python-modul Version
graphframes graphframes 0.7.0-DB1-Spark 2.4
Spark-djupgående-inlärning sparkdl 1.5.0-db4-Spark 2.4
tensorframes tensorframes 0.7.0-s_2.11

R-bibliotek

R-biblioteken är identiska med r-biblioteken i Databricks Runtime 5,5.

Java-och Scala-bibliotek (Scala 2,11-kluster)

Förutom Java-och Scala-biblioteken i Databricks Runtime 5,5 innehåller Databricks Runtime 5,5 LTS för Machine Learning följande jar v7:

Grupp-ID Artefakt-ID Version
com. databricks Spark-djupgående-inlärning 1.5.0-db4-Spark 2.4
com. typesafe. Akka Akka-actor_2.11 2.3.11
ml. förbrännings. mleap mleap-databricks-runtime_2.11 0.13.0
ml. dmlc xgboost4j 0,90
ml. dmlc xgboost4j-Spark 0,90
org. graphframes graphframes_2.11 0.7.0-DB1-Spark 2.4
org. tensorflow libtensorflow 1.13.1
org. tensorflow libtensorflow_jni 1.13.1
org. tensorflow Spark-tensorflow-connector_2.11 1.13.1
org. tensorflow tensorflow 1.13.1
org. tensorframes tensorframes 0.7.0-s_2.11