Share via

Databricks Runtime 5.5 LTS para ML (sem suporte)

  • Artigo

O Databricks lançou esta imagem em julho de 2019. O suporte foi encerrado em 27 de julho de 2021. Databricks Runtime 5.5 ML suporte estendido (sem suporte) estende o suporte a 5.5 ML até dezembro de 2021. Ele usa o Ubuntu 18.04.5 LTS em vez da distribuição preterida do Ubuntu 16.04.6 LTS usada no Databricks Runtime 5.5 ML LTS original. O suporte ao Ubuntu 16.04.6 LTS foi encerrado em 1º de abril de 2021.

O Databricks Runtime 5.5 LTS para Machine Learning fornece um ambiente pronto para uso em aprendizado de máquina e ciência de dados com base no Databricks Runtime 5.5 LTS (sem suporte). O Databricks Runtime ML contém várias bibliotecas de aprendizado de máquina conhecidas, como TensorFlow, PyTorch, Keras e XGBoost. Ele também dá suporte ao treinamento de aprendizado profundo distribuído com o uso do Horovod.

Para obter mais informações, como instruções para a criação de um cluster do Databricks Runtime ML, confira IA e Machine Learning no Databricks.

Novos recursos

O Databricks Runtime 5.5 LTS para Machine Learning foi criado com base no Databricks Runtime 5.5 LTS. Para obter informações sobre as novidades do Databricks Runtime 5.5 LTS, confira as notas sobre a versão do Databricks Runtime 5.5 LTS (sem suporte).

Além das atualizações de bibliotecas, o Databricks Runtime 5.5 LTS para Machine Learning apresenta os seguintes novos recursos:

Aprimoramentos

  • Atualização das bibliotecas de aprendizado de máquina

    • TensorFlow atualizado da 1.12.0 para a 1.13.1
    • PyTorch atualizado de 0.4.1 para 1.1.0
    • scikit-learn atualizado de 0.19.1 para 0.20.3

  • Operação de nó único para HorovodRunner

    Habilitar o HorovodRunner para ser executado apenas no nó do driver. Anteriormente, para usar o HorovodRunner, você teria que executar um driver e pelo menos um nó de trabalho. Com essa alteração, agora você pode distribuir o treinamento em um único nó (ou seja, um nó de várias GPU) e, portanto, usar recursos de computação com mais eficiência.

Reprovação

Na biblioteca hyperopt, as seguintes propriedades de hyperopt.SparkTrials foram removidas:

  • SparkTrials.successful_trials_count
  • SparkTrials.failed_trials_count
  • SparkTrials.cancelled_trials_count
  • SparkTrials.total_trials_count

e substituíram as propriedades com as seguintes funções:

  • SparkTrials.count_successful_trials()
  • SparkTrials.count_failed_trials()
  • SparkTrials.count_cancelled_trials()
  • SparkTrials.count_total_trials()

Ambiente do sistema

O ambiente do sistema no Databricks Runtime 5.5 LTS para Machine Learning difere do Databricks Runtime 5.5 nestes pontos:

Bibliotecas

As seções a seguir listam as bibliotecas incluídas no Databricks Runtime 5.5 LTS for Machine Learning que diferem daquelas incluídas no Databricks Runtime 5.5.

Bibliotecas de camada superior

O Databricks Runtime 5.5 LTS for Machine Learning inclui as seguintes bibliotecas de camada superior:

Bibliotecas do Python

O Databricks Runtime 5.5 LTS para Machine Learning usa o Conda para gerenciamento de pacotes do Python. Como resultado, há grandes diferenças nas bibliotecas do Python pré-instaladas em comparação com o Databricks Runtime. As seções a seguir descrevem os ambientes do Conda para Databricks Runtime 5.5 LTS para clusters Machine Learning usando Python 2 ou 3 e máquinas habilitadas para CPU ou GPU.

Python 3 em clusters de CPU

name: null
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=2.0=cpu_0
  - _tflow_select=2.3.0=mkl
  - absl-py=0.7.1=py36_0
  - asn1crypto=0.24.0=py36_0
  - astor=0.7.1=py36_0
  - backcall=0.1.0=py36_0
  - backports=1.0=py_2
  - bcrypt=3.1.6=py36h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py36_0
  - boto=2.48.0=py36_1
  - boto3=1.7.62=py36h28b3542_1
  - botocore=1.10.62=py36h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - cffi=1.11.5=py36he75722e_1
  - chardet=3.0.4=py36_1
  - click=7.0=py36_0
  - cloudpickle=0.8.0=py36_0
  - colorama=0.3.9=py36h489cec4_0
  - configparser=3.7.3=py36_1
  - cryptography=2.2.2=py36h14c3975_0
  - cycler=0.10.0=py36h93f1223_0
  - cython=0.28.2=py36h14c3975_0
  - decorator=4.3.0=py36_0
  - docutils=0.14=py36hb0f60f5_0
  - entrypoints=0.2.3=py36_2
  - et_xmlfile=1.0.1=py36hd6bccc3_0
  - flask=1.0.2=py36_1
  - freetype=2.8=hab7d2ae_1
  - gast=0.2.2=py36_0
  - gitdb2=2.0.5=py36_0
  - gitpython=2.1.11=py36_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py36hdbcaa40_0
  - gunicorn=19.9.0=py36_0
  - h5py=2.8.0=py36h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py36_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py36h82fb2a8_1
  - intel-openmp=2018.0.0=8
  - ipython=6.4.0=py36_1
  - ipython_genutils=0.2.0=py36_0
  - itsdangerous=0.24=py36_1
  - jdcal=1.4=py36_0
  - jedi=0.12.0=py36_1
  - jinja2=2.10=py36_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py36_0
  - jupyter_client=5.2.3=py36_0
  - jupyter_core=4.4.0=py36_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py36_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=he6710b0_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - llvmlite=0.23.1=py36hdbcaa40_0
  - lxml=4.2.1=py36h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py36_0
  - markupsafe=1.0=py36h14c3975_1
  - mistune=0.8.3=py36h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py36ha843d7b_0
  - mkl_random=1.0.2=py36hd81dba3_0
  - mock=3.0.5=py36_0
  - msgpack-python=0.5.6=py36h6bb024c_1
  - nbconvert=5.3.1=py36_0
  - nbformat=4.4.0=py36h31c9010_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py36hfd86e86_0
  - numba=0.38.0=py36h637b7d7_0
  - numpy=1.16.2=py36h7e9f1db_0
  - numpy-base=1.16.2=py36hde5b4d6_0
  - olefile=0.45.1=py36_0
  - openpyxl=2.5.3=py36_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py36h637b7d7_0
  - pandocfilters=1.4.2=py36_1
  - paramiko=2.4.2=py36_0
  - parso=0.2.0=py36_0
  - pathlib2=2.3.2=py36_0
  - patsy=0.5.0=py36_0
  - pexpect=4.5.0=py36_0
  - pickleshare=0.7.4=py36_0
  - pillow=5.1.0=py36h3deb7b8_0
  - pip=10.0.1=py36_0
  - ply=3.11=py36_0
  - prompt_toolkit=1.0.15=py36h17d85b1_0
  - protobuf=3.8.0=py36he6710b0_0
  - psycopg2=2.7.5=py36hb7f436b_0
  - ptyprocess=0.5.2=py36h69acd42_0
  - py-xgboost=0.90=py36he6710b0_0
  - py-xgboost-cpu=0.90=py36_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py36_1
  - pygments=2.2.0=py36_0
  - pynacl=1.3.0=py36h7b6447c_0
  - pyopenssl=18.0.0=py36_0
  - pyparsing=2.2.0=py36_1
  - pysocks=1.6.8=py36_0
  - python=3.6.5=hc3d631a_2
  - python-dateutil=2.7.3=py36_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py36_0
  - pyyaml=5.1=py36h7b6447c_0
  - pyzmq=17.0.0=py36h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py36he2e5f8d_1
  - s3transfer=0.1.13=py36_0
  - scikit-learn=0.20.3=py36hd81dba3_0
  - scipy=1.1.0=py36h7c811a0_2
  - setuptools=39.1.0=py36_0
  - simplegeneric=0.8.1=py36_2
  - simplejson=3.16.0=py36h14c3975_0
  - singledispatch=3.4.0.3=py36_0
  - six=1.11.0=py36_1
  - smmap2=2.0.5=py36_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py36h035aef0_0
  - tabulate=0.8.3=py36_0
  - tensorboard=1.13.1=py36hf484d3e_0
  - tensorflow=1.13.1=mkl_py36h27d456a_0
  - tensorflow-base=1.13.1=mkl_py36h7ce6ba3_0
  - tensorflow-estimator=1.13.0=py_0
  - tensorflow-mkl=1.13.1=h4fcabd2_0
  - termcolor=1.1.0=py36_1
  - testpath=0.3.1=py36h8cadb63_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py36h14c3975_0
  - traitlets=4.3.2=py36_0
  - urllib3=1.22=py36hbe7ace6_0
  - virtualenv=16.0.0=py36_0
  - wcwidth=0.1.7=py36hdf4376a_0
  - webencodings=0.5.1=py36_1
  - werkzeug=0.14.1=py36_0
  - wheel=0.31.1=py36_0
  - wrapt=1.11.1=py36h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch-cpu=1.1.0=py3.6_cpu_0
  - torchvision-cpu=0.3.0=py36_cuNone_1
  - pip:
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - future==0.17.1
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - tensorboardx==1.7
    - torchvision==0.3.0
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python3

Python 3 em clusters de GPU

name: null
channels:
  - pytorch
  - Databricks
  - defaults
dependencies:
  - tensorflow=1.13.1.db1=gpu_py36h2903d8e_0
  - tensorflow-base=1.13.1.db1=gpu_py36he292aa2_0
  - tensorflow-gpu=1.13.1.db1=h0d30ee6_0
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=1.0=gpu_0
  - _tflow_select=2.1.0=gpu
  - absl-py=0.7.1=py36_0
  - asn1crypto=0.24.0=py36_0
  - astor=0.7.1=py36_0
  - backcall=0.1.0=py36_0
  - backports=1.0=py_2
  - bcrypt=3.1.6=py36h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py36_0
  - boto=2.48.0=py36_1
  - boto3=1.7.62=py36h28b3542_1
  - botocore=1.10.62=py36h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - cffi=1.11.5=py36he75722e_1
  - chardet=3.0.4=py36_1
  - click=7.0=py36_0
  - cloudpickle=0.8.0=py36_0
  - colorama=0.3.9=py36h489cec4_0
  - configparser=3.7.3=py36_1
  - cryptography=2.2.2=py36h14c3975_0
  - cudnn=7.6.0=cuda10.0_0
  - cupti=10.0.130=0
  - cycler=0.10.0=py36_0
  - cython=0.28.2=py36h14c3975_0
  - decorator=4.3.0=py36_0
  - docutils=0.14=py36_0
  - entrypoints=0.2.3=py36_2
  - et_xmlfile=1.0.1=py36hd6bccc3_0
  - flask=1.0.2=py36_1
  - freetype=2.8=hab7d2ae_1
  - gast=0.2.2=py36_0
  - gitdb2=2.0.5=py36_0
  - gitpython=2.1.11=py36_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py36hdbcaa40_0
  - gunicorn=19.9.0=py36_0
  - h5py=2.8.0=py36h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py36_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py36h82fb2a8_1
  - intel-openmp=2018.0.0=8
  - ipython=6.4.0=py36_1
  - ipython_genutils=0.2.0=py36hb52b0d5_0
  - itsdangerous=0.24=py36_1
  - jdcal=1.4=py36_0
  - jedi=0.12.0=py36_1
  - jinja2=2.10=py36_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py36_0
  - jupyter_client=5.2.3=py36_0
  - jupyter_core=4.4.0=py36_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py36_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=h688424c_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - llvmlite=0.23.1=py36hdbcaa40_0
  - lxml=4.2.1=py36h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py36_0
  - markupsafe=1.0=py36h14c3975_1
  - mistune=0.8.3=py36h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py36ha843d7b_0
  - mkl_random=1.0.2=py36hd81dba3_0
  - mock=3.0.5=py36_0
  - msgpack-python=0.5.6=py36h6bb024c_1
  - nbconvert=5.3.1=py36_0
  - nbformat=4.4.0=py36h31c9010_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py36hfd86e86_0
  - numba=0.38.0=py36h637b7d7_0
  - numpy=1.16.2=py36h7e9f1db_0
  - numpy-base=1.16.2=py36hde5b4d6_0
  - olefile=0.45.1=py36_0
  - openpyxl=2.5.3=py36_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py36h637b7d7_0
  - pandocfilters=1.4.2=py36_1
  - paramiko=2.4.2=py36_0
  - parso=0.2.0=py36_0
  - pathlib2=2.3.2=py36_0
  - patsy=0.5.0=py36_0
  - pexpect=4.5.0=py36_0
  - pickleshare=0.7.4=py36h63277f8_0
  - pillow=5.1.0=py36h3deb7b8_0
  - pip=10.0.1=py36_0
  - ply=3.11=py36_0
  - prompt_toolkit=1.0.15=py36_0
  - protobuf=3.8.0=py36he6710b0_0
  - psycopg2=2.7.5=py36hb7f436b_0
  - ptyprocess=0.5.2=py36h69acd42_0
  - py-xgboost=0.90=py36h688424c_0
  - py-xgboost-gpu=0.90=py36h28bbb66_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py36_1
  - pygments=2.2.0=py36_0
  - pynacl=1.3.0=py36h7b6447c_0
  - pyopenssl=18.0.0=py36_0
  - pyparsing=2.2.0=py36_1
  - pysocks=1.6.8=py36_0
  - python=3.6.5=hc3d631a_2
  - python-dateutil=2.7.3=py36_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py36_0
  - pyyaml=5.1=py36h7b6447c_0
  - pyzmq=17.0.0=py36h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py36he2e5f8d_1
  - s3transfer=0.1.13=py36_0
  - scikit-learn=0.20.3=py36hd81dba3_0
  - scipy=1.1.0=py36h7c811a0_2
  - setuptools=39.1.0=py36_0
  - simplegeneric=0.8.1=py36_2
  - simplejson=3.16.0=py36h14c3975_0
  - singledispatch=3.4.0.3=py36h7a266c3_0
  - six=1.11.0=py36_1
  - smmap2=2.0.5=py36_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py36h035aef0_0
  - tabulate=0.8.3=py36_0
  - tensorboard=1.13.1=py36hf484d3e_0
  - tensorflow-estimator=1.13.0=py_0
  - termcolor=1.1.0=py36_1
  - testpath=0.3.1=py36_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py36h14c3975_0
  - traitlets=4.3.2=py36h674d592_0
  - urllib3=1.22=py36hbe7ace6_0
  - virtualenv=16.0.0=py36_0
  - wcwidth=0.1.7=py36hdf4376a_0
  - webencodings=0.5.1=py36_1
  - werkzeug=0.14.1=py36_0
  - wheel=0.31.1=py36_0
  - wrapt=1.11.1=py36h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch=1.1.0=py3.6_cuda10.0.130_cudnn7.5.1_0
  - torchvision=0.3.0=py36_cu10.0.130_1
  - pip:
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - future==0.17.1
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - tensorboardx==1.7
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python3

Python 2 em clusters de CPU

name: null
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=2.0=cpu_0
  - _tflow_select=2.3.0=mkl
  - absl-py=0.7.1=py27_0
  - asn1crypto=0.24.0=py27_0
  - astor=0.7.1=py27_0
  - backports=1.0=py_2
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports.weakref=1.0.post1=py_1
  - backports_abc=0.5=py_0
  - bcrypt=3.1.6=py27h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py27_0
  - boto=2.48.0=py27_1
  - boto3=1.7.62=py27h28b3542_1
  - botocore=1.10.62=py27h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py27_0
  - cffi=1.11.5=py27he75722e_1
  - chardet=3.0.4=py27_1
  - click=7.0=py27_0
  - cloudpickle=0.8.0=py27_0
  - colorama=0.3.9=py27h5cde069_0
  - configparser=3.7.3=py27_1
  - cryptography=2.2.2=py27h14c3975_0
  - cycler=0.10.0=py27hc7354d3_0
  - cython=0.28.2=py27h14c3975_0
  - decorator=4.3.0=py27_0
  - docutils=0.14=py27_0
  - entrypoints=0.2.3=py27_2
  - enum34=1.1.6=py27_1
  - et_xmlfile=1.0.1=py27_0
  - flask=1.0.2=py27_1
  - freetype=2.8=hab7d2ae_1
  - funcsigs=1.0.2=py27_0
  - functools32=3.2.3.2=py27_1
  - future=0.17.1=py27_0
  - futures=3.2.0=py27_0
  - gast=0.2.2=py27_0
  - gitdb2=2.0.5=py27_0
  - gitpython=2.1.11=py27_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py27hdbcaa40_0
  - gunicorn=19.9.0=py27_0
  - h5py=2.8.0=py27h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py27_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py27h5722d68_1
  - intel-openmp=2018.0.0=8
  - ipaddress=1.0.22=py27_0
  - ipython=5.7.0=py27_0
  - ipython_genutils=0.2.0=py27_0
  - itsdangerous=0.24=py27_1
  - jdcal=1.4=py27_0
  - jinja2=2.10=py27_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py27h7ed5aa4_0
  - jupyter_client=5.2.3=py27_0
  - jupyter_core=4.4.0=py27_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py27_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=he6710b0_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - linecache2=1.0.0=py27_0
  - llvmlite=0.23.1=py27hdbcaa40_0
  - lxml=4.2.1=py27h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py27_0
  - markupsafe=1.0=py27h14c3975_1
  - mistune=0.8.3=py27h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py27ha843d7b_0
  - mkl_random=1.0.2=py27hd81dba3_0
  - mock=3.0.5=py27_0
  - msgpack-python=0.5.6=py27h6bb024c_1
  - nbconvert=5.3.1=py27_0
  - nbformat=4.4.0=py27hed7f2b2_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py27hfd86e86_0
  - numba=0.38.0=py27h637b7d7_0
  - numpy=1.16.2=py27h7e9f1db_0
  - numpy-base=1.16.2=py27hde5b4d6_0
  - olefile=0.45.1=py27_0
  - openpyxl=2.5.3=py27_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py27h637b7d7_0
  - pandocfilters=1.4.2=py27_1
  - paramiko=2.4.2=py27_0
  - pathlib2=2.3.2=py27_0
  - patsy=0.5.0=py27_0
  - pexpect=4.5.0=py27_0
  - pickleshare=0.7.4=py27_0
  - pillow=5.1.0=py27h3deb7b8_0
  - pip=10.0.1=py27_0
  - ply=3.11=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - protobuf=3.8.0=py27he6710b0_0
  - psycopg2=2.7.5=py27hb7f436b_0
  - ptyprocess=0.5.2=py27h4ccb14c_0
  - py-xgboost=0.90=py27he6710b0_0
  - py-xgboost-cpu=0.90=py27_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py27_1
  - pygments=2.2.0=py27_0
  - pynacl=1.3.0=py27h7b6447c_0
  - pyopenssl=18.0.0=py27_0
  - pyparsing=2.2.0=py27_1
  - pysocks=1.6.8=py27_0
  - python=2.7.15=h1571d57_0
  - python-dateutil=2.7.3=py27_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py27_0
  - pyyaml=5.1=py27h7b6447c_0
  - pyzmq=17.0.0=py27h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py27hc5b0589_1
  - s3transfer=0.1.13=py27_0
  - scandir=1.7=py27h14c3975_0
  - scikit-learn=0.20.3=py27hd81dba3_0
  - scipy=1.1.0=py27h7c811a0_2
  - setuptools=39.1.0=py27_0
  - simplegeneric=0.8.1=py27_2
  - simplejson=3.16.0=py27h14c3975_0
  - singledispatch=3.4.0.3=py27_0
  - six=1.11.0=py27_1
  - smmap2=2.0.5=py27_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py27h035aef0_0
  - tabulate=0.8.3=py27_0
  - tensorboard=1.13.1=py27hf484d3e_0
  - tensorflow=1.13.1=mkl_py27h74ee40f_0
  - tensorflow-base=1.13.1=mkl_py27h7ce6ba3_0
  - tensorflow-estimator=1.13.0=py_0
  - tensorflow-mkl=1.13.1=h4fcabd2_0
  - termcolor=1.1.0=py27_1
  - testpath=0.3.1=py27hc38d2c4_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py27h14c3975_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27_0
  - unittest2=1.1.0=py27_0
  - urllib3=1.22=py27ha55213b_0
  - virtualenv=16.0.0=py27_0
  - wcwidth=0.1.7=py27h9e3e1ab_0
  - webencodings=0.5.1=py27_1
  - werkzeug=0.14.1=py27_0
  - wheel=0.31.1=py27_0
  - wrapt=1.11.1=py27h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch-cpu=1.1.0=py2.7_cpu_0
  - torchvision-cpu=0.3.0=py27_cuNone_1
  - pip:
    - backports.functools-lru-cache==1.5
    - backports.ssl-match-hostname==3.7.0.1
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - subprocess32==3.5.4
    - tensorboardx==1.7
    - torchvision==0.3.0
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python2

Python 2 em clusters de GPU

name: null
channels:
  - Databricks
  - pytorch
  - defaults
dependencies:
  - tensorflow=1.13.1.db1=gpu_py27h8e347d7_0
  - tensorflow-base=1.13.1.db1=gpu_py27he292aa2_0
  - tensorflow-gpu=1.13.1.db1=h0d30ee6_0
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=1.0=gpu_0
  - _tflow_select=2.1.0=gpu
  - absl-py=0.7.1=py27_0
  - asn1crypto=0.24.0=py27_0
  - astor=0.7.1=py27_0
  - backports=1.0=py_2
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports.weakref=1.0.post1=py_1
  - backports_abc=0.5=py_0
  - bcrypt=3.1.6=py27h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py27_0
  - boto=2.48.0=py27_1
  - boto3=1.7.62=py27h28b3542_1
  - botocore=1.10.62=py27h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py27_0
  - cffi=1.11.5=py27he75722e_1
  - chardet=3.0.4=py27_1
  - click=7.0=py27_0
  - cloudpickle=0.8.0=py27_0
  - colorama=0.3.9=py27_0
  - configparser=3.7.3=py27_1
  - cryptography=2.2.2=py27h14c3975_0
  - cudnn=7.6.0=cuda10.0_0
  - cupti=10.0.130=0
  - cycler=0.10.0=py27_0
  - cython=0.28.2=py27h14c3975_0
  - decorator=4.3.0=py27_0
  - docutils=0.14=py27hae222c1_0
  - entrypoints=0.2.3=py27_2
  - enum34=1.1.6=py27_1
  - et_xmlfile=1.0.1=py27h75840f5_0
  - flask=1.0.2=py27_1
  - freetype=2.8=hab7d2ae_1
  - funcsigs=1.0.2=py27_0
  - functools32=3.2.3.2=py27_1
  - future=0.17.1=py27_0
  - futures=3.2.0=py27_0
  - gast=0.2.2=py27_0
  - gitdb2=2.0.5=py27_0
  - gitpython=2.1.11=py27_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py27hdbcaa40_0
  - gunicorn=19.9.0=py27_0
  - h5py=2.8.0=py27h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py27_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py27h5722d68_1
  - intel-openmp=2018.0.0=8
  - ipaddress=1.0.22=py27_0
  - ipython=5.7.0=py27_0
  - ipython_genutils=0.2.0=py27h89fb69b_0
  - itsdangerous=0.24=py27_1
  - jdcal=1.4=py27_0
  - jinja2=2.10=py27_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py27h7ed5aa4_0
  - jupyter_client=5.2.3=py27_0
  - jupyter_core=4.4.0=py27_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py27_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=h688424c_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - linecache2=1.0.0=py27_0
  - llvmlite=0.23.1=py27hdbcaa40_0
  - lxml=4.2.1=py27h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py27_0
  - markupsafe=1.0=py27h14c3975_1
  - mistune=0.8.3=py27h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py27ha843d7b_0
  - mkl_random=1.0.2=py27hd81dba3_0
  - mock=3.0.5=py27_0
  - msgpack-python=0.5.6=py27h6bb024c_1
  - nbconvert=5.3.1=py27_0
  - nbformat=4.4.0=py27hed7f2b2_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py27hfd86e86_0
  - numba=0.38.0=py27h637b7d7_0
  - numpy=1.16.2=py27h7e9f1db_0
  - numpy-base=1.16.2=py27hde5b4d6_0
  - olefile=0.45.1=py27_0
  - openpyxl=2.5.3=py27_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py27h637b7d7_0
  - pandocfilters=1.4.2=py27_1
  - paramiko=2.4.2=py27_0
  - pathlib2=2.3.2=py27_0
  - patsy=0.5.0=py27_0
  - pexpect=4.5.0=py27_0
  - pickleshare=0.7.4=py27h09770e1_0
  - pillow=5.1.0=py27h3deb7b8_0
  - pip=10.0.1=py27_0
  - ply=3.11=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - protobuf=3.8.0=py27he6710b0_0
  - psycopg2=2.7.5=py27hb7f436b_0
  - ptyprocess=0.5.2=py27h4ccb14c_0
  - py-xgboost=0.90=py27h688424c_0
  - py-xgboost-gpu=0.90=py27h28bbb66_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py27_1
  - pygments=2.2.0=py27_0
  - pynacl=1.3.0=py27h7b6447c_0
  - pyopenssl=18.0.0=py27_0
  - pyparsing=2.2.0=py27_1
  - pysocks=1.6.8=py27_0
  - python=2.7.15=h1571d57_0
  - python-dateutil=2.7.3=py27_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py27_0
  - pyyaml=5.1=py27h7b6447c_0
  - pyzmq=17.0.0=py27h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py27hc5b0589_1
  - s3transfer=0.1.13=py27_0
  - scandir=1.7=py27h14c3975_0
  - scikit-learn=0.20.3=py27hd81dba3_0
  - scipy=1.1.0=py27h7c811a0_2
  - setuptools=39.1.0=py27_0
  - simplegeneric=0.8.1=py27_2
  - simplejson=3.16.0=py27h14c3975_0
  - singledispatch=3.4.0.3=py27h9bcb476_0
  - six=1.11.0=py27_1
  - smmap2=2.0.5=py27_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py27h035aef0_0
  - tabulate=0.8.3=py27_0
  - tensorboard=1.13.1=py27hf484d3e_0
  - tensorflow-estimator=1.13.0=py_0
  - termcolor=1.1.0=py27_1
  - testpath=0.3.1=py27_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py27h14c3975_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27hd6ce930_0
  - unittest2=1.1.0=py27_0
  - urllib3=1.22=py27ha55213b_0
  - virtualenv=16.0.0=py27_0
  - wcwidth=0.1.7=py27_0
  - webencodings=0.5.1=py27_1
  - werkzeug=0.14.1=py27_0
  - wheel=0.31.1=py27_0
  - wrapt=1.11.1=py27h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch=1.1.0=py2.7_cuda10.0.130_cudnn7.5.1_0
  - torchvision=0.3.0=py27_cu10.0.130_1
  - pip:
    - backports.functools-lru-cache==1.5
    - backports.ssl-match-hostname==3.7.0.1
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - subprocess32==3.5.4
    - tensorboardx==1.7
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python2

Pacotes do Spark que contêm módulos do Python

Pacote do Spark Módulo do Python Versão
graphframes graphframes 0.7.0-db1-spark2.4
spark-deep-learning sparkdl 1.5.0-db4-spark2.4
tensorframes tensorframes 0.7.0-s_2.11

Bibliotecas do R

As bibliotecas do R são idênticas às Bibliotecas do R no Databricks Runtime 5.5.

Bibliotecas do Java e do Scala (cluster do Scala 2.11)

Além das bibliotecas do Java e do Scala no Databricks Runtime 5.5, o Databricks Runtime 5.5 LTS para Machine Learning contém os seguintes JARs:

ID do Grupo Artifact ID Versão
com.databricks spark-deep-learning 1.5.0-db4-spark2.4
com.typesafe.akka akka-actor_2.11 2.3.11
ml.combust.mleap mleap-databricks-runtime_2.11 0.13.0
ml.dmlc xgboost4j 0,90
ml.dmlc xgboost4j-spark 0,90
org.graphframes graphframes_2.11 0.7.0-db1-spark2.4
org.tensorflow libtensorflow 1.13.1
org.tensorflow libtensorflow_jni 1.13.1
org.tensorflow spark-tensorflow-connector_2.11 1.13.1
org.tensorflow tensorflow 1.13.1
org.tensorframes tensorframes 0.7.0-s_2.11