Databricks Runtime 9.1 LTS

  • Article

The following release notes provide information about Databricks Runtime 9.1 LTS and Databricks Runtime 9.1 LTS Photon, powered by Apache Spark 3.1.2. Databricks released these images in September 2021. Photon is in Public Preview.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

New features and improvements

Auto Loader schema hints now work with array and map types

Array and map types are supported in Override schema inference with schema hints for Auto Loader.

Examples of schema hints for arrays include:

  1. arr Array<TYPE> changes the array type.
  2. arr.element TYPE changes the array type (by using the element keyword).
  3. arr.element.x TYPE changes a nested field type in an array of structures.

The first two examples are hints to change the type of array arr to TYPE, but they use different syntax. In the third example, arr is an array of structures with a field x. This example shows how to change the type of x to TYPE.

Examples of schema hints for maps include:

  1. m Map<KEY-TYPE, VALUE-TYPE> changes map key and value types.
  2. m.key TYPE changes the type of map keys.
  3. m.value TYPE changes the type of map values.
  4. m.key.x TYPE changes the field type in a map key.
  5. m.value.x TYPE changes the field type of a map value.

The first example changes both the key and value types of map m to KEY_TYPE and VALUE_TYPE respectively. The second and third examples can be used if only the key type or only the value type needs to be changed. In the fourth and fifth examples, m is a map with key and value of structure types with a field x. This example shows how to change the type of x to TYPE.

Avro file support for merge schema

The Avro file format now supports the mergeSchema option when reading files. Setting mergeSchema to true when reading Avro files will infer a schema from a set of Avro files rather than from a single file. This improves usability by inferring a schema that may be able to read all files even if their individual schemas differ. See Configuration.

Auto Loader incremental listing support (Public Preview)

In the case of lexicographically generated files, What is Auto Loader? now leverages lexical file ordering and existing optimized APIs to make the directory listing more efficient by listing from previously-ingested files rather than by listing the entire directory. Auto Loader automatically detects whether a given directory is suitable for incremental listing by default. To control this behavior explicitly, set the new cloudFiles.useIncrementalListing option to on (true), off (false), or automatic (auto). If you set this behavior to true, you can also set the cloudFiles.backfillInterval option to schedule regular backfills over your data, to make sure all of your data is completely ingested.

Delta now supports arbitrary replaceWhere

In combination with overwrite mode, the replaceWhere option can be used to simultaneously overwrite data that matches a predicate defined in the option. Previously, replaceWhere supported a predicate only over partition columns, but it can now be an arbitrary expression. See Write to a table.

Auto Loader for Google Cloud now supports file notifications (Public Preview)

Auto Loader now supports file notification mode on Google Cloud. Set .option("cloudFiles.useNotifications", "true") to allow Auto Loader to automatically set up Google Cloud Pub/Sub resources for you. With file notification mode, new files are detected and ingested as they arrive without listing the input directory. See What is Auto Loader file notification mode?.

CREATE FUNCTION now supports creating table functions

In addition to creating a scalar function that returns a scalar value, you can now create a table function that returns a set of rows. See CREATE FUNCTION (SQL and Python).

Kafka Streaming Source now reports estimatedTotalBytesBehindLatest metric

The Kafka streaming source now reports an estimate of how many bytes the consumer is behind the latest available byte after every batch. You can use this metric to track stream progress. See Retrieve Kafka metrics.

Example metric output:

StreamingQueryProgress {
  "batchId": 0,
  .....
  "sources": [ {
    "description" : "KafkaV2[Subscribe[topic-0]]",
    "metrics":{
      "avgOffsetsBehindLatest" : "1.0",
      "estimatedTotalBytesBehindLatest" : "80.0", // new
      "maxOffsetsBehindLatest" : "1",
      "minOffsetsBehindLatest" : "1"
    } ],
  ....
 }

For structs inside of arrays, Delta MERGE INTO now resolves struct fields by name and evolves struct schemas

Delta MERGE INTO now supports resolution of struct fields by name and automatic schema evolution for arrays of structs. When automatic schema evolution is enabled by setting spark.databricks.delta.schema.autoMerge.enabled to true, UPDATE and INSERT clauses will resolve struct fields inside of an array by name, casting to the corresponding data type that is defined in the target array and filling additional or missing fields in the source or target with null values. When automatic schema evolution is disabled, UPDATE and INSERT clauses will resolve struct fields inside of an array by name but will not be able to evolve the additional fields. See Automatic schema evolution for arrays of structs.

Bug fixes

  • Fixed a memory leak in the Amazon S3 connector that could happen in long running jobs or services, which was caused by JVM DeleteOnExit functionality.

Library upgrades

  • Upgraded Python libraries:
    • plotly from 4.14.3 to 5.1.0
  • Upgraded R libraries:
    • base from 4.1.0 to 4.1.1
    • bslib from 0.2.5.1 to 0.3.0
    • cachem from 1.0.5 to 1.0.6
    • compiler from 4.1.0 to 4.1.1
    • datasets from 4.1.0 to 4.1.1
    • future from 1.21.0 to 1.22.1
    • gert from 1.3.1 to 1.3.2
    • graphics from 4.1.0 to 4.1.1
    • grDevices from 4.1.0 to 4.1.1
    • grid from 4.1.0 to 4.1.1
  • Upgraded Java libraries:
    • org.eclipse.jetty from 9.4.36.v20210114 to 9.4.42.v20210604

Apache Spark

Databricks Runtime 9.1 LTS includes Apache Spark 3.1.2. This release includes all Spark fixes and improvements included in Databricks Runtime 9.0 (unsupported), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-36674][SQL][CHERRY-PICK] Support ILIKE - case insensitive LIKE
  • [SPARK-36353][SQL][3.1] RemoveNoopOperators should keep output schema
  • [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
  • [SPARK-36398][SQL] Redact sensitive information in Spark Thrift Server log
  • [SPARK-36498][SQL] Reorder inner fields of the input query in byName V2 write
  • [SPARK-36614][CORE][UI] Correct executor loss reason caused by decommission in UI
  • [SPARK-36012][SQL] Add null flag in SHOW CREATE TABLE
  • [SPARK-36509][CORE] Fix the issue that executors are never re-scheduled if the worker stops with standalone cluster
  • [SPARK-36603][CORE] Use WeakReference not SoftReference in LevelDB
  • [SPARK-36564][CORE] Fix NullPointerException in LiveRDDDistribution.toApi
  • [SPARK-36086][SQL][3.1] CollapseProject project replace alias should use origin column name
  • [SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
  • [SPARK-36400][SPARK-36398][SQL][WEBUI] Make ThriftServer recognize spark.sql.redaction.string.regex
  • [SPARK-34054][CORE] BlockManagerDecommissioner code cleanup
  • [SPARK-36500][CORE] Fix temp_shuffle file leaking when a task is interrupted
  • [SPARK-36489][SQL] Aggregate functions over no grouping keys, on tables with a single bucket, return multiple rows
  • [SPARK-36464][CORE] Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data
  • [SPARK-36339][SQL][3.0] References to grouping that not part of aggregation should be replaced
  • [SPARK-36354][CORE] EventLogFileReader should skip rolling event log directories with no logs
  • [SPARK-36242][CORE][3.1] Ensure spill file closed before set success = true in ExternalSorter.spillMemoryIteratorToDisk method
  • [SPARK-36211][PYTHON] Correct typing of udf return value
  • [SPARK-34222][SQL] Enhance boolean simplification rule
  • [SPARK-35027][CORE] Close the inputStream in FileAppender when writin…
  • [SPARK-36269][SQL] Fix only set data columns to Hive column names config
  • [SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec
  • [SPARK-36210][SQL] Preserve column insertion order in Dataset.withColumns
  • [SPARK-36079][SQL] Null-based filter estimate should always be in the range [0, 1]
  • [SPARK-28266][SQL] convertToLogicalRelation should not interpret path property when reading Hive tables

Maintenance updates

See Databricks Runtime 9.1 LTS maintenance updates.

System environment

  • Operating System: Ubuntu 20.04.4 LTS
  • Java: Zulu 8.56.0.21-CA-linux64
  • Scala: 2.12.10
  • Python: 3.8.10
  • R: 4.1.1
  • Delta Lake: 1.0.0

Installed Python libraries

Library Version Library Version Library Version
Antergos Linux 2015.10 (ISO-Rolling) appdirs 1.4.4 backcall 0.2.0
boto3 1.16.7 botocore 1.19.7 certifi 2020.12.5
chardet 4.0.0 cycler 0.10.0 Cython 0.29.23
dbus-python 1.2.16 decorator 5.0.6 distlib 0.3.2
distro-info 0.23ubuntu1 facets-overview 1.0.0 filelock 3.0.12
idna 2.10 ipykernel 5.3.4 ipython 7.22.0
ipython-genutils 0.2.0 jedi 0.17.2 jmespath 0.10.0
joblib 1.0.1 jupyter-client 6.1.12 jupyter-core 4.7.1
kiwisolver 1.3.1 koalas 1.8.1 matplotlib 3.4.2
numpy 1.19.2 pandas 1.2.4 parso 0.7.0
patsy 0.5.1 pexpect 4.8.0 pickleshare 0.7.5
Pillow 8.2.0 pip 21.0.1 plotly 5.1.0
prompt-toolkit 3.0.17 protobuf 3.17.2 psycopg2 2.8.5
ptyprocess 0.7.0 pyarrow 4.0.0 Pygments 2.8.1
PyGObject 3.36.0 pyparsing 2.4.7 python-apt 2.0.0+ubuntu0.20.4.5
python-dateutil 2.8.1 pytz 2020.5 pyzmq 20.0.0
requests 2.25.1 requests-unixsocket 0.2.0 s3transfer 0.3.7
scikit-learn 0.24.1 scipy 1.6.2 seaborn 0.11.1
setuptools 52.0.0 six 1.15.0 ssh-import-id 5.10
statsmodels 0.12.2 tenacity 8.0.1 threadpoolctl 2.1.0
tornado 6.1 traitlets 5.0.5 unattended-upgrades 0.1
urllib3 1.25.11 virtualenv 20.4.1 wcwidth 0.2.5
wheel 0.36.2

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2021-09-08.

Library Version Library Version Library Version
askpass 1.1 assertthat 0.2.1 backports 1.2.1
base 4.1.1 base64enc 0.1-3 bit 4.0.4
bit64 4.0.5 blob 1.2.2 boot 1.3-28
brew 1.0-6 brio 1.1.2 broom 0.7.9
bslib 0.3.0 cachem 1.0.6 callr 3.7.0
caret 6.0-88 cellranger 1.1.0 chron 2.3-56
class 7.3-19 cli 3.0.1 clipr 0.7.1
cluster 2.1.2 codetools 0.2-18 colorspace 2.0-2
commonmark 1.7 compiler 4.1.1 config 0.3.1
cpp11 0.3.1 crayon 1.4.1 credentials 1.3.1
curl 4.3.2 data.table 1.14.0 datasets 4.1.1
DBI 1.1.1 dbplyr 2.1.1 desc 1.3.0
devtools 2.4.2 diffobj 0.3.4 digest 0.6.27
dplyr 1.0.7 dtplyr 1.1.0 ellipsis 0.3.2
evaluate 0.14 fansi 0.5.0 farver 2.1.0
fastmap 1.1.0 forcats 0.5.1 foreach 1.5.1
foreign 0.8-81 forge 0.2.0 fs 1.5.0
future 1.22.1 future.apply 1.8.1 gargle 1.2.0
generics 0.1.0 gert 1.3.2 ggplot2 3.3.5
gh 1.3.0 gitcreds 0.1.1 glmnet 4.1-2
globals 0.14.0 glue 1.4.2 googledrive 2.0.0
googlesheets4 1.0.0 gower 0.2.2 graphics 4.1.1
grDevices 4.1.1 grid 4.1.1 gridExtra 2.3
gsubfn 0.7 gtable 0.3.0 haven 2.4.3
highr 0.9 hms 1.1.0 htmltools 0.5.2
htmlwidgets 1.5.3 httpuv 1.6.2 httr 1.4.2
hwriter 1.3.2 hwriterPlus 1.0-3 ids 1.0.1
ini 0.3.1 ipred 0.9-11 isoband 0.2.5
iterators 1.0.13 jquerylib 0.1.4 jsonlite 1.7.2
KernSmooth 2.23-20 knitr 1.33 labeling 0.4.2
later 1.3.0 lattice 0.20-44 lava 1.6.10
lifecycle 1.0.0 listenv 0.8.0 lubridate 1.7.10
magrittr 2.0.1 markdown 1.1 MASS 7.3-54
Matrix 1.3-4 memoise 2.0.0 methods 4.1.1
mgcv 1.8-36 mime 0.11 ModelMetrics 1.2.2.2
modelr 0.1.8 munsell 0.5.0 nlme 3.1-152
nnet 7.3-16 numDeriv 2016.8-1.1 openssl 1.4.5
parallel 4.1.1 parallelly 1.27.0 pillar 1.6.2
pkgbuild 1.2.0 pkgconfig 2.0.3 pkgload 1.2.1
plogr 0.2.0 plyr 1.8.6 praise 1.0.0
prettyunits 1.1.1 pROC 1.18.0 processx 3.5.2
prodlim 2019.11.13 progress 1.2.2 progressr 0.8.0
promises 1.2.0.1 proto 1.0.0 ps 1.6.0
purrr 0.3.4 r2d3 0.2.5 R6 2.5.1
randomForest 4.6-14 rappdirs 0.3.3 rcmdcheck 1.3.3
RColorBrewer 1.1-2 Rcpp 1.0.7 readr 2.0.1
readxl 1.3.1 recipes 0.1.16 rematch 1.0.1
rematch2 2.1.2 remotes 2.4.0 reprex 2.0.1
reshape2 1.4.4 rlang 0.4.11 rmarkdown 2.10
RODBC 1.3-18 roxygen2 7.1.1 rpart 4.1-15
rprojroot 2.0.2 Rserve 1.8-8 RSQLite 2.2.8
rstudioapi 0.13 rversions 2.1.1 rvest 1.0.1
sass 0.4.0 scales 1.1.1 selectr 0.4-2
sessioninfo 1.1.1 shape 1.4.6 shiny 1.6.0
sourcetools 0.1.7 sparklyr 1.7.1 SparkR 3.1.1
spatial 7.3-11 splines 4.1.1 sqldf 0.4-11
SQUAREM 2021.1 stats 4.1.1 stats4 4.1.1
stringi 1.7.4 stringr 1.4.0 survival 3.2-13
sys 3.4 tcltk 4.1.1 TeachingDemos 2.10
testthat 3.0.4 tibble 3.1.4 tidyr 1.1.3
tidyselect 1.1.1 tidyverse 1.3.1 timeDate 3043.102
tinytex 0.33 tools 4.1.1 tzdb 0.1.2
usethis 2.0.1 utf8 1.2.2 utils 4.1.1
uuid 0.1-4 vctrs 0.3.8 viridisLite 0.4.0
vroom 1.5.4 waldo 0.3.0 whisker 0.4
withr 2.4.2 xfun 0.25 xml2 1.3.2
xopen 1.0.0 xtable 1.8-4 yaml 2.2.1
zip 2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.11.655
com.amazonaws aws-java-sdk-cloudformation 1.11.655
com.amazonaws aws-java-sdk-cloudfront 1.11.655
com.amazonaws aws-java-sdk-cloudhsm 1.11.655
com.amazonaws aws-java-sdk-cloudsearch 1.11.655
com.amazonaws aws-java-sdk-cloudtrail 1.11.655
com.amazonaws aws-java-sdk-cloudwatch 1.11.655
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.11.655
com.amazonaws aws-java-sdk-codedeploy 1.11.655
com.amazonaws aws-java-sdk-cognitoidentity 1.11.655
com.amazonaws aws-java-sdk-cognitosync 1.11.655
com.amazonaws aws-java-sdk-config 1.11.655
com.amazonaws aws-java-sdk-core 1.11.655
com.amazonaws aws-java-sdk-datapipeline 1.11.655
com.amazonaws aws-java-sdk-directconnect 1.11.655
com.amazonaws aws-java-sdk-directory 1.11.655
com.amazonaws aws-java-sdk-dynamodb 1.11.655
com.amazonaws aws-java-sdk-ec2 1.11.655
com.amazonaws aws-java-sdk-ecs 1.11.655
com.amazonaws aws-java-sdk-efs 1.11.655
com.amazonaws aws-java-sdk-elasticache 1.11.655
com.amazonaws aws-java-sdk-elasticbeanstalk 1.11.655
com.amazonaws aws-java-sdk-elasticloadbalancing 1.11.655
com.amazonaws aws-java-sdk-elastictranscoder 1.11.655
com.amazonaws aws-java-sdk-emr 1.11.655
com.amazonaws aws-java-sdk-glacier 1.11.655
com.amazonaws aws-java-sdk-glue 1.11.655
com.amazonaws aws-java-sdk-iam 1.11.655
com.amazonaws aws-java-sdk-importexport 1.11.655
com.amazonaws aws-java-sdk-kinesis 1.11.655
com.amazonaws aws-java-sdk-kms 1.11.655
com.amazonaws aws-java-sdk-lambda 1.11.655
com.amazonaws aws-java-sdk-logs 1.11.655
com.amazonaws aws-java-sdk-machinelearning 1.11.655
com.amazonaws aws-java-sdk-marketplacecommerceanalytics 1.11.655
com.amazonaws aws-java-sdk-marketplacemeteringservice 1.11.655
com.amazonaws aws-java-sdk-opsworks 1.11.655
com.amazonaws aws-java-sdk-rds 1.11.655
com.amazonaws aws-java-sdk-redshift 1.11.655
com.amazonaws aws-java-sdk-route53 1.11.655
com.amazonaws aws-java-sdk-s3 1.11.655
com.amazonaws aws-java-sdk-ses 1.11.655
com.amazonaws aws-java-sdk-simpledb 1.11.655
com.amazonaws aws-java-sdk-simpleworkflow 1.11.655
com.amazonaws aws-java-sdk-sns 1.11.655
com.amazonaws aws-java-sdk-sqs 1.11.655
com.amazonaws aws-java-sdk-ssm 1.11.655
com.amazonaws aws-java-sdk-storagegateway 1.11.655
com.amazonaws aws-java-sdk-sts 1.11.655
com.amazonaws aws-java-sdk-support 1.11.655
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.11.655
com.amazonaws jmespath-java 1.11.655
com.chuusai shapeless_2.12 2.3.3
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.10.0
com.fasterxml.jackson.core jackson-core 2.10.0
com.fasterxml.jackson.core jackson-databind 2.10.0
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.10.0
com.fasterxml.jackson.datatype jackson-datatype-joda 2.10.0
com.fasterxml.jackson.module jackson-module-paranamer 2.10.0
com.fasterxml.jackson.module jackson-module-scala_2.12 2.10.0
com.github.ben-manes.caffeine caffeine 2.3.4
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.joshelser dropwizard-metrics-hadoop-metrics2-reporter 0.1.2
com.github.luben zstd-jni 1.4.8-1
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.2.4
com.google.crypto.tink tink 1.6.0
com.google.flatbuffers flatbuffers-java 1.9.0
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.h2database h2 1.4.195
com.helger profiler 1.1.1
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.microsoft.sqlserver mssql-jdbc 9.2.1.jre8
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.9.5
com.twitter chill_2.12 0.9.5
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 3.1.0
commons-beanutils commons-beanutils 1.9.4
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-fileupload commons-fileupload 1.3.3
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 3.1
commons-pool commons-pool 1.5.4
hive-2.3__hadoop-2.7 jets3t-0.7 liball_deps_2.12
hive-2.3__hadoop-2.7 zookeeper-3.4 liball_deps_2.12
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.10
io.delta delta-sharing-spark_2.12 0.1.0
io.dropwizard.metrics metrics-core 4.1.1
io.dropwizard.metrics metrics-graphite 4.1.1
io.dropwizard.metrics metrics-healthchecks 4.1.1
io.dropwizard.metrics metrics-jetty9 4.1.1
io.dropwizard.metrics metrics-jmx 4.1.1
io.dropwizard.metrics metrics-json 4.1.1
io.dropwizard.metrics metrics-jvm 4.1.1
io.dropwizard.metrics metrics-servlets 4.1.1
io.netty netty-all 4.1.51.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.10.5
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
maven-trees hive-2.3__hadoop-2.7 liball_deps_2.12
net.java.dev.jna jna 5.8.0
net.razorvine pyrolite 4.30
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.snowflake snowflake-jdbc 3.13.3
net.snowflake spark-snowflake_2.12 2.9.0-spark_3.1
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.8-1
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.arrow arrow-format 2.0.0
org.apache.arrow arrow-memory-core 2.0.0
org.apache.arrow arrow-memory-netty 2.0.0
org.apache.arrow arrow-vector 2.0.0
org.apache.avro avro 1.8.2
org.apache.avro avro-ipc 1.8.2
org.apache.avro avro-mapred-hadoop2 1.8.2
org.apache.commons commons-compress 1.20
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.10
org.apache.commons commons-math3 3.4.1
org.apache.commons commons-text 1.6
org.apache.curator curator-client 2.7.1
org.apache.curator curator-framework 2.7.1
org.apache.curator curator-recipes 2.7.1
org.apache.derby derby 10.12.1.1
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.4
org.apache.hadoop hadoop-auth 2.7.4
org.apache.hadoop hadoop-client 2.7.4
org.apache.hadoop hadoop-common 2.7.4
org.apache.hadoop hadoop-hdfs 2.7.4
org.apache.hadoop hadoop-mapreduce-client-app 2.7.4
org.apache.hadoop hadoop-mapreduce-client-common 2.7.4
org.apache.hadoop hadoop-mapreduce-client-core 2.7.4
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.4
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.4
org.apache.hadoop hadoop-yarn-api 2.7.4
org.apache.hadoop hadoop-yarn-client 2.7.4
org.apache.hadoop hadoop-yarn-common 2.7.4
org.apache.hadoop hadoop-yarn-server-common 2.7.4
org.apache.hive hive-beeline 2.3.7
org.apache.hive hive-cli 2.3.7
org.apache.hive hive-jdbc 2.3.7
org.apache.hive hive-llap-client 2.3.7
org.apache.hive hive-llap-common 2.3.7
org.apache.hive hive-serde 2.3.7
org.apache.hive hive-shims 2.3.7
org.apache.hive hive-storage-api 2.7.2
org.apache.hive.shims hive-shims-0.23 2.3.7
org.apache.hive.shims hive-shims-common 2.3.7
org.apache.hive.shims hive-shims-scheduler 2.3.7
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.6
org.apache.httpcomponents httpcore 4.4.12
org.apache.ivy ivy 2.4.0
org.apache.mesos mesos-shaded-protobuf 1.4.0
org.apache.orc orc-core 1.5.12
org.apache.orc orc-mapreduce 1.5.12
org.apache.orc orc-shims 1.5.12
org.apache.parquet parquet-column 1.10.1-databricks9
org.apache.parquet parquet-common 1.10.1-databricks9
org.apache.parquet parquet-encoding 1.10.1-databricks9
org.apache.parquet parquet-format 2.4.0
org.apache.parquet parquet-hadoop 1.10.1-databricks9
org.apache.parquet parquet-jackson 1.10.1-databricks9
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.xbean xbean-asm7-shaded 4.15
org.apache.yetus audience-annotations 0.5.0
org.apache.zookeeper zookeeper 3.4.14
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.jetty jetty-client 9.4.42.v20210604
org.eclipse.jetty jetty-continuation 9.4.42.v20210604
org.eclipse.jetty jetty-http 9.4.42.v20210604
org.eclipse.jetty jetty-io 9.4.42.v20210604
org.eclipse.jetty jetty-jndi 9.4.42.v20210604
org.eclipse.jetty jetty-plus 9.4.42.v20210604
org.eclipse.jetty jetty-proxy 9.4.42.v20210604
org.eclipse.jetty jetty-security 9.4.42.v20210604
org.eclipse.jetty jetty-server 9.4.42.v20210604
org.eclipse.jetty jetty-servlet 9.4.42.v20210604
org.eclipse.jetty jetty-servlets 9.4.42.v20210604
org.eclipse.jetty jetty-util 9.4.42.v20210604
org.eclipse.jetty jetty-util-ajax 9.4.42.v20210604
org.eclipse.jetty jetty-webapp 9.4.42.v20210604
org.eclipse.jetty jetty-xml 9.4.42.v20210604
org.eclipse.jetty.websocket websocket-api 9.4.42.v20210604
org.eclipse.jetty.websocket websocket-client 9.4.42.v20210604
org.eclipse.jetty.websocket websocket-common 9.4.42.v20210604
org.eclipse.jetty.websocket websocket-server 9.4.42.v20210604
org.eclipse.jetty.websocket websocket-servlet 9.4.42.v20210604
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.30
org.glassfish.jersey.containers jersey-container-servlet-core 2.30
org.glassfish.jersey.core jersey-client 2.30
org.glassfish.jersey.core jersey-common 2.30
org.glassfish.jersey.core jersey-server 2.30
org.glassfish.jersey.inject jersey-hk2 2.30
org.glassfish.jersey.media jersey-media-jaxb 2.30
org.hibernate.validator hibernate-validator 6.1.0.Final
org.javassist javassist 3.25.0-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M5
org.json4s json4s-core_2.12 3.7.0-M5
org.json4s json4s-jackson_2.12 3.7.0-M5
org.json4s json4s-scalap_2.12 3.7.0-M5
org.lz4 lz4-java 1.7.1
org.mariadb.jdbc mariadb-java-client 2.2.5
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.1.4
org.roaringbitmap RoaringBitmap 0.9.14
org.roaringbitmap shims 0.9.14
org.rocksdb rocksdbjni 6.20.3
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.10
org.scala-lang scala-library_2.12 2.12.10
org.scala-lang scala-reflect_2.12 2.12.10
org.scala-lang.modules scala-collection-compat_2.12 2.1.1
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.0.8
org.scalanlp breeze-macros_2.12 1.0
org.scalanlp breeze_2.12 1.0
org.scalatest scalatest_2.12 3.0.8
org.slf4j jcl-over-slf4j 1.7.30
org.slf4j jul-to-slf4j 1.7.30
org.slf4j slf4j-api 1.7.30
org.slf4j slf4j-log4j12 1.7.30
org.spark-project.spark unused 1.0.0
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.threeten threeten-extra 1.5.0
org.tukaani xz 1.5
org.typelevel algebra_2.12 2.0.0-M2
org.typelevel cats-kernel_2.12 2.0.0-M4
org.typelevel machinist_2.12 0.6.8
org.typelevel macro-compat_2.12 1.1.1
org.typelevel spire-macros_2.12 0.17.0-M1
org.typelevel spire-platform_2.12 0.17.0-M1
org.typelevel spire-util_2.12 0.17.0-M1
org.typelevel spire_2.12 0.17.0-M1
org.wildfly.openssl wildfly-openssl 1.0.7.Final
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.8.2
org.yaml snakeyaml 1.24
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1
xmlenc xmlenc 0.52