Release notes for Azure HDInsight

• 01/02/2019
• 49 minutes to read
• Contributors
  • 
  • 
  • 
  • 
  • 
  • all

This article provides information about the most recent Azure HDInsight release updates. For information on earlier releases, see HDInsight Release Notes Archive.

Summary

Azure HDInsight is one of the most popular services among enterprise customers for open-source Apache Hadoop and Apache Spark analytics on Azure. With the plus 50 percent price cut on HDInsight, customers moving to the cloud are reaping more savings than ever.

New features

The new updates and capabilities fall in to the following categories:

1. Update Hadoop and other open-source projects – In addition to 1000+ bug fixes across 20+ open-source projects, this update contains a new version of Spark (2.3) and Kafka (1.0).

   a. New features in Apache Spark 2.3

   b. New features in Apache Kafka 1.0

2. Update R Server 9.1 to Machine Learning Services 9.3 – With this release, we are providing data scientists and engineers with the best of open source enhanced with algorithmic innovations and ease of operationalization, all available in their preferred language with the speed of Apache Spark. This release expands upon the capabilities offered in R Server with added support for Python, leading to the cluster name change from R Server to ML Services.

3. Support for Azure Data Lake Storage Gen2 – HDInsight will support the Preview release of Azure Data Lake Storage Gen2. In the available regions, customers will be able to choose an ADLS Gen2 account as the Primary or Secondary store for their HDInsight clusters.

4. HDInsight Enterprise Security Package Updates (Preview) – (Preview) Virtual Network Service Endpoints support for Azure Blob Storage, ADLS Gen1, Cosmos DB, and Azure DB.

Component versions

The official Apache versions of all HDInsight 3.6 components are listed below. All components listed here are official Apache releases of the most recent stable versions available.

• Apache Hadoop 2.7.3

• Apache HBase 1.1.2

• Apache Hive 1.2.1

• Apache Hive 2.1.0

• Apache Kafka 1.0.0

• Apache Mahout 0.9.0+

• Apache Oozie 4.2.0

• Apache Phoenix 4.7.0

• Apache Pig 0.16.0

• Apache Ranger 0.7.0

• Apache Slider 0.92.0

• Apache Spark 2.2.0/2.3.0

• Apache Sqoop 1.4.6

• Apache Storm 1.1.0

• Apache TEZ 0.7.0

• Apache Zeppelin 0.7.3

• Apache ZooKeeper 3.4.6

Later versions of a few Apache components are sometimes bundled in the HDP distribution in addition to the versions listed above. In this case, these later versions are listed in the Technical Previews table and should not substitute for the Apache component versions of the above list in a production environment.

Apache patch information

Hadoop

This release provides Hadoop Common 2.7.3 and the following Apache patches:

• HADOOP-13190: Mention LoadBalancingKMSClientProvider in KMS HA documentation.

• HADOOP-13227: AsyncCallHandler should use an event driven architecture to handle async calls.

• HADOOP-14104: Client should always ask namenode for kms provider path.

• HADOOP-14799: Update nimbus-jose-jwt to 4.41.1.

• HADOOP-14814: Fix incompatible API change on FsServerDefaults to HADOOP-14104.

• HADOOP-14903: Add json-smart explicitly to pom.xml.

• HADOOP-15042: Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0.

• HADOOP-15255: Upper/Lower case conversion support for group names in LdapGroupsMapping.

• HADOOP-15265: exclude json-smart explicitly from hadoop-auth pom.xml.

• HDFS-7922: ShortCircuitCache#close is not releasing ScheduledThreadPoolExecutors.

• HDFS-8496: Calling stopWriter() with FSDatasetImpl lock held may block other threads (cmccabe).

• HDFS-10267: Extra "synchronized" on FsDatasetImpl#recoverAppend and FsDatasetImpl#recoverClose.

• HDFS-10489: Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones.

• HDFS-11384: Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike.

• HDFS-11689: New exception thrown by DFSClient%isHDFSEncryptionEnabled broke hacky hive code.

• HDFS-11711: DN should not delete the block On "Too many open files" Exception.

• HDFS-12347: TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently.

• HDFS-12781: After Datanode down, In Namenode UI Datanode tab is throwing warning message.

• HDFS-13054: Handling PathIsNotEmptyDirectoryException in DFSClient delete call.

• HDFS-13120: Snapshot diff could be corrupted after concat.

• YARN-3742: YARN RM will shut down if ZKClient creation times out.

• YARN-6061: Add an UncaughtExceptionHandler for critical threads in RM.

• YARN-7558: yarn logs command fails to get logs for running containers if UI authentication is enabled.

• YARN-7697: Fetching logs for finished application fails even though log aggregation is complete.

HDP 2.6.4 provided Hadoop Common 2.7.3 and the following Apache patches:

• HADOOP-13700: Remove unthrown IOException from TrashPolicy#initialize and #getInstance signatures.

• HADOOP-13709: Ability to clean up subprocesses spawned by Shell when the process exits.

• HADOOP-14059: typo in s3a rename(self, subdir) error message.

• HADOOP-14542: Add IOUtils.cleanupWithLogger that accepts slf4j logger API.

• HDFS-9887: WebHdfs socket timeouts should be configurable.

• HDFS-9914: Fix configurable WebhDFS connect/read timeout.

• MAPREDUCE-6698: Increase timeout on TestUnnecessaryBlockingOnHist oryFileInfo.testTwoThreadsQueryingDifferentJobOfSameUser.

• YARN-4550: Some tests in TestContainerLanch fail on non-english locale environment.

• YARN-4717: TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails Intermittently due to IllegalArgumentException from cleanup.

• YARN-5042: Mount /sys/fs/cgroup into Docker containers as readonly mount.

• YARN-5318: Fix intermittent test failure of TestRMAdminService#te stRefreshNodesResourceWithFileSystemBasedConfigurationProvider.

• YARN-5641: Localizer leaves behind tarballs after container is complete.

• YARN-6004: Refactor TestResourceLocalizationService#testDownloadingResourcesOnContainer so that it is less than 150 lines.

• YARN-6078: Containers stuck in Localizing state.

• YARN-6805: NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code.

HBase

This release provides HBase 1.1.2 and the following Apache patches.

• HBASE-13376: Improvements to Stochastic load balancer.

• HBASE-13716: Stop using Hadoop's FSConstants.

• HBASE-13848: Access InfoServer SSL passwords through Credential Provider API.

• HBASE-13947: Use MasterServices instead of Server in AssignmentManager.

• HBASE-14135: HBase Backup/Restore Phase 3: Merge backup images.

• HBASE-14473: Compute region locality in parallel.

• HBASE-14517: Show regionserver's version in master status page.

• HBASE-14606: TestSecureLoadIncrementalHFiles tests timed out in trunk build on apache.

• HBASE-15210: Undo aggressive load balancer logging at tens of lines per millisecond.

• HBASE-15515: Improve LocalityBasedCandidateGenerator in Balancer.

• HBASE-15615: Wrong sleep time when RegionServerCallable need retry.

• HBASE-16135: PeerClusterZnode under rs of removed peer may never be deleted.

• HBASE-16570: Compute region locality in parallel at startup.

• HBASE-16810: HBase Balancer throws ArrayIndexOutOfBoundsException when regionservers are in /hbase/draining znode and unloaded.

• HBASE-16852: TestDefaultCompactSelection failed on branch-1.3.

• HBASE-17387: Reduce the overhead of exception report in RegionActionResult for multi().

• HBASE-17850: Backup system repair utility.

• HBASE-17931: Assign system tables to servers with highest version.

• HBASE-18083: Make large/small file clean thread number configurable in HFileCleaner.

• HBASE-18084: Improve CleanerChore to clean from directory which consumes more disk space.

• HBASE-18164: Much faster locality cost function and candidate generator.

• HBASE-18212: In Standalone mode with local filesystem HBase logs Warning message: Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream.

• HBASE-18808: Ineffective config check in BackupLogCleaner#getDeletableFiles().

• HBASE-19052: FixedFileTrailer should recognize CellComparatorImpl class in branch-1.x.

• HBASE-19065: HRegion#bulkLoadHFiles() should wait for concurrent Region#flush() to finish.

• HBASE-19285: Add per-table latency histograms.

• HBASE-19393: HTTP 413 FULL head while accessing HBase UI using SSL.

• HBASE-19395: [branch-1] TestEndToEndSplitTransaction.testMasterOpsWhileSplitting fails with NPE.

• HBASE-19421: branch-1 does not compile against Hadoop 3.0.0.

• HBASE-19934: HBaseSnapshotException when read replicas is enabled and online snapshot is taken after region splitting.

• HBASE-20008: [backport] NullPointerException when restoring a snapshot after splitting a region.

Hive

This release provides Hive 1.2.1 and Hive 2.1.0 in addition to the following patches:

Hive 1.2.1 Apache patches:

• HIVE-10697: ObjectInspectorConvertors#UnionConvertor does a faulty conversion.

• HIVE-11266: count(*) wrong result based on table statistics for external tables.

• HIVE-12245: Support column comments for an HBase backed table.

• HIVE-12315: Fix Vectorized double divide by zero.

• HIVE-12360: Bad seek in uncompressed ORC with predicate pushdown.

• HIVE-12378: Exception on HBaseSerDe.serialize binary field.

• HIVE-12785: View with union type and UDF to the struct is broken.

• HIVE-14013: Describe table doesn't show unicode properly.

• HIVE-14205: Hive doesn't support union type with AVRO file format.

• HIVE-14421: FS.deleteOnExit holds references to _tmp_space.db files.

• HIVE-15563: Ignore Illegal Operation state transition exception in SQLOperation.runQuery to expose real exception.

• HIVE-15680: Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query, in MR mode.

• HIVE-15883: HBase mapped table in Hive insert fail for decimal.

• HIVE-16232: Support stats computation for columns in QuotedIdentifier.

• HIVE-16828: With CBO enabled, Query on partitioned views throws IndexOutOfBoundException.

• HIVE-17013: Delete request with a subquery based on select over a view.

• HIVE-17063: insert overwrite partition onto a external table fail when drop partition first.

• HIVE-17259: Hive JDBC does not recognize UNIONTYPE columns.

• HIVE-17419: ANALYZE TABLE...COMPUTE STATISTICS FOR COLUMNS command shows computed stats for masked tables.

• HIVE-17530: ClassCastException when converting uniontype.

• HIVE-17621: Hive-site settings are ignored during HCatInputFormat split-calculation.

• HIVE-17636: Add multiple_agg.q test for blobstores.

• HIVE-17729: Add Database and Explain related blobstore tests.

• HIVE-17731: add a backward compat option for external users to HIVE-11985.

• HIVE-17803: With Pig multi-query, 2 HCatStorers writing to the same table will trample each other's outputs.

• HIVE-17829: ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2.

• HIVE-17845: insert fails if target table columns are not lowercase.

• HIVE-17900: analyze stats on columns triggered by Compactor generates malformed SQL with > 1 partition column.

• HIVE-18026: Hive webhcat principal configuration optimization.

• HIVE-18031: Support replication for Alter Database operation.

• HIVE-18090: acid heartbeat fails when metastore is connected via hadoop credential.

• HIVE-18189: Hive query returning wrong results when set hive.groupby.orderby.position.alias to true.

• HIVE-18258: Vectorization: Reduce-Side GROUP BY MERGEPARTIAL with duplicate columns is broken.

• HIVE-18293: Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore.

• HIVE-18327: Remove the unnecessary HiveConf dependency for MiniHiveKdc.

• HIVE-18341: Add repl load support for adding "raw" namespace for TDE with same encryption keys.

• HIVE-18352: introduce a METADATAONLY option while doing REPL DUMP to allow integrations of other tools.

• HIVE-18353: CompactorMR should call jobclient.close() to trigger cleanup (Prabhu Joseph via Thejas Nair).

• HIVE-18390: IndexOutOfBoundsException when query a partitioned view in ColumnPruner.

• HIVE-18429: Compaction should handle a case when it produces no output.

• HIVE-18447: JDBC: Provide a way for JDBC users to pass cookie info via connection string.

• HIVE-18460: Compactor doesn't pass Table properties to the Orc writer.

• HIVE-18467: support whole warehouse dump / load + create/drop database events (Anishek Agarwal, reviewed by Sankar Hariappan).

• HIVE-18551: Vectorization: VectorMapOperator tries to write too many vector columns for Hybrid Grace.

• HIVE-18587: insert DML event may attempt to calculate a checksum on directories.

• HIVE-18613: Extend JsonSerDe to support BINARY type.

• HIVE-18626: Repl load "with" clause does not pass config to tasks.

• HIVE-18660: PCR doesn't distinguish between partition and virtual columns.

• HIVE-18754: REPL STATUS should support 'with' clause.

• HIVE-18754: REPL STATUS should support 'with' clause.

• HIVE-18788: Clean up inputs in JDBC PreparedStatement.

• HIVE-18794: Repl load "with" clause does not pass config to tasks for non-partition tables.

• HIVE-18808: Make compaction more robust when stats update fails.

• HIVE-18817: ArrayIndexOutOfBounds exception during read of ACID table.

• HIVE-18833: Auto Merge fails when "insert into directory as orcfile".

• HIVE-18879: Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in classpath.

• HIVE-18907: Create utility to fix acid key index issue from HIVE-18817.

Hive 2.1.0 Apache Patches:

• HIVE-14013: Describe table doesn't show unicode properly.

• HIVE-14205: Hive doesn't support union type with AVRO file format.

• HIVE-15563: Ignore Illegal Operation state transition exception in SQLOperation.runQuery to expose real exception.

• HIVE-15680: Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query, in MR mode.

• HIVE-15883: HBase mapped table in Hive insert fail for decimal.

• HIVE-16757: Remove calls to deprecated AbstractRelNode.getRows.

• HIVE-16828: With CBO enabled, Query on partitioned views throws IndexOutOfBoundException.

• HIVE-17063: insert overwrite partition onto a external table fail when drop partition first.

• HIVE-17259: Hive JDBC does not recognize UNIONTYPE columns.

• HIVE-17530: ClassCastException when converting uniontype.

• HIVE-17600: Make OrcFile's enforceBufferSize user-settable.

• HIVE-17601: improve error handling in LlapServiceDriver.

• HIVE-17613: remove object pools for short, same-thread allocations.

• HIVE-17617: Rollup of an empty resultset should contain the grouping of the empty grouping set.

• HIVE-17621: Hive-site settings are ignored during HCatInputFormat split-calculation.

• HIVE-17629: CachedStore: Have a whitelist/blacklist config to allow selective caching of tables/partitions and allow read while prewarming.

• HIVE-17636: Add multiple_agg.q test for blobstores.

• HIVE-17702: incorrect isRepeating handling in decimal reader in ORC.

• HIVE-17729: Add Database and Explain related blobstore tests.

• HIVE-17731: add a backward compat option for external users to HIVE-11985.

• HIVE-17803: With Pig multi-query, 2 HCatStorers writing to the same table will trample each other's outputs.

• HIVE-17845: insert fails if target table columns are not lowercase.

• HIVE-17900: analyze stats on columns triggered by Compactor generates malformed SQL with > 1 partition column.

• HIVE-18006: Optimize memory footprint of HLLDenseRegister.

• HIVE-18026: Hive webhcat principal configuration optimization.

• HIVE-18031: Support replication for Alter Database operation.

• HIVE-18090: acid heartbeat fails when metastore is connected via hadoop credential.

• HIVE-18189: Order by position does not work when cbo is disabled.

• HIVE-18258: Vectorization: Reduce-Side GROUP BY MERGEPARTIAL with duplicate columns is broken.

• HIVE-18269: LLAP: Fast llap io with slow processing pipeline can lead to OOM.

• HIVE-18293: Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore.

• HIVE-18318: LLAP record reader should check interrupt even when not blocking.

• HIVE-18326: LLAP Tez scheduler - only preempt tasks if there's a dependency between them.

• HIVE-18327: Remove the unnecessary HiveConf dependency for MiniHiveKdc.

• HIVE-18331: Add relogin when TGT expire and some logging/lambda.

• HIVE-18341: Add repl load support for adding "raw" namespace for TDE with same encryption keys.

• HIVE-18352: introduce a METADATAONLY option while doing REPL DUMP to allow integrations of other tools.

• HIVE-18353: CompactorMR should call jobclient.close() to trigger cleanup.

• HIVE-18384: ConcurrentModificationException in log4j2.x library.

• HIVE-18390: IndexOutOfBoundsException when query a partitioned view in ColumnPruner.

• HIVE-18447: JDBC: Provide a way for JDBC users to pass cookie info via connection string.

• HIVE-18460: Compactor doesn't pass Table properties to the Orc writer.

• HIVE-18462: (Explain formatted for queries with map join has columnExprMap with unformatted column name).

• HIVE-18467: support whole warehouse dump / load + create/drop database events.

• HIVE-18488: LLAP ORC readers are missing some null checks.

• HIVE-18490: Query with EXISTS and NOT EXISTS with non-equi predicate can produce wrong result.

• HIVE-18506: LlapBaseInputFormat - negative array index.

• HIVE-18517: Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag correctly to support LLAP Caching).

• HIVE-18523: Fix summary row in case there are no inputs.

• HIVE-18528: Aggregate stats in ObjectStore get wrong result.

• HIVE-18530: Replication should skip MM table (for now).

• HIVE-18548: Fix log4j import.

• HIVE-18551: Vectorization: VectorMapOperator tries to write too many vector columns for Hybrid Grace.

• HIVE-18577: SemanticAnalyzer.validate has some pointless metastore calls.

• HIVE-18587: insert DML event may attempt to calculate a checksum on directories.

• HIVE-18597: LLAP: Always package the log4j2 API jar for org.apache.log4j.

• HIVE-18613: Extend JsonSerDe to support BINARY type.

• HIVE-18626: Repl load "with" clause does not pass config to tasks.

• HIVE-18643: don't check for archived partitions for ACID ops.

• HIVE-18660: PCR doesn't distinguish between partition and virtual columns.

• HIVE-18754: REPL STATUS should support 'with' clause.

• HIVE-18788: Clean up inputs in JDBC PreparedStatement.

• HIVE-18794: Repl load "with" clause does not pass config to tasks for non-partition tables.

• HIVE-18808: Make compaction more robust when stats update fails.

• HIVE-18815: Remove unused feature in HPL/SQL.

• HIVE-18817: ArrayIndexOutOfBounds exception during read of ACID table.

• HIVE-18833: Auto Merge fails when "insert into directory as orcfile".

• HIVE-18879: Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in classpath.

• HIVE-18944: Groupping sets position is set incorrectly during DPP.

Kafka

This release provides Kafka 1.0.0 and the following Apache patches.

• KAFKA-4827: Kafka connect: error with special characters in connector name.

• KAFKA-6118: Transient failure in kafka.api.SaslScramSslEndToEndAuthorizationTest.testTwoConsumersWithDifferentSaslCredentials.

• KAFKA-6156: JmxReporter can't handle windows style directory paths.

• KAFKA-6164: ClientQuotaManager threads prevent shutdown when encountering an error loading logs.

• KAFKA-6167: Timestamp on streams directory contains a colon, which is an illegal character.

• KAFKA-6179: RecordQueue.clear() does not clear MinTimestampTracker's maintained list.

• KAFKA-6185: Selector memory leak with high likelihood of OOM in case of down conversion.

• KAFKA-6190: GlobalKTable never finishes restoring when consuming transactional messages.

• KAFKA-6210: IllegalArgumentException if 1.0.0 is used for inter.broker.protocol.version or log.message.format.version.

• KAFKA-6214: Using standby replicas with an in memory state store causes Streams to crash.

• KAFKA-6215: KafkaStreamsTest fails in trunk.

• KAFKA-6238: Issues with protocol version when applying a rolling upgrade to 1.0.0.

• KAFKA-6260: AbstractCoordinator not clearly handles NULL Exception.

• KAFKA-6261: Request logging throws exception if acks=0.

• KAFKA-6274: Improve KTable Source state store auto-generated names.

Mahout

In HDP-2.3.x and 2.4.x, instead of shipping a specific Apache release of Mahout, we synchronized to a particular revision point on Apache Mahout trunk. This revision point is after the 0.9.0 release, but before the 0.10.0 release. This provides a large number of bug fixes and functional enhancements over the 0.9.0 release, but provides a stable release of the Mahout functionality before the complete conversion to new Spark-based Mahout in 0.10.0.

The revision point chosen for Mahout in HDP 2.3.x and 2.4.x is from the "mahout-0.10.x" branch of Apache Mahout, as of 19 December 2014, revision 0f037cb03e77c096 in GitHub.

In HDP-2.5.x and 2.6.x, we removed the "commons-httpclient" library from Mahout because we view it as an obsolete library with possible security issues, and upgraded the Hadoop-Client in Mahout to version 2.7.3, the same version used in HDP-2.5. As a result:

• Previously compiled Mahout jobs will need to be recompiled in the HDP-2.5 or 2.6 environment.

• There is a small possibility that some Mahout jobs may encounter "ClassNotFoundException" or "could not load class" errors related to "org.apache.commons.httpclient", "net.java.dev.jets3t", or related class name prefixes. If these errors happen, you may consider whether to manually install the needed jars in your classpath for the job, if the risk of security issues in the obsolete library is acceptable in your environment.

• There is an even smaller possibility that some Mahout jobs may encounter crashes in Mahout's hbase-client code calls to the hadoop-common libraries, due to binary compatibility problems. Regrettably, there is no way to resolve this issue except revert to the HDP-2.4.2 version of Mahout, which may have security issues. Again, this should be very unusual, and is unlikely to occur in any given Mahout job suite.

Oozie

This release provides Oozie 4.2.0 with the following Apache patches.

• OOZIE-2571: Add spark.scala.binary.version Maven property so that Scala 2.11 can be used.

• OOZIE-2606: Set spark.yarn.jars to fix Spark 2.0 with Oozie.

• OOZIE-2658: --driver-class-path can overwrite the classpath in SparkMain.

• OOZIE-2787: Oozie distributes application jar twice making the spark job fail.

• OOZIE-2792: Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark.

• OOZIE-2799: Setting log location for spark sql on hive.

• OOZIE-2802: Spark action failure on Spark 2.1.0 due to duplicate sharelibs.

• OOZIE-2923: Improve Spark options parsing.

• OOZIE-3109: SCA: Cross-Site Scripting: Reflected.

• OOZIE-3139: Oozie validates workflow incorrectly.

• OOZIE-3167: Upgrade tomcat version on Oozie 4.3 branch.

Phoenix

This release provides Phoenix 4.7.0 and the following Apache patches:

• PHOENIX-1751: Perform aggregations, sorting, etc., in the preScannerNext instead of postScannerOpen.

• PHOENIX-2714: Correct byte estimate in BaseResultIterators and expose as interface.

• PHOENIX-2724: Query with large number of guideposts is slower compared to no stats.

• PHOENIX-2855: Workaround Increment TimeRange not being serialized for HBase 1.2.

• PHOENIX-3023: Slow performance when limit queries are executed in parallel by default.

• PHOENIX-3040: Don't use guideposts for executing queries serially.

• PHOENIX-3112: Partial row scan not handled correctly.

• PHOENIX-3240: ClassCastException from Pig loader.

• PHOENIX-3452: NULLS FIRST/NULL LAST should not impact whether GROUP BY is order preserving.

• PHOENIX-3469: Incorrect sort order for DESC primary key for NULLS LAST/NULLS FIRST.

• PHOENIX-3789: Execute cross region index maintenance calls in postBatchMutateIndispensably.

• PHOENIX-3865: IS NULL does not return correct results when first column family not filtered against.

• PHOENIX-4290: Full table scan performed for DELETE with table having immutable indexes.

• PHOENIX-4373: Local index variable length key can have trailing nulls while upserting.

• PHOENIX-4466: java.lang.RuntimeException: response code 500 - Executing a spark job to connect to phoenix query server and load data.

• PHOENIX-4489: HBase Connection leak in Phoenix MR Jobs.

• PHOENIX-4525: Integer overflow in GroupBy execution.

• PHOENIX-4560: ORDER BY with GROUP BY doesn't work if there is WHERE on pk column.

• PHOENIX-4586: UPSERT SELECT doesn't take in account comparison operators for subqueries.

• PHOENIX-4588: Clone expression also if its children have Determinism.PER_INVOCATION.

Pig

This release provides Pig 0.16.0 with the following Apache patches.

• PIG-5159: Fix Pig not saving grunt history.

• PIG-5175: Upgrade jruby to 1.7.26.

Ranger

This release provides Ranger 0.7.0 and the following Apache patches:

• RANGER-1805: Code improvement to follow best practices in js.

• RANGER-1960: Take snapshot's table name into consideration for deletion.

• RANGER-1982: Error Improvement for Analytics Metric of Ranger Admin and Ranger KMS.

• RANGER-1984: Hbase audit log records may not show all tags associated with accessed column.

• RANGER-1988: Fix insecure randomness.

• RANGER-1990: Add One-way SSL MySQL support in Ranger Admin.

• RANGER-2006: Fix problems detected by static code analysis in ranger usersync for ldap sync source.

• RANGER-2008: Policy evaluation is failing for multiline policy conditions.

Slider

This release provides Slider 0.92.0 with no additional Apache patches.

Spark

This release provides Spark 2.3.0 and the following Apache patches:

• SPARK-13587: Support virtualenv in pyspark.

• SPARK-19964: Avoid reading from remote repos in SparkSubmitSuite.

• SPARK-22882: ML test for structured streaming: ml.classification.

• SPARK-22915: Streaming tests for spark.ml.feature, from N to Z.

• SPARK-23020: Fix another race in the in-process launcher test.

• SPARK-23040: Returns interruptible iterator for shuffle reader.

• SPARK-23173: Avoid creating corrupt parquet files when loading data from JSON.

• SPARK-23264: Fix scala.MatchError in literals.sql.out.

• SPARK-23288: Fix output metrics with parquet sink.

• SPARK-23329: Fix documentation of trigonometric functions.

• SPARK-23406: Enable stream-stream self-joins for branch-2.3.

• SPARK-23434: Spark should not warn `metadata directory` for a HDFS file path.

• SPARK-23436: Infer partition as Date only if it can be casted to Date.

• SPARK-23457: Register task completion listeners first in ParquetFileFormat.

• SPARK-23462: improve missing field error message in `StructType`.

• SPARK-23490: Check storage.locationUri with existing table in CreateTable.

• SPARK-23524: Big local shuffle blocks should not be checked for corruption.

• SPARK-23525: Support ALTER TABLE CHANGE COLUMN COMMENT for external hive table.

• SPARK-23553: Tests should not assume the default value of `spark.sql.sources.default`.

• SPARK-23569: Allow pandas_udf to work with python3 style type-annotated functions.

• SPARK-23570: Add Spark 2.3.0 in HiveExternalCatalogVersionsSuite.

• SPARK-23598: Make methods in BufferedRowIterator public to avoid runtime error for a large query.

• SPARK-23599: Add a UUID generator from Pseudo-Random Numbers.

• SPARK-23599: Use RandomUUIDGenerator in Uuid expression.

• SPARK-23601: Remove .md5 files from release.

• SPARK-23608: Add synchronization in SHS between attachSparkUI and detachSparkUI functions to avoid concurrent modification issue to Jetty Handlers.

• SPARK-23614: Fix incorrect reuse exchange when caching is used.

• SPARK-23623: Avoid concurrent use of cached consumers in CachedKafkaConsumer (branch-2.3).

• SPARK-23624: Revise doc of method pushFilters in Datasource V2.

• SPARK-23628: calculateParamLength should not return 1 + num of expressions.

• SPARK-23630: Allow user's hadoop conf customizations to take effect.

• SPARK-23635: Spark executor env variable is overwritten by same name AM env variable.

• SPARK-23637: Yarn might allocate more resource if a same executor is killed multiple times.

• SPARK-23639: Obtain token before init metastore client in SparkSQL CLI.

• SPARK-23642: AccumulatorV2 subclass isZero scaladoc fix.

• SPARK-23644: Use absolute path for REST call in SHS.

• SPARK-23645: Add docs RE `pandas_udf` with keyword args.

• SPARK-23649: Skipping chars disallowed in UTF-8.

• SPARK-23658: InProcessAppHandle uses the wrong class in getLogger.

• SPARK-23660: Fix exception in yarn cluster mode when application ended fast.

• SPARK-23670: Fix memory leak on SparkPlanGraphWrapper.

• SPARK-23671: Fix condition to enable the SHS thread pool.

• SPARK-23691: Use sql_conf util in PySpark tests where possible.

• SPARK-23695: Fix the error message for Kinesis streaming tests.

• SPARK-23706: spark.conf.get(value, default=None) should produce None in PySpark.

• SPARK-23728: Fix ML tests with expected exceptions running streaming tests.

• SPARK-23729: Respect URI fragment when resolving globs.

• SPARK-23759: Unable to bind Spark UI to specific host name / IP.

• SPARK-23760: CodegenContext.withSubExprEliminationExprs should save/restore CSE state correctly.

• SPARK-23769: Remove comments that unnecessarily disable Scalastyle check.

• SPARK-23788: Fix race in StreamingQuerySuite.

• SPARK-23802: PropagateEmptyRelation can leave query plan in unresolved state.

• SPARK-23806: Broadcast.unpersist can cause fatal exception when used with dynamic allocation.

• SPARK-23808: Set default Spark session in test-only spark sessions.

• SPARK-23809: Active SparkSession should be set by getOrCreate.

• SPARK-23816: Killed tasks should ignore FetchFailures.

• SPARK-23822: Improve error message for Parquet schema mismatches.

• SPARK-23823: Keep origin in transformExpression.

• SPARK-23827: StreamingJoinExec should ensure that input data is partitioned into specific number of partitions.

• SPARK-23838: Running SQL query is displayed as "completed" in SQL tab.

• SPARK-23881: Fix flaky test JobCancellationSuite."interruptible iterator of shuffle reader".

Sqoop

This release provides Sqoop 1.4.6 with no additional Apache patches.

Storm

This release provides Storm 1.1.1 and the following Apache patches:

• STORM-2652: Exception thrown in JmsSpout open method.

• STORM-2841: testNoAcksIfFlushFails UT fails with NullPointerException.

• STORM-2854: Expose IEventLogger to make event logging pluggable.

• STORM-2870: FileBasedEventLogger leaks non-daemon ExecutorService which prevents process to be finished.

• STORM-2960: Better to stress importance of setting up proper OS account for Storm processes.

Tez

This release provides Tez 0.7.0 and the following Apache patches:

• TEZ-1526: LoadingCache for TezTaskID slow for large jobs.

Zeppelin

This release provides Zeppelin 0.7.3 with no additionalApache patches.

• ZEPPELIN-3072: Zeppelin UI becomes slow/unresponsive if there are too many notebooks.

• ZEPPELIN-3129: Zeppelin UI doesn't sign out in IE.

• ZEPPELIN-903: Replace CXF with Jersey2.

ZooKeeper

This release provides ZooKeeper 3.4.6 and the following Apache patches:

• ZOOKEEPER-1256: ClientPortBindTest is failing on Mac OS X.

• ZOOKEEPER-1901: [JDK8] Sort children for comparison in AsyncOps tests.

• ZOOKEEPER-2423: Upgrade Netty version due to security vulnerability (CVE-2014-3488).

• ZOOKEEPER-2693: DOS attack on wchp/wchc four letter words (4lw).

• ZOOKEEPER-2726: Patch for introduces potential race condition.

Fixed Common Vulnerabilities and Exposures

This section covers all Common Vulnerabilities and Exposures (CVE) that are addressed in this release.

​CVE-2017-7676

​CVE-2017-7677

​CVE-2017-9799

​CVE-2016-4970

​CVE-2016-8746

​CVE-2016-8751

Fixed issues for support

Fixed issues represent selected issues that were previously logged via Hortonworks Support, but are now addressed in the current release. These issues may have been reported in previous versions within the Known Issues section; meaning they were reported by customers or identified by Hortonworks Quality Engineering team.

Incorrect Results

Other

Performance

Potential Data Loss

Query Failure

Security

Stability

Supportability

Upgrade

Usability

Behavioral changes

Known issues

• HDInsight integration with ADLS Gen 2 There are two issues on HDInsight ESP clusters using Azure Data Lake Storage Gen 2 with user directories and permissions:

  1. Home directories for users are not getting created on Head Node 1. As a workaround, create the directories manually and change ownership to the respective user’s UPN.

  2. Permissions on /hdp directory is currently not set to 751. This needs to be set to

     chmod 751 /hdp 
     chmod –R 755 /hdp/apps
     

• Spark 2.3

  • [SPARK-23523][SQL] Incorrect result caused by the rule OptimizeMetadataOnlyQuery

  • [SPARK-23406] Bugs in stream-stream self-joins

  • Spark sample notebooks are not available when Azure Data Lake Storage (Gen2) is default storage of the cluster.

• Enterprise Security Package

  • Spark Thrift Server does not accept connections from ODBC clients. Workaround steps:
    1. Wait for about 15 minutes after cluster creation.
    2. Check ranger UI for existence of hivesampletable_policy.
    3. Restart Spark service. STS connection should work now.

• Workaround for Ranger service check failure

  • RANGER-1607: Workaround for Ranger service check failure while upgrading to HDP 2.6.2 from previous HDP versions.

    Note

    Only when Ranger is SSL enabled.

  This issue arises when attempting to upgrade to HDP-2.6.1 from previous HDP versions through Ambari. Ambari uses a curl call to do a service check to Ranger service in Ambari. If the JDK version used by Ambari is JDK-1.7, the curl call will fail with the below error:

  curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

  The reason for this error is the tomcat version used in Ranger is Tomcat-7.0.7*. Using JDK-1.7 conflicts with default ciphers provided in Tomcat-7.0.7*.

  You can resolve this issue in two ways:

  • Update the JDK used in Ambari from JDK-1.7 to JDK-1.8 (see the section Change the JDK Version in the Ambari Reference Guide).

  • If you want to continue supporting a JDK-1.7 environment:

    1. Add the property ranger.tomcat.ciphers in the ranger-admin-site section in your Ambari Ranger configuration with the below value:

       SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA

    2. If your environment is configured for Ranger-KMS, add the property ranger.tomcat.ciphers in theranger-kms-site section in your Ambari Ranger configuration with the below value:

       SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA

  Note

  The noted values are working examples and may not be indicative of your environment. Ensure that the way you set these properties matches how your environment is configured.

• RangerUI: Escape of policy condition text entered in the policy form

  Component Affected: Ranger

  ​Description of Problem

  If a user wants to create policy with custom policy conditions and the expression or text contains special characters, then policy enforcement will not work. Special characters are converted into ASCII before saving the policy into the database.

  Special Characters: & < > " ` '

  For example, the condition tags.attributes['type']='abc' would get converted to the following once the policy is saved.

  tags.attds[&#x27;dsds&#x27;]=&#x27;cssdfs&#x27;

  You can see the policy condition with these characters by opening the policy in edit mode.

  ​Workaround

  • ​Option #1: Create/Update policy via Ranger Rest API

    REST URL: http://<host>:6080/service/plugins/policies

    Creating policy with policy condition:

    The following example will create policy with tags as `tags-test` and assign it to `public` group with policy condition astags.attr['type']=='abc' by selecting all hive component permissions like select, update, create, drop, alter, index, lock, all.

    Example:

    curl -H "Content-Type: application/json" -X POST http://localhost:6080/service/plugins/policies -u admin:admin -d '{"policyType":"0","name":"P100","isEnabled":true,"isAuditEnabled":true,"description":"","resources":{"tag":{"values":["tags-test"],"isRecursive":"","isExcludes":false}},"policyItems":[{"groups":["public"],"conditions":[{"type":"accessed-after-expiry","values":[]},{"type":"tag-expression","values":["tags.attr['type']=='abc'"]}],"accesses":[{"type":"hive:select","isAllowed":true},{"type":"hive:update","isAllowed":true},{"type":"hive:create","isAllowed":true},{"type":"hive:drop","isAllowed":true},{"type":"hive:alter","isAllowed":true},{"type":"hive:index","isAllowed":true},{"type":"hive:lock","isAllowed":true},{"type":"hive:all","isAllowed":true}]}],"denyPolicyItems":[],"allowExceptions":[],"denyExceptions":[],"service":"tagdev"}'

    Update existing policy with policy condition:

    The following example will update policy with tags as `tags-test` and assign it to `public` group with policy condition astags.attr['type']=='abc' by selecting all hive component permissions like select, update, create, drop, alter, index, lock, all.

    REST URL: http://<host-name>:6080/service/plugins/policies/<policy-id>

    Example:

    curl -H "Content-Type: application/json" -X PUT http://localhost:6080/service/plugins/policies/18 -u admin:admin -d '{"id":18,"guid":"ea78a5ed-07a5-447a-978d-e636b0490a54","isEnabled":true,"createdBy":"Admin","updatedBy":"Admin","createTime":1490802077000,"updateTime":1490802077000,"version":1,"service":"tagdev","name":"P0101","policyType":0,"description":"","resourceSignature":"e5fdb911a25aa7f77af5a9546938d9ed","isAuditEnabled":true,"resources":{"tag":{"values":["tags"],"isExcludes":false,"isRecursive":false}},"policyItems":[{"accesses":[{"type":"hive:select","isAllowed":true},{"type":"hive:update","isAllowed":true},{"type":"hive:create","isAllowed":true},{"type":"hive:drop","isAllowed":true},{"type":"hive:alter","isAllowed":true},{"type":"hive:index","isAllowed":true},{"type":"hive:lock","isAllowed":true},{"type":"hive:all","isAllowed":true}],"users":[],"groups":["public"],"conditions":[{"type":"ip-range","values":["tags.attributes['type']=abc"]}],"delegateAdmin":false}],"denyPolicyItems":[],"allowExceptions":[],"denyExceptions":[],"dataMaskPolicyItems":[],"rowFilterPolicyItems":[]}'

  • ​Option #2: Apply Javascript changes

    Steps to update JS file:

    1. Find out PermissionList.js file under /usr/hdp/current/ranger-admin

    2. Find out definition of renderPolicyCondtion function (line no:404).

    3. Remove following line from that function i.e under display function(line no:434)

       val = _.escape(val);//Line No:460

       After removing the above line, the Ranger UI will allow you to create policies with policy condition that can contain special characters and policy evaluation will be successful for the same policy.

HDInsight Integration with ADLS Gen 2: User directories and permissions issue with ESP clusters

1. Home directories for users are not getting created on Head Node 1. Workaround is to create these manually and change ownership to the respective user’s UPN.
2. Permissions on /hdp is currently not set to 751. This needs to be set to a. chmod 751 /hdp b. chmod –R 755 /hdp/apps

Deprecation

• OMS Portal: We have removed the link from HDInsight resource page that was pointing to OMS portal. Log Analytics initially used its own portal called the OMS portal to manage its configuration and analyze collected data. All functionality from this portal has been moved to the Azure portal where it will continue to be developed. HDInsight has deprecated the support for OMS portal. Customers will use HDInsight Log Analytics integration in Azure portal.

• Spark 2.3

  • https://spark.apache.org/releases/spark-release-2-3-0.html#deprecations

​Upgrading

All of these features are available in HDInsight 3.6. To get the latest version of Spark, Kafka and R Server (Machine Learning Services), please choose the Spark, Kafka, ML Services version when you create a HDInsight 3.6 cluster. To get support for ADLS, you can choose the ADLS storage type as an option. Existing clusters will not be upgraded to these versions automatically.

All new clusters created after June 2018 will automatically get the 1000+ bug fixes across all the open-source projects. Please follow this guide for best practices around upgrading to a newer HDInsight version.