Azure HDInsight
An Azure managed cluster service for open-source analytics.
200 questions
1 answer
Accessing adls gen2 from an esp enabled HDinsights cluster
Hallo All, I have configured an HDI Cluster with ESP enabled and from the Ambari and Azure Console I can see that all the services are running fine. But when trying to execute a sample pi job even to list files using hdfs dfs -ls , I am getting the…
1 answer
Can we add new blob storages to an existing HDI cluster without r
Hi Team, I have a scenario where I have an existing cluster. I created 2 new blob storages. I wanted to add these storages to that existing HDI. Is there any way to add these storages to the HDI cluster and get effect without having to delete…
asked 2021-10-01T17:57:42.57+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 49%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
Kinya Munini
1
Reputation point
commented 2021-10-11T09:00:50.483+00:00
PRADEEPCHEEKATLA-MSFT
81,391
Reputation points Microsoft Employee
1 answer
How to model thousands of files from Azure Data Lake Gen 2 to Single dataset for analysis?
Hi, I have an initial 1000s of delimited files in Azure Data Lake Gen 2 storage account. I need to read all these files and create them as single dataset for analysis. This dataset must be preserved for future files. After these files are processed,…
asked 2021-09-21T21:30:45.83+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 34%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERD%3C/text%3E%3C/svg%3E)
Ramanathan Dakshina Murthy
21
Reputation points
commented 2021-09-28T04:30:39.677+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 63%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
svijay-MSFT
5,226
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Unable to access 101-hdinsight-linux-add-edge-node template
I am trying to Create an Edge node to HDInsight Cluster using link…
asked 2021-09-24T10:06:03.717+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 94%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Mahima Khatri
26
Reputation points
commented 2021-09-27T10:15:41.01+00:00
PRADEEPCHEEKATLA-MSFT
81,391
Reputation points Microsoft Employee
0 answers
Can't access the Hadoop services on HDInsight 4.0
Hi, ppl I deployed the ESP Hadoop clusters in my VNET. But, I can't access to some Hadoop services as NameNode UI and Solr Ambari UI because I can't access to the FQDN:hn0-[clustername initial 6 characters].[AAD-DS DNS domain name] :…
asked 2021-09-22T11:04:50.397+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 55.00000000000001%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EII%3C/text%3E%3C/svg%3E)
井上 樹/ITSUKI INOUE
1
Reputation point
commented 2021-09-27T10:08:37.617+00:00
PRADEEPCHEEKATLA-MSFT
81,391
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Import HDI external HIVE Metadata DB into Synapse
I have an HDI 4.0 cluster I am trying to turn off and move to Synapse. I have an external HIVE DB. How can I import HIVE tables+metadata? I know I can first recreate the metadata by hand and then read from ADLS. I want to automate/eliminate…
asked 2021-09-15T22:17:04.87+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 56.00000000000001%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPB%3C/text%3E%3C/svg%3E)
Pablo Barcenas
21
Reputation points
accepted 2021-09-22T01:06:12.86+00:00
Pablo Barcenas
21
Reputation points
1 answer
HDInsights Cluster with ESP
I know that in-order to create a Kerberized HDInsights Cluster, we have to enable the Azure Active Directory Domain Services. My question is, Will it be possible to create a Kerberized HDInsights Cluster with external KDC Server...? By external KDC…
asked 2021-09-14T06:10:33.95+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 26%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVB%3C/text%3E%3C/svg%3E)
VIVEK B
1
Reputation point
commented 2021-09-21T09:43:54.517+00:00
PRADEEPCHEEKATLA-MSFT
81,391
Reputation points Microsoft Employee
0 answers
Data Migration between Kerberized CDH and ESP enabled HDInsights.
Hi All, We have a requirement for transferring HDFS data residing in our on-premise Kerberised CDH Cluster(MIT KDC) into an HDInsights cluster with ESP Enabled. Source will be CDH and destination will be adls. How can I do that since I see issues…
asked 2021-09-15T12:46:07.89+00:00
VIVEK B
1
Reputation point
commented 2021-09-20T06:05:35.347+00:00
PRADEEPCHEEKATLA-MSFT
81,391
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Need to delete HDICluster as of no use.
Hi, Need to know if there is any impact on storage or any other component when I decide to delete an HDI Cluster.
asked 2021-09-13T13:39:16.21+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 89%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMT%3C/text%3E%3C/svg%3E)
Mukul.Tanwar
21
Reputation points
accepted 2021-09-15T10:02:46.653+00:00
Mukul.Tanwar
21
Reputation points
0 answers
HDinsight cluster deletion failure
Hello, Currently, I am facing an issue in which my On-demand HDinsight cluster is not automatically deleted after the job execution and it is causing huge costs for me. I am looking for an automated process to delete the HDcluster if there is no…
asked 2021-09-13T09:08:52.557+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 59%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERR%3C/text%3E%3C/svg%3E)
Rinshad R
1
Reputation point
commented 2021-09-15T09:22:06.75+00:00
PRADEEPCHEEKATLA-MSFT
81,391
Reputation points Microsoft Employee
0 answers
Pyspark HDInsight DataFactory Eviroment Variable
I'm faced a problem with Pyspark, datafactory and HdInsight I create a HDInsight with 2 master and 2 slaves. I created environment variables in all server like sudo echo 'TEST=server' >> /etc/environment After that, in all server I…
.NET
.NET
Microsoft Technologies based on the .NET software framework.
3,502 questions
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
200 questions
asked 2021-09-08T21:57:22.217+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 82%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAJ%3C/text%3E%3C/svg%3E)
ALBERTO JUNIOR
1
Reputation point
commented 2021-09-14T07:03:51.75+00:00
PRADEEPCHEEKATLA-MSFT
81,391
Reputation points Microsoft Employee
0 answers
HDInsight cluster is failing when upgrading the version to 4.0
Hi Team, I am getting this error : Operation on target Txxxxxx failed: Hadoop job failed with exit code '2'. See…
asked 2021-08-02T10:50:45.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 56.00000000000001%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKD%3C/text%3E%3C/svg%3E)
Kajol Dhosewan
1
Reputation point
commented 2021-08-18T06:28:40.037+00:00
Kajol Dhosewan
1
Reputation point
3 answers
One of the answers was accepted by the question author.
What is the difference between : Azure Synapsis Analytics - Azure Databricks - Azure HD insight
Hello Everybody, I'm running a project where we need to propose an azure-based architecture to import data from an on-premises data warehouse (databases) to azure-based data platform. Data are aimed to be exposed to company operators through a web…
asked 2020-08-06T12:48:15.723+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 34%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
CloudRock
366
Reputation points
answered 2021-08-12T18:40:38.117+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 57.00000000000001%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Andrei Calin Juganaru
1
Reputation point
0 answers
Error with Hive Warehouse Connector Jar in Azure java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
I have the following Azure env HDP Spark version 2.3.2.3.1.0.319-3 Hadoop Version : 3.1.0.319-3 Hive Version : 3.1.0 Can anyone please suggest which versions of jars need to be used with the above env configuration. I am using following Jar…
asked 2021-07-30T13:45:09.27+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 12%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELM%3C/text%3E%3C/svg%3E)
Latif Mohammad Khan
1
Reputation point
commented 2021-08-11T01:23:31.147+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
HDInsight cluster stuck in 'delete' state
I issued a delete request for an HDInsight cluster over 12 hours ago, and it now says 'Deleting'. Anyway to complete the deletion of the cluster?
asked 2021-07-30T12:35:29.703+00:00
EB
1
Reputation point
commented 2021-08-10T18:42:17.04+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,771
Reputation points Microsoft Employee
0 answers
How to do big data analysis on Append Block with HDInsight or any other alternative service on Azure? (source log data produced by Azure Monitor)
Hello, I enable the Diagnostic settings in Monitoring of storage account, and the log will be sent to another storage account. And the default type of the log JSON file is append blob rather than block blob, and seems the type can not be changed. …
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 45%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIF%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Best practices for submitting spark batch jobs in Azure HDinsights.
Hi, I'm looking to submit my pyspark scripts in HDInsight. Currently, HDInsight provided Livy for job submission, using curl. However, If I want to productionize it, then what authentication mechanism to use. Also, How can I check the progress of…
asked 2021-07-19T08:22:48.617+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 52%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBS%3C/text%3E%3C/svg%3E)
Bipin Singh
21
Reputation points
accepted 2021-07-22T04:13:00.32+00:00
Bipin Singh
21
Reputation points
1 answer
One of the answers was accepted by the question author.
How to access an external storage account in HDInsight cluster without access key?
How does the HDInsight cluster to read data from a private blob container not set as the default or additional storage account during the cluster's creation. We don't want to use access key. Can we add additional storage accounts after the cluster…
asked 2021-07-14T09:55:18.937+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 74%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESH%3C/text%3E%3C/svg%3E)
Salander He
66
Reputation points Microsoft Employee
accepted 2021-07-16T03:04:53.627+00:00
Salander He
66
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How To Create an Azure Cluster In Another Region Using Azure Data Factory
I'm Trying to change the region where the Hd Insight Linked Service create an on-demand cluster. I Want to change from East Us to East Us 2 (Or West US) But I can't do it I've added in the following places the Location key without success: …
asked 2021-06-30T04:59:57.097+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 42%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJC%3C/text%3E%3C/svg%3E)
Joaquin Chemile
41
Reputation points
accepted 2021-07-03T15:37:07.52+00:00
Joaquin Chemile
41
Reputation points
0 answers
Llap is disabled and cant be enabled due to some reason and query with joins taking long to fetch the result
Llap is disabled in hdi4 manually and now query with joins taking 30 mins extra to fetch 30 millions rows
asked 2021-06-28T22:51:12.01+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 40%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
amrita
1
Reputation point
commented 2021-07-01T09:35:51.44+00:00
PRADEEPCHEEKATLA-MSFT
81,391
Reputation points Microsoft Employee