Azure HDInsight
An Azure managed cluster service for open-source analytics.
199 questions
4 answers
Unable to create HDInsight cluster using Microsoft tutorial
I am a student trying to learn how to setup HDInsight Hadoop cluster. I have signed up and received the free $200 credit. I am following the steps laid out in Microsoft's tutorial here:…
asked 2020-12-21T18:46:02.527+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 57.00000000000001%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJB%3C/text%3E%3C/svg%3E)
Justin Birchard
1
Reputation point
commented 2022-07-24T05:26:36.617+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 35%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
Vrunda
21
Reputation points
1 answer
azure hdinsight There are not enough cores available
I want to create HDInsight in my pay as you go subscription, but I get error: There are not enough cores available to support the selected number of nodes. I checked in my subscription usage and quotas for computing and usage is for every processor…
asked 2022-07-06T21:41:26.09+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 84%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Ales Ventus
46
Reputation points
commented 2022-07-11T05:39:06.973+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Files not getting saved in Azure blob using Spark in HDInsights cluster
We've setup HDInsights cluster on Azure with Blob as the storage for Hadoop. We tried uploading files to the Hadoop using hadoop CLI and the files were getting uploaded to the Azure Blob. Command used to upload: Hadoop fs -put somefile…
asked 2022-06-14T11:44:29.873+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 94%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Saif Ahmad
21
Reputation points
commented 2022-06-21T13:09:24.653+00:00
Saif Ahmad
21
Reputation points
1 answer
One of the answers was accepted by the question author.
Connect Synapse Spark Pool with Kafka on HDInsight
I have created a Kafka on HDinsight cluster . I have also created Azure Synapse Analytics - Spark Pool on same region as HDinsight. I need guidance on how to consume topics from Kafka into Spark Structured Streaming. Any documentation or steps will be of…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 44%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
HDInsight - Kafka - Version 3.2
Hi all Is there a roadmap to release a cluster with a higher kafka version than 2.4.1 in the near future? Thanks for the info in advance. Best reagrads, Michael
asked 2022-05-30T13:56:47.557+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 49%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Michael Ahrens
21
Reputation points
accepted 2022-06-07T09:26:48.037+00:00
Michael Ahrens
21
Reputation points
1 answer
One of the answers was accepted by the question author.
Can Azure Streaming Analytics read from Kafka on HDInsight and write to Deltalake table on Synapse lake.
Hello I am looking for guidance on building a new event driven platform. The options we are exploring for processing are - Azure Stream Analytics Apache Spark Structured Streaming in Synapse Source is like going to be Kafka on HDInsight …
1 answer
what is the best way to copy data from my hadoop on prem cluster to the azure hdinsight cluster?
hi experts, what is the best way to copy data from my hadoop on prem cluster to the azure hdinsight cluster? So we recently deployed a new hdinsight cluster and now I would like to copy some data from my onprem cluster to hdinsight. Thanks,
asked 2022-05-16T20:57:25.917+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 80%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERY%3C/text%3E%3C/svg%3E)
Richmond Yu
1
Reputation point
commented 2022-06-01T05:47:08.503+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
How to run hdfs commands from my on prem cluster to azure?
Hi experts, How to run hdfs commands from my on prem cluster to azure? So I have an on prem cluster that I would like to run hdfs commands to read files that are from my Azure HDinsight cluster. How can I do this? Thanks,
asked 2022-05-16T20:56:10.407+00:00
Richmond Yu
1
Reputation point
commented 2022-06-01T05:46:30.057+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
HDinsight monitering
Hi Friends i am new to HD insight any ida about hd insight clusters monitor What are the major things we need to observe and moniter we are using apache Ambari
asked 2022-05-10T08:17:22.893+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 15%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anshal
1,886
Reputation points
commented 2022-05-15T11:30:21.27+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 6%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELR%3C/text%3E%3C/svg%3E)
Luis Rodriguez
6,191
Reputation points Microsoft Employee
1 answer
Looking for HDInsight support team alias
I am running into an error while running a spark job in Azure Data Factory's pipeline and would like to connect to the HDInsight support team for further assistance. If you can please provide the alias
asked 2022-04-29T20:18:15.62+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 61%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHD%3C/text%3E%3C/svg%3E)
Harsha Deshmukh
1
Reputation point Microsoft Employee
commented 2022-05-10T07:31:36+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
Can't create HDInsight Cluster(Hadoop)
Dear All, I am struggling against creating HDInsight. After reviewing documents and other posts, I upgraded free-trial to paid subscription and created paid subscription as well. However, regardless of subscription types(both subscription paid type) I…
asked 2022-05-03T15:57:25.88+00:00
Jongmin Lee
6
Reputation points
commented 2022-05-09T05:15:42.647+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to leverage Azure key vault secrets from HD Insight Jupyter notebook?
Hi, I am trying to store the user id and password in Secrets and retrieve them in HD Insight Jupyter notebook? Any guidance.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 91%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
How to use GnuPG in HDInsight for encryption and decryption?
Hi, I am working with the HDInsight Spark cluster on Azure. Trying to encrypt files with pgp encryption using our private key. Is there a way that this can achieve rather than using the inbuilt encryption mechanism? How to set the home for GnuPG…
asked 2022-04-17T03:02:56.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 81%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
vijay singh parmar
26
Reputation points
accepted 2022-04-28T07:21:51.667+00:00
vijay singh parmar
26
Reputation points
1 answer
Not enough cores error while deploying resource group on Azure for Students
Hello! I am very new to Azure. I have an Azure for Students subscription and I'm trying to create an Apache Kafka cluster using Azure HDInsight. I selected West Europe as my region. I'm using this resource as a guide:…
asked 2022-04-15T00:28:01.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 32%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELM%3C/text%3E%3C/svg%3E)
Leila Moussa
1
Reputation point
commented 2022-04-22T03:57:47.553+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
Zeppelin notebook - sc.textFile does not work for HDI with ESP
We have HDI cluster with ESP enabled. From our zeppelin notebook, when I read data to a dataset (spark.read.text) it works but when I try to read it to an RDD (sc.textFile), I get an authentication exception: Note that, while sc.textFile…
asked 2021-02-11T15:14:14.283+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 71%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESL%3C/text%3E%3C/svg%3E)
Steven Lai
1
Reputation point
commented 2022-04-01T04:03:14.057+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
0 answers
unable to access index for repository https://mran.microsoft.com/snapshot/2017-03-15/src/contrib
The last time I did same thing last month, it was still ok, but today When I tried to install R package from MRAN repository; I got this error Checking the repository via browser also error could not find repository Could you please help me in the…
asked 2022-03-19T10:46:16.717+00:00
Suryanto
16
Reputation points
commented 2022-03-21T22:31:04.06+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,751
Reputation points Microsoft Employee
1 answer
HDInsight: Commands to clean up the space
Hi, at my workplace, we are using HDInsight 3.6. We have encountered space issues before, but we were able to resolve them by simply executing the simple cleanup commands from the edge node. Unfortunately, these commands have not been useful recently.…
asked 2022-03-10T19:51:28.77+00:00
vijay singh parmar
26
Reputation points
commented 2022-03-17T10:07:15.783+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
run job in HDInsight compute linked service
is username and password is the only way to submit job to HDInsight cluster? is managed identity or msi or service principal supported? Added question: can HDI team build API which uses AAD tokens as password instead of user input password? we have…
asked 2022-03-04T21:52:05.777+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 79%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBK%3C/text%3E%3C/svg%3E)
Bill Kan
21
Reputation points
commented 2022-03-11T03:03:01.097+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
Kafka REST Proxy Authentication
We would like to develop a HDInsight Kafka cluster to share real time data with a subcontractor. The REST proxy documentation indicates that "Kafka clients that need access to the REST proxy should be registered to a group by the group owner."…
asked 2022-02-18T18:30:46.19+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 43%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Mike McNulty
1
Reputation point
commented 2022-02-27T17:35:16.017+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
0 answers
Run C# mapreduce job
I am a beginner to Hadoop MapReduce. I have implemented a MapReduce job in visual C# and want to run it locally. As I understood, the HDInsight emulator hasn't been updated for a long time. What else options I have, to run the job locally?
asked 2022-02-17T17:59:18.127+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 86%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELS%3C/text%3E%3C/svg%3E)
Lilukshi Silva
1
Reputation point
commented 2022-02-25T20:30:24.82+00:00
Lilukshi Silva
1
Reputation point