Azure HDInsight
An Azure managed cluster service for open-source analytics.
199 questions
0 answers
Configure HDFS Storage for Zeppelin Notebooks on HDInsight Spark Clusters with ESP
Dear All We followed this step-by-step tutorial to configure HDFS storage for Zeppelin notebooks on our ESP-enabled HDInsight Spark Cluster (HDI 3.9, Spark 2.3):…
asked 2020-08-28T09:55:45.47+00:00
Christoph Kiefer
141
Reputation points
commented 2020-09-01T08:18:44.543+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
HDInsight Zeppelin Notebook Not Working
Hi All We are running an ESP-enabled HDInsight Spark cluster in Azure. We have no clue why some of our domain users are not able to use Zeppelin notebooks (usint the pyspark interpreter in our case). This is the very simple code that results in…
asked 2020-08-12T11:46:21.087+00:00
Christoph Kiefer
141
Reputation points
accepted 2020-08-28T05:44:19.06+00:00
Christoph Kiefer
141
Reputation points
0 answers
How to alter kafka topic config in ESP enabled HDInsight
Hi All, I am not able to alter topic config using kafka-configs.sh binary. I am passing jaas config file and I am sure the user has sufficient permissions. But I am always getting "org.apache.zookeeper.KeeperException$NoAuthException:…
asked 2020-08-14T15:37:30.83+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 60%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
mvidya
1
Reputation point
commented 2020-08-24T21:40:37.467+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
1 answer
Ad integration pass through for HDInsight
HI, I know HDInsight with ESP feature enables AD integration while connecting to cluster. Also teh access to underlying Hive tables can be controlled using Apache Ranger. But i would like to know if the access permission on storage account or datalake…
asked 2020-07-20T10:50:21.813+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 61%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHG%3C/text%3E%3C/svg%3E)
Hari GS
21
Reputation points
commented 2020-08-17T22:01:24.007+00:00
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Unable to run spark.sql queries on hive table using spark-shell (Class org.apache.hadoop.fs.adl.HdiAdlFileSystem not found)
I am trying to run below query on spark-shell on HDInsight cluster: val df=spark.sql("select * from hivesampletable") But it is giving below error repeatedly (irrespective of the query): 2020-08-10 07:29:21 WARN ObjectStore:568 -…
asked 2020-08-10T09:00:42.843+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 35%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Surya Agarwal
21
Reputation points
commented 2020-08-12T04:11:55.933+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
Proper Cores/Executors Configuration in HD-Insight
Proper Cores/Executors Configuration in HD-Insight And for this cluster i've this configuration Which is the best way to make a proper configuration in order to run efficiently a job in Spark. Is it Ok this configuration? Thanks!
asked 2020-08-04T15:19:31.557+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 42%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJC%3C/text%3E%3C/svg%3E)
Joaquin Chemile
41
Reputation points
commented 2020-08-10T03:41:14.82+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
2 answers
Is there any kind of Powerbi connector for Hadoop?
I've been testing some visualization tools and next is PowerBi. Some tools made me use apache drill, but it seems that Powerbi is full of connectors. Is there a way to connect naturally to hadoop (not hdinsight) or an easy workaround?
asked 2020-08-05T00:43:32.19+00:00
Geiber Arturo Ugalde Gutierrez
1
Reputation point
commented 2020-08-10T03:40:20.1+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
Using Spark action in HDInsight Hue
I have created a Spark 2.4 cluster using HDInsights in Azure. I have installed Hue over it using Script actions. Also did the necessary steps for SSH tunneling and connecting to Hue UI. However, on the Hue UI I am able to only see Pig and Hive…
asked 2020-07-23T09:32:26.923+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 45%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Abhijeet Bane
1
Reputation point
commented 2020-08-04T11:17:42.693+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
What are Recommended Solutions to work with RStudio and HDInsight Spark Cluster
Dear All We are currently implementing in-house / on-prem machine learning solutions in R (RStudio). We are in the process of moving our data to the cloud by the means of a sophisticated ingestion process through Apache Nifi. Currently the data lands…
asked 2020-07-01T20:37:32.087+00:00
Christoph Kiefer
141
Reputation points
commented 2020-07-27T08:36:26.973+00:00
Christoph Kiefer
141
Reputation points
1 answer
Not able to edit core-site.xml file
Hello, I am using free version of azure.i am trying to add some entry in core-site.xml for HDInsight configuration Which is at location /etc/hadoop/conf location. But,that file is read-only.how i can change permissions of that file So that i can…
asked 2020-07-02T19:18:07.353+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 77%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Jitendra7337
1
Reputation point
commented 2020-07-20T11:32:50.92+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
How to change Azure HDInsight Hadoop to Azure Private endpoint.
I already create a Azure HDInsight Hadoop cluster but in public network. Now,I want to change it into Private network using Azure Private Link to private endpoint. How can I change "https://<CLUSTERNAME>.azurehdinsight.net" to "…
asked 2020-05-08T23:16:35.12+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
answered 2020-07-20T03:35:32.467+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 88%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZZ%3C/text%3E%3C/svg%3E)
Zhenyu Zhou
1
Reputation point Microsoft Employee
0 answers
java.io.IOException: Stream is closed! Error in HDInsight with ADLS Gen 2
I am currently using Hail for the pyspark library to perform varying operations on Genomic data in ADLS Gen 2 with an HDInsight 4.0, Spark 2.4 cluster. I have been in touch with the development team regarding this error I have been getting when running a…
asked 2020-06-29T14:55:04.947+00:00
EagleByte
1
Reputation point
commented 2020-07-10T17:49:55.523+00:00
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
3 answers
Unable to access Azure Blob storage from HDInsight cluster
Hi, I have spun up a HDInsight Spark cluster and am trying to access blob storage on the cluster as follows, but getting an exception: hdfs dfs -ls wasbs://deploy@nisumstorageaccount2.blob.core.windows.net/ 20/06/16 19:28:39 ERROR…
asked 2020-06-17T00:52:46.397+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 10%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
azureuser586
1
Reputation point
commented 2020-07-10T05:44:31.297+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Unable to change spark.executor.heartbeatInterval parameter
I try to run a Jupyter Notebook on HDInsight with Spark; after some time (observed: 15, 17, 30 minutes), execution fails with error message: Error with 400 StatusCode: "requirement failed: Session isn't active." According to Stack Overflow…
asked 2020-06-22T06:27:09.24+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 73%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Kurmann Simon (Helbling Technik)
21
Reputation points
accepted 2020-06-24T06:17:19.367+00:00
Kurmann Simon (Helbling Technik)
21
Reputation points
1 answer
Configuring yarn alerts in Azure Monitor
Hi All, Is there any customized alerts to monitor the HDInsight yarn memory from Azure monitor.
asked 2020-06-13T18:29:29.653+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 60%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERT%3C/text%3E%3C/svg%3E)
Revanth Thavidishetty
1
Reputation point
commented 2020-06-17T17:47:23.807+00:00
Revanth Thavidishetty
1
Reputation point
1 answer
One of the answers was accepted by the question author.
Change node size in Azure Data Factory (Spark)
Hello! How do I change the node Size in Azure Data Factory for HD Insight? I Want to create an Standard_D13_v2. But I didn't find a tutorial for do it:
asked 2020-06-06T00:51:51.307+00:00
Joaquin Chemile
41
Reputation points
commented 2020-06-09T05:34:06.25+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
HDInsight Cluster create a file with write mask
Hi, We are using Data Lake Gen 1 as the data store for HDInsight Cluster version 3.6. I wrote a simple spark code to write a file using saveas command of pyspark. The file is created in the DataLake with a write mask on. As a result. Any other user is…
1 answer
Accessing Grafana on HDInsight Clusters
Hi All, We have multiple HDI clusters setup(Spark, Kafka, IQR). I have trouble accessing Grafana dashboards. I followed the guide (https://learn.microsoft.com/en-us/azure/hdinsight/interactive-query/hdinsight-grafana) with Ambari credentials…
asked 2020-02-11T06:32:17.82+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 62%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERD%3C/text%3E%3C/svg%3E)
Roshan Dissanayake
1
Reputation point
answered 2020-02-17T09:21:02.037+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
1 answer
Sample azure function for HBASE
Hello, I am trying to push data in HDInsights Hbase Cluster via Stream analytics jobs. Since there is no connectors available, i am looking for some sample azure function to complete my need as i am new to azure functions. Thanks & Regards, …
asked 2019-11-26T13:27:03.123+00:00
Sourav Nayak
1
Reputation point
answered 2019-12-01T14:53:20.007+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 94%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Balamurugan Balakreshnan
86
Reputation points Microsoft Employee