question

DebashisPaul-8155 avatar image
0 Votes"
DebashisPaul-8155 asked PRADEEPCHEEKATLA-MSFT commented

Databricks Spark connection issue over Simba JDBC

I am trying to connect Spark Databricks from PERL code over Simba JDBC (Databricks recommended way) .For ref this is the JDBC driver: https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/jdbc/2.6.17/SimbaSparkJDBC42-2.6.17.1021.zip

So far I managed to setup PERL and all PERL related module config and below issue is nothing to do with PERL which I strongly believe.

I have below code trying to connect Spark Databricks

 #!/usr/bin/perl
    
 use strict;
 use DBI;
    
 my $user = "token";
 my $pass = "replaceme";
 my $host = "DBhost.azuredatabricks.net";
 my $port = 9001;
 my $url = "jdbc:spark://DBhost.azuredatabricks.net:443/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/853imaskedthis14/1005-imaskedthis-okra138;AuthMech=3;UID=token;PWD=replaceme";  # Get this URL from JDBC data src
    
 my %properties = ('user' => $user, 
                   'password' => $pass, 
                   'host.name' => $host, 
                   'host.port' => $port);
    
 my $dsn = "dbi:JDBC:hostname=localhost;port=$port;url=$url";
 my $dbh = DBI->connect($dsn, undef, undef, 
           { PrintError => 0, RaiseError => 1, jdbc_properties => \%properties })
           or die "Failed to connect: ($DBI::err) $DBI::errstr\n";
 my $sql = qq/select * from table/;
 my $sth = $dbh->prepare($sql);
 $sth->execute();
 my @row;
 while (@row = $sth->fetchrow_array) {
  print join(", ", @row), "\n";
 }

I am ending up below issue and error with SIMBA driver connecting to SPARK THRIFT server as Authentication issue.

 failed: [Simba][SparkJDBCDriver](500164) Error initialized or created transport for authentication: Invalid status 21
 Also, could not send response: com.simba.spark.jdbc42.internal.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed). at ./perldatabricksconntest.pl line 18.
 The logger recorded below Java stack trace:
    
 [Thread-1] 05:40:16,718 WARN  - Error
 java.sql.SQLException: [Simba][SparkJDBCDriver](500164) Error initialized or created transport for authentication: Invalid status 21
 Also, could not send response: com.simba.spark.jdbc42.internal.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed).
         at com.simba.spark.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
         at com.simba.spark.hivecommon.api.ServiceDiscoveryFactory.createClient(Unknown Source)
         at com.simba.spark.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
         at com.simba.spark.spark.core.SparkJDBCConnection.establishConnection(Unknown Source)
         at com.simba.spark.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
         at com.simba.spark.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
         at com.simba.spark.jdbc.common.AbstractDriver.connect(Unknown Source)
         at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
         at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
         at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:417)
         at com.vizdom.dbd.jdbc.Connection.run(Connection.java:211)
 Caused by: com.simba.spark.support.exceptions.GeneralException: [Simba][SparkJDBCDriver](500164) Error initialized or created transport for authentication: Invalid status 21
 Also, could not send response: com.simba.spark.jdbc42.internal.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed).
         ... 11 more

Also as per SIMBA JDBC connector document I have tried NO authentication mode, Username , Username / Password none of them working .

So wonder where is the Authentication issue here in transport layer . Note I already have created token and mentioned that in password section while initiating jdbc:spark call .

azure-databricks
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @DebashisPaul-8155,

Welcome to the Microsoft Q&A platform.

For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket.

0 Votes 0 ·

Hello @DebashisPaul-8155,

Did you get a chance to open a support ticket?

0 Votes 0 ·

1 Answer

DebashisPaul-8155 avatar image
0 Votes"
DebashisPaul-8155 answered PRADEEPCHEEKATLA-MSFT commented

Yes I did . thanks for following up .

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @DebashisPaul-8155,

Could you please share the Support Request (SR) number to track internally?

0 Votes 0 ·

Hello @DebashisPaul-8155,

Follow up to see if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.

0 Votes 0 ·

TrackingID#2104230010008280

0 Votes 0 ·
Show more comments