question

prathyushparuchuri-4352 avatar image
0 Votes"
prathyushparuchuri-4352 asked PRADEEPCHEEKATLA-MSFT commented

Databricks community edition- Error in SQL statement: IOException: Path: /FileStore/tables/clinicaltrial_2021/mesh.csv is a directory, which is not supported by the record reader when `mapreduce.input.fileinputformat.input.dir.recursive` is false.

I am writing code in hiveQL using databricks community edition.
I have loaded the csv dataset files into dbfs, and created hive external table and tried loading table with data from uploaded dbfs file.

CREATE EXTERNAL TABLE IF NOT EXISTS data_table(
Id STRING,
Sponsor STRING,
Status STRING,
Start_Date STRING,
Completion_Date STRING,
Type STRING,
Submission STRING,
Conditions ARRAY<STRING>,
Interventions STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
LOCATION '/FileStore/tables/CT21/clinicaltrial_2021-1.csv'
TBLPROPERTIES("skip.header.line.count"="2");

result was OK.
But when I queried
**select from data_table*

It throws an exception:
Error in SQL statement: IOException: Path: /FileStore/tables/clinicaltrial_2021/mesh.csv is a directory, which is not supported by the record reader when mapreduce.input.fileinputformat.input.dir.recursive is false.


When I checked dbfs directory:
There exists a new directory mesh.csv created by dbfs with many subdirectories with same mesh.csv201197-capture.png

Any help is highly appreciated.


azure-databricksazure-immersive-reader
capture.png (55.8 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @prathyushparuchuri-4352,

Thanks for the question and using MS Q&A platform.

Could you please share the code snippet which you are trying to run?

0 Votes 0 ·

Hello @prathyushparuchuri-4352,

Just checking in if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.

0 Votes 0 ·

1 Answer

PrathyushParuchuri-9618 avatar image
1 Vote"
PrathyushParuchuri-9618 answered PRADEEPCHEEKATLA-MSFT commented

Hello @PRADEEPCHEEKATLA-MSFT
I didn't able to check my email in a while. I have resolved the aforementioned error using the command
SET mapreduce.input.fileinputformat.input.dir.recursive=True.

Thank you very much for your response.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @prathyushparuchuri-4352,

Glad to know that your issue has been resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread.

0 Votes 0 ·