Databricks community edition- Error in SQL statement: IOException: Path: /FileStore/tables/clinicaltrial_2021/mesh.csv is a directory, which is not supported by the record reader when `mapreduce.input.fileinputformat.input.dir.recursive` is false.

Question

question

prathyushparuchuri-4352 asked May 11, '22 PRADEEPCHEEKATLA-MSFT commented Jun 6, '22

Databricks community edition- Error in SQL statement: IOException: Path: /FileStore/tables/clinicaltrial_2021/mesh.csv is a directory, which is not supported by the record reader when `mapreduce.input.fileinputformat.input.dir.recursive` is false.

I am writing code in hiveQL using databricks community edition.
I have loaded the csv dataset files into dbfs, and created hive external table and tried loading table with data from uploaded dbfs file.

CREATE EXTERNAL TABLE IF NOT EXISTS data_table(
Id STRING,
Sponsor STRING,
Status STRING,
Start_Date STRING,
Completion_Date STRING,
Type STRING,
Submission STRING,
Conditions ARRAY<STRING>,
Interventions STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
LOCATION '/FileStore/tables/CT21/clinicaltrial_2021-1.csv'
TBLPROPERTIES("skip.header.line.count"="2");

result was OK.
But when I queried
**select from data_table*

It throws an exception:
Error in SQL statement: IOException: Path: /FileStore/tables/clinicaltrial_2021/mesh.csv is a directory, which is not supported by the record reader when mapreduce.input.fileinputformat.input.dir.recursive is false.

When I checked dbfs directory:
There exists a new directory mesh.csv created by dbfs with many subdirectories with same mesh.csv

Any help is highly appreciated.

azure-databricks azure-immersive-reader

capture.png (55.8 KiB)

· 2

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT · May 12 at 09:28 AM

Hello @prathyushparuchuri-4352,

Thanks for the question and using MS Q&A platform.

Could you please share the code snippet which you are trying to run?

0 ·

PRADEEPCHEEKATLA-MSFT · May 30 at 05:37 AM

Hello @prathyushparuchuri-4352,

Just checking in if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.

0 ·

Answer 1 · 2022-06-02T00:43:13.22Z

PrathyushParuchuri-9618 answered Jun 2, '22 PRADEEPCHEEKATLA-MSFT commented Jun 6, '22

Hello @PRADEEPCHEEKATLA-MSFT
I didn't able to check my email in a while. I have resolved the aforementioned error using the command
SET mapreduce.input.fileinputformat.input.dir.recursive=True.

Thank you very much for your response.

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT · Jun 06 at 04:44 AM

Hello @prathyushparuchuri-4352,

Glad to know that your issue has been resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread.

0 ·

question