question

ChaviGupta-1099 avatar image
0 Votes"
ChaviGupta-1099 asked PRADEEPCHEEKATLA-MSFT commented

File paths in python package (.egg) in Databricks

I created a python package that contains a config file (.yaml) that is read in one of the functions. I use os.path to find the paths to read the file. The package works correctly on Windows.

In databricks I upload the .egg file to the cluster libraries. I get an error that the config file doesn't exist. While trying to list all the files in the running package directory, I get an error that the directory does not exist (os.path.dirname(file).

What is the best way to access the config file in the package in Databricks?

azure-databricks
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @ChaviGupta-1099,

Welcome to the Microsoft Q&A platform.

Could you please share the code which you are trying?

0 Votes 0 ·

Hello @ChaviGupta-1099,

Just checking in if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.

0 Votes 0 ·

Hi @PRADEEPCHEEKATLA-MSFT. Apologies for not replying sooner. I was able to figure out the issue. It was with the way I was accessing the file path inside the package. Instead of using os.path, we need to use pkg_resources.resource_filename().

0 Votes 0 ·

1 Answer

ChaviGupta-1099 avatar image
1 Vote"
ChaviGupta-1099 answered PRADEEPCHEEKATLA-MSFT commented

Use pkg_resources.resource_filename to access files within the package rather than os.path.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @ChaviGupta-1099,

Glad to know that your issue has resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread.

0 Votes 0 ·