I created a python package that contains a config file (.yaml) that is read in one of the functions. I use os.path to find the paths to read the file. The package works correctly on Windows.
In databricks I upload the .egg file to the cluster libraries. I get an error that the config file doesn't exist. While trying to list all the files in the running package directory, I get an error that the directory does not exist (os.path.dirname(file).
What is the best way to access the config file in the package in Databricks?