I can't use geopandas on Synapse SparkPool

Question

question

lucaspontes-9996 asked Apr 27, '22 PRADEEPCHEEKATLA-MSFT commented May 10, '22

I can't use geopandas on Synapse SparkPool

I need use some geospatial python packages on my Synapse SparkPool such as geopandas. I was able to install it as described in the documentation geospatial-processing-analytics using the yml file :

 name: aoi-env
 channels:
   - conda-forge
   - defaults
 dependencies:
   - gdal>=3.3.0
   - pip>=20.1.1
   - azure-storage-file-datalake
   - libgdal
   - shapely
   - pyproj
   - pip:
     - rasterio
     - geopandas
     - apache-sedona

However it doesn't work when trying to read data from Data Lake Gen2, it return the following error: 'No such file or directoryDriverError' and I'm pretty shure the path is correct.

I think it may be a dependency problem, so I tried to pass geopandas as a dependency but it doesn't work because it fails to install. After 30 min running the installation process it is cancelled

  name: aoi-env
     channels:
       - conda-forge
       - defaults
     dependencies:
       - gdal>=3.3.0
       - pip>=20.1.1
       - azure-storage-file-datalake
       - libgdal
       - shapely
       - pyproj
        - geopandas
       - pip:
         - rasterio
         - apache-sedona

azure-synapse-analytics

· 6

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT · Apr 28 at 10:01 AM

Hello @lucaspontes-9996,

Thanks for the question and using MS Q&A platform.

Could you please share the code snippet which you are trying to read data from Data Lake Gen2 and also share the stack trace or screenshot of the error message which you are experiencing?

0 ·

lucaspontes-9996 PRADEEPCHEEKATLA-MSFT · Apr 28 at 07:06 PM

Hello PRADEEPCHEEKATLA-MSFT,

There is the snippet and a screenshot of the error

 path = 'abfss://landing@datalaketeste1.dfs.core.windows.net/agrotools/geodata/solos_br.gpkg'
 df_pedo = gpd.read_file(path)

0 ·

image.png (72.5 KiB)

PRADEEPCHEEKATLA-MSFT lucaspontes-9996 · May 02 at 09:27 AM

Hello @lucaspontes-9996,

Thanks for sharing the additional details.

Could you please confirm - whether you have successfully installed geopandas library using the above environment[dot]yml file?

As per the repro from our end, we have found the error details from the Apache Spark Application =>SystemReservedJob-LibraryManagement => Stdout logs

0 ·

image.png (89.3 KiB)

Show more comments

Answer 1 · 2022-05-10T11:08:02.967Z

lucaspontes-9996 answered May 10, '22 PRADEEPCHEEKATLA-MSFT commented May 10, '22

Hi,

I used a work around. Just open the file with pandas and converted it to geopandas. It's not ideal but works for now

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT · May 10 at 11:12 AM

Hello @lucaspontes-9996,

Glad to know that your issue has been resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread.

0 ·

question