question

carlosjosegonzalezarenas-2557 avatar image
0 Votes"
carlosjosegonzalezarenas-2557 asked carlosjosegonzalezarenas-2557 answered

Function that works in python but sql doesn't pyspark

Function that works in python but sql doesn't pyspark

I have successfully performed a function, and I have used it without problems, python, I have seen in videos and I have read that it can be implemented in both contexts, but I cannot do it myself

Here is an image where I use it in python and then in another cell I use it in sql, and the error message appears

184656-funcionregistrada.png


Function applied

184731-functionaplicada.png




Error message PythonException: 'AttributeError: 'NoneType' object has no attribute '_tools''


but if the geocode function works, it is because it is registered, I installed arcgis, and then I did this

%python
import unicodedata
from pathlib import Path
import pyspark.sql.functions as function
from pyspark.sql.types import *
import pandas as pd
import numpy as np
import re

import utm

import arcgis
from arcgis.geocoding import reverse_geocode
from arcgis.gis import GIS
from arcgis.geocoding import geocode
from arcgis.gis import GIS


azure-databricksdotnet-ml-big-data
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @carlosjosegonzalezarenas-2557,

Thank you for posting query in Microsoft Q&A Platform.

I tried same by creating function in python and registering it as udf and then calling it in SQL. Its working fine in my case.
185323-image.png

From your error message it seems some internal code of your function throwing error from line 16. Kindly see if we debug from there if any hint.

0 Votes 0 ·
image.png (82.8 KiB)

Hi @carlosjosegonzalezarenas-2557, Just checking if you got any clue from your internal debug,

0 Votes 0 ·

1 Answer

carlosjosegonzalezarenas-2557 avatar image
0 Votes"
carlosjosegonzalezarenas-2557 answered

186241-image.png


186169-image.png


186205-image.png



image.png (123.7 KiB)
image.png (85.1 KiB)
image.png (88.4 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.