I know that HDInsight is using Python 3.5, but is there a way to upgrade the minor version to Python 3.6 or above? The reason is that we have a third party package which only works on Python 3.6 or above.
https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-python-package-installation
The documentation did not mention about how to do that. Or is it allowed to upgrade at all?