I have tried editing the log4j properties file in
https://CLUSTERNAME.azurehdinsight.net/#/main/services/SPARK2/configs,
as mentioned in the following link:
https://docs.microsoft.com/en-us/azure/hdinsight/spark/troubleshoot-debug-wasb

I have done the following additions as shown in screenshot:
log4j.appender.audit.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.audit.triggeringPolicy.maxFileSize=50MB
][2]
But, the changes are not working, what can I do to limit the size of logs as you can see the stderr log size is increasing