I have a databricks cluster with logging enabled in a dbfs location. My process requires to read the cluster logs, specifically the driver/stdout logs. This stdout is nothing but the console output which is also visible in UI : Clusters -> ClusterName -> Driver Logs -> Standard Output.
Every time, the UI shows me the stdout information very quickly but when I try to read the same from dbfs location, it's not yet present there and usually takes 4-5 minutes to reach there.
Is this delay expected? Is there any way to immediately sync the UI stdout and the dbfs stdout logs?
Enterprise Databricks configuration - 7.5 ML (includes Apache Spark 3.0.1, Scala 2.12)
