Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,955 questions
How to reduce unnecessary high memory usage in a Databricks cluster?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 39%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESH%3C/text%3E%3C/svg%3E)
Senad Hadzikic
20
Reputation points
We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing else was executed).
Cluster config:
Some settings i tried:
Spark Config:
spark.executor.extraJavaOptions -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:ParallelGCThreads=20 -XX:ConcGCThreads=5 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:G1HeapRegionSize=8M spark.memory.storageFraction 0.5 spark.dynamicAllocation.maxExecutors 10 spark.driver.extraJavaOptions -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10M -Xloggc:/databricks/driver/logs/gc.log -XX:G1HeapRegionSize=8M -XX:+ExplicitGCInvokesConcurrent spark.dynamicAllocation.enabled true spark.memory.fraction 0.6 spark.dynamicAllocation.minExecutors 1
{count} votes
-
PRADEEPCHEEKATLA-MSFT 78,986 Reputation points Microsoft Employee2024-05-08T11:01:48.0733333+00:00 @Senad Hadzikic - Thanks for the question and using MS Q&A platform.
Based on the provided information, it seems that the cluster is not getting back to the idle state even after the script execution is completed. To reduce unnecessary high memory usage in a Databricks cluster, you can try the following steps:
- Turn on Auto Optimize by adding the following properties to your Spark configuration:
spark.databricks.delta.optimizeWrite.enabled true spark.databricks.delta.autoCompact.enabled true - Configure your cluster depending on your integration and scaling needs. For cluster configuration details, see Configure clusters.
- You can also try setting the
spark.dynamicAllocation.enabledproperty totrueto enable dynamic allocation of executors. This will allow the cluster to automatically scale up or down based on the workload. - Additionally, you can try setting the
spark.memory.fractionproperty to a lower value, such as0.4or0.5, to reduce the amount of memory allocated to Spark. - You can also try setting the
spark.dynamicAllocation.minExecutorsproperty to a lower value, such as1, to reduce the number of executors that are running when the cluster is idle.
If these steps do not help, you may need to further investigate the root cause of the high memory usage. You can use the Databricks monitoring and logging features to identify the source of the memory usage and take appropriate actions to optimize your cluster.
Hope this helps. Do let us know if you any further queries.
- Turn on Auto Optimize by adding the following properties to your Spark configuration:
-
Senad Hadzikic 20 Reputation points
2024-05-08T12:10:04.9066667+00:00 Hello and thank you for the answer, I'll try that out.
The monitoring tools didn't help much in debugging the issue to be honest. I've done some of my own investigations:- Ran this notebook:
%sh free -h top -b -n 1 | head -n 20 jps -lThe result:
total used free shared buff/cache available Mem: 57Gi 35Gi 12Gi 0.0Ki 9.5Gi 21Gi Swap: 9Gi 0.0Ki 9Gi top - 11:45:55 up 15 min, 0 users, load average: 1.38, 1.65, 1.44 Tasks: 32 total, 1 running, 30 sleeping, 0 stopped, 1 zombie %Cpu(s): 8.2 us, 0.4 sy, 0.0 ni, 91.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 58770.0 total, 12420.7 free, 36636.6 used, 9712.7 buff/cache MiB Swap: 10240.0 total, 10240.0 free, 0.0 used. 22133.4 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 759 root 20 0 57.3g 33.3g 186028 S 126.7 58.0 17:50.24 java 365 root 20 0 8775452 1.0g 47792 S 6.7 1.8 1:06.52 java 1 root 20 0 17964 10776 8272 S 0.0 0.0 0:00.19 systemd 45 root 20 0 39572 11828 10740 S 0.0 0.0 0:00.05 systemd+ 55 systemd+ 20 0 25540 12980 8788 S 0.0 0.0 0:00.01 systemd+ 57 root 20 0 3888 2600 2360 S 0.0 0.0 0:00.00 cron 58 message+ 20 0 8592 4776 4192 S 0.0 0.0 0:00.05 dbus-da+ 61 root 20 0 35280 22048 10344 S 0.0 0.0 0:00.16 network+ 63 syslog 20 0 160960 5056 4496 S 0.0 0.0 0:00.01 rsyslogd 67 root 20 0 15360 7616 6608 S 0.0 0.0 0:00.02 systemd+ 72 root 20 0 15436 8808 7256 S 0.0 0.0 0:00.00 sshd 74 root 20 0 2820 1108 1020 S 0.0 0.0 0:00.00 agetty 76 root 20 0 121200 23328 12720 S 0.0 0.0 0:00.14 unatten+ 759 com.databricks.backend.daemon.driver.DriverDaemon 508 org.apache.spark.deploy.master.Master 365 com.databricks.spark.chauffeur.ChauffeurDaemon 5598 sun.tools.jps.Jps- Then this notebook:
%python import os pid = os.popen("jps | grep DriverDaemon | awk '{print $1}'").read().strip() heap_dump_command = f"jmap -dump:live,format=b,file=/tmp/heapdump_{pid}.hprof {pid}" print(os.popen(heap_dump_command).read()) gc_stats_command = f"jstat -gc {pid}" print(os.popen(gc_stats_command).read()) memory_summary_command = f"jmap -heap {pid}" print(os.popen(memory_summary_command).read()) open_files_command = "lsof | grep java | wc -l" print(os.popen(open_files_command).read())The result:
1. Dumping heap to /tmp/heapdump_759.hprof ... Heap dump file created 2. S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT 0.0 0.0 0.0 0.0 29671424.0 73728.0 17424384.0 566112.3 499948.0 474495.4 0.0 0.0 76 4.449 1 0.958 5.406 3. Attaching to process ID 759, please wait... Debugger attached successfully. Server compiler detected. JVM version is 25.392-b08 using thread-local object allocation. Garbage-First (G1) GC with 13 thread(s) Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 48226107392 (45992.0MB) NewSize = 1363144 (1.2999954223632812MB) MaxNewSize = 28932308992 (27592.0MB) OldSize = 5452592 (5.1999969482421875MB) NewRatio = 2 SurvivorRatio = 8 MetaspaceSize = 21807104 (20.796875MB) CompressedClassSpaceSize = 1073741824 (1024.0MB) MaxMetaspaceSize = 17592186044415 MB G1HeapRegionSize = 8388608 (8.0MB) Heap Usage: G1 Heap: regions = 5749 capacity = 48226107392 (45992.0MB) used = 1418559840 (1352.8440856933594MB) free = 46807547552 (44639.15591430664MB) 2.9414769648925017% used G1 Young Generation: Eden Space: regions = 100 capacity = 30383538176 (28976.0MB) used = 838860800 (800.0MB) free = 29544677376 (28176.0MB) 2.7609055770292654% used Survivor Space: regions = 0 capacity = 0 (0.0MB) used = 0 (0.0MB) free = 0 (0.0MB) 0.0% used G1 Old Generation: regions = 70 capacity = 17842569216 (17016.0MB) used = 579699040 (552.8440856933594MB) free = 17262870176 (16463.15591430664MB) 3.2489661829652055% used 91848 interned Strings occupying 12598088 bytes. 4. lsof: WARNING: can't stat() fuse file system /local_disk0/.wsfs Output information may be incomplete.If this helps you give even better feedback, it didn't help me though.
-
Senad Hadzikic 20 Reputation points
2024-05-08T13:04:43.77+00:00 Thank you for the answer. I applied the suggested changes.
The memory usage is still really huge:I ran this notebook to get little more details:
%sh free -h top -b -n 1 | head -n 20 jps -lThe result:
total used free shared buff/cache available Mem: 57Gi 35Gi 10Gi 0.0Ki 10Gi 21Gi Swap: 9Gi 0.0Ki 9Gi top - 12:54:50 up 32 min, 0 users, load average: 2.48, 1.54, 1.23 Tasks: 30 total, 1 running, 28 sleeping, 0 stopped, 1 zombie %Cpu(s): 6.5 us, 0.0 sy, 0.0 ni, 93.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 58770.0 total, 11114.9 free, 36754.9 used, 10900.2 buff/cache MiB Swap: 10240.0 total, 10240.0 free, 0.0 used. 22015.1 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 796 root 20 0 57.2g 33.7g 185796 S 100.0 58.8 26:35.27 java 389 root 20 0 8704676 1.0g 47760 S 6.2 1.8 1:27.04 java 1 root 20 0 17972 10808 8284 S 0.0 0.0 0:00.23 systemd 41 root 20 0 39576 11764 10668 S 0.0 0.0 0:00.06 systemd+ 71 systemd+ 20 0 25540 12948 8756 S 0.0 0.0 0:00.01 systemd+ 77 root 20 0 3888 2396 2160 S 0.0 0.0 0:00.00 cron 78 message+ 20 0 8592 4764 4180 S 0.0 0.0 0:00.04 dbus-da+ 81 root 20 0 35228 21864 10208 S 0.0 0.0 0:00.16 network+ 83 syslog 20 0 160960 4924 4364 S 0.0 0.0 0:00.01 rsyslogd 87 root 20 0 15512 7772 6780 S 0.0 0.0 0:00.03 systemd+ 92 root 20 0 15436 8880 7332 S 0.0 0.0 0:00.00 sshd 94 root 20 0 2820 1072 984 S 0.0 0.0 0:00.00 agetty 99 root 20 0 121080 22996 12532 S 0.0 0.0 0:00.12 unatten+ 6881 sun.tools.jps.Jps 532 org.apache.spark.deploy.master.Master 389 com.databricks.spark.chauffeur.ChauffeurDaemon 796 com.databricks.backend.daemon.driver.DriverDaemonThis is happening on both of our clusters that we use.
-
PRADEEPCHEEKATLA-MSFT 78,986 Reputation points Microsoft Employee
2024-05-09T05:21:33.6966667+00:00 @Senad Hadzikic - Based on the output you provided, it looks like your cluster is still using a significant amount of memory even after applying the suggested changes. Here are a few more things you can try:
- Check if there are any long-running Spark jobs or notebooks that might be consuming memory. You can check this by going to the "Jobs" and "Notebooks" tabs in the Databricks workspace and looking for any jobs or notebooks that have been running for a long time.
- Try reducing the
spark.executor.memoryvalue to allocate less memory to each executor. You can also try reducing thespark.driver.memoryvalue to allocate less memory to the driver. - Check if there are any external libraries or dependencies that might be causing memory leaks. You can try removing any unnecessary libraries or dependencies and see if that helps reduce memory usage.
- Try monitoring the cluster's memory usage over time using the Databricks monitoring tools. This can help you identify any patterns or trends in memory usage and adjust your settings accordingly.
- Finally, you can try optimizing your Spark code to reduce memory usage. This can include techniques such as using smaller data partitions, using more efficient data serialization formats, and optimizing your Spark configuration settings.
I hope this helps! Let me know if you have any further questions.
-
Senad Hadzikic 20 Reputation points
2024-05-09T11:14:59.3033333+00:00 Thank you for the response once again. I have optimized the code to the max, as much as possible. These tips helped a bit but we still have a really huge memory usage. I was wondering if there's an additional possible way to release the cached memory which is 22GB (that is not restarting the cluster itself)? I've tried a couple of steps through a notebook cell but still no luck.
And this is the OS-level cache. I just want to be able to clear it, but can't manage to do it with a notebook it seems. Any other way?
-
PRADEEPCHEEKATLA-MSFT 78,986 Reputation points Microsoft Employee
2024-05-10T03:29:28.78+00:00 @Senad Hadzikic - If you want to release the cached memory in your Databricks cluster without restarting the cluster itself, you can try the following steps:
- Use the
spark.catalog.clearCache()method to clear the cached data in Spark. This method removes all cached data from memory and disk. You can run this method in a notebook cell to clear the cached data. - Use the
dbutils.fs.unmount()method to unmount any mounted file systems. Mounted file systems can consume memory, so unmounting them can help free up memory. You can run this method in a notebook cell to unmount any mounted file systems. - Use the
synccommand to flush the file system buffers and free up memory. You can run this command in a notebook cell to flush the file system buffers. - Use the
echo /proc/sys/vm/drop_cachescommand to drop the page cache, dentries, and inodes. This command can help free up memory that is being used by the operating system cache. However, this command requires root access, so you might need to contact your Databricks administrator to run this command. - Consider using a different type of Databricks cluster. For example, you might try using a different instance type or a different number of nodes to see if this improves memory usage.
Note that these steps might not free up all of the memory that is being used by your Databricks cluster, but they can help free up some memory. If you are still experiencing high memory usage after trying these steps, you might need to consider opening a support ticket for further assistance.
Hope this helps. Do let us know if you any further queries.
- Use the
-
PRADEEPCHEEKATLA-MSFT 78,986 Reputation points Microsoft Employee
2024-05-14T04:09:06.0466667+00:00 @Senad Hadzikic - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Sign in to comment