I have a HPC on-prem environment, one head node and 4 compute nodes. My company has a new security want to apply, kick off any user session if the session is being inactive for 30 minutes. After applying this policy, any jobs which need to be running for 30 mins are then cannot be completed. The job seems for stuck. And from the log file, i can find some message saying
"[RC] Exception detail: Microsoft.Hpc.Scheduler.Properties.SchedulerException: An exception occurred while attempting to access the scheduler database System.Data.SqlClient.SqlException. SHUTDOWN is in progress. .. Login failed for user 'domainABC\userABC'. .. Cannot continue execution because the session is in a terminated state. .. A serious error occurred in the current command. Any possible results should be discarded......."
Does any HPC users and the user account to be used for the DB connection need to keep connecting while the job is running?