This is strange. These can be difficult to troubleshoot, as there are 2 parts that could be having an issue: Azure Bastion & the VM. Azure Bastion typically just translates protocol in these cases.
To start, figure out where the issue is coming from by removing Azure Bastion from the scenario. The best way to do this is to connect to the VM via SSH outside of Bastion, typically via another VM in the VNET, or by attaching a Public IP to the VM. Once connected, see if the issue persists after waiting an hour.
If the issue is with the VM, here is a great doc that outlines common SSH errors to a Linux VM. If the issue is only with Azure Bastion, your best bet is to create a support request.