Hello @anil kumar ,
Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.
1) Azure Bastion is a service that lets you connect to a virtual machine using your browser and the Azure portal. Azure Bastion opens the RDP/SSH connection to your Azure virtual machine using private IP on your VM. You don't need a public IP on your virtual machine.
Reference : https://learn.microsoft.com/en-us/azure/bastion/bastion-overview
https://learn.microsoft.com/en-us/azure/bastion/tutorial-create-host-portal
2) P2S VPN should allow you to connect to a VM using it's private IP. May I know where the VPN client is installed? Are you accessing Azure VM from your local laptop via Azure VPN client?
From your intial statement : "Setup the VPN connection on Azure VM, I am able a access a website and RDP another VM using it's public IP address successfully even though source VM and destination VM were in different VNets and there was no peering between them." - I understand that your P2S VPN setup is completely on Azure. Is that correct? You have 2 Vnets. You are using one Vnet as Azure and other Vnet as on-prem and have installed P2S VPN on the other Vnet's VM.
Do you have any overlapping address spaces between Azure and your on-prem setup? Is the P2S VPN address pool setup correctly and not overlapping? You can also try resetting the Azure VPN gateway once and check again.
References : https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-point-to-site-resource-manager-portal#addresspool
https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-troubleshoot-vpn-point-to-site-connection-problems#the-point-to-site-vpn-connection-is-established-but-you-still-cannot-connect-to-azure-resources
3) The only way to connect to Azure VM using private IP from your laptop (provided there is no VPN set up and AD is not federated) would be via Bastion host.
Kindly let us know if the above helps or you need further assistance on this issue.
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.