Hello @Ajaz Nawaz
Welcome to Microsoft Q&A Platform.
If you want to connect to a specific VM via Load Balancer (so the traffic is not being balanced) you have to use inbound NAT rules:
https://azure.microsoft.com/en-us/blog/manage-port-forwarding-for-backend-pool-with-azure-load-balancer/
https://learn.microsoft.com/en-us/azure/load-balancer/tutorial-load-balancer-port-forwarding-portal
I hope this helps!
----------
Please don’t forget to "Accept the answer" and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.