@Girish Prajwal Here are the steps to make sure all the users have read access to Azure Load Balancer.
1) Create the Dynamic Group by following the steps mentioned this document so that it includes all the users of type Member and Guest of Azure Active Directory.
2) Create a Custom Role which has contain read access permissions to the Azure Load Balancer by following the steps mentioned in this document.
3) Assign the Custom Role to the Dynamic Group so that the permissions will be applied to all the users. For more information, refer this document.
Kindly try above steps and revert if you have further questions.