It sounds like you may have users in incorrect roles if they are permitted to do things you don't want them to do.
If you went into this in detail then it would mean a reworking of your Azure RBAC controls - removing users from the Contributor role and adding them to more granular roles where possible.
I'm guessing that this approach is less feasible for you - hence the question of how to pare back on a defined Role? It may be possible to look at Azure Policies to prevent changes to the Firewall Rules except to Owners? I haven't dived deeply into the Azure Role's detailed ability, but this would be my starting point to see if it can be achieved there.