I do not fully understand the scenario.
What does the user experience look like? The user is not presented with a form to authenticate? The use is presented an authentication pop-up? Or does it fails when the user types username and password? Even if the password is correct? What are the error messages on the ADFS server if any? What are the security events generated on ADFS while trying (which would also implies you have enabled audit correctly on ADFS).