We don't use private endpoints but we do use the firewall.
I have noticed that after waiting for some random period time (for example 20 minutes, but it seems to keep changing) I can make some OPENROWSET selects again. However, when calling one particular OPENROWSET it breaks again and nothing works. I don't believe that there should be anything special about this one openrowset. It is in the same datalake and container as the ones that work.
I have used them in a case. If I understand you correctly the suggestion is not to use special characters in a case but the far easier solution would be for special characters be allowed and then ADF cleaning the case before using it as a "name". Or the name being something else than the case. I believe this was the case before (no pun intended). For example, I would imagine quite a few need to use "/" or paths as a part of their case.
I have mitigated the issue by using replace for special characters but this is not a good long term solution.
We have faced this bug. I hope this would get fixed...