@lifonghsu , This error states that the key algorithm of the certificate used is not supported. It seems, somehow the the certificate being used to SAML request to AAD is not liked by AAD and hence AAD is throwing this error.
To help you further, it would be great if you can share some more details on this app and may be screenshots of your configuration for us to understand this in a better way.
Hope this helps.
Do let us know if this helps and if there are any more queries around this, please do let us know so that we can help you further. Also, please do not forget to accept the response as Answer; if the above response helped in answering your query.