Basic auth to /hpc/ API

Scott Day 101

Hi,

I'm having trouble with the HPC REST API with HPC Pack 2019 update 1 when I use a particular AD account using the REST API.

The REST API works fine with my account, just not this dedicated AD account.

This account works when I clear my cached credentials in cluster manager and launch a job from there.

I have tried it with a separate grid running on HPC Pack 2019 (pre-update) and the REST API works fine with this account.

For example,

curl -k -v https://(head node running 2019).(my url).com/hpc/cluster/nodes -u "(redacted username):(redacted password)"

responds with the node info XML
but

curl -k -v https://(head node running 2019u1).(my url).com/hpc/cluster/nodes -u "(redacted username):(redacted password)"

responds with 401 unauthorized.

Normally accounts are given permission via an AD group, but I've tried manually adding it in Cluster Manager.

Is there some way to check what's going on here? How would I debug this?

vipullag-MSFT 24,206 Reputation points Microsoft Employee

2022-12-16T06:05:51.703+00:00

@Scott Day

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

Is there any difference between the dedicated AD account and the user account? Since the user account works fine with the REST call. Can you check if you have grant the same permission on the head node for the dedicated account?

You can use the LogViewer GUI tool to check the REST service logs at %CCP_DATA%LogFiles\RestService\RestService_*.bin on the head node to see if there is any related error.
Scott Day 101 Reputation points

2022-12-16T16:12:29.293+00:00

Hi, Thanks for the response.

Not really sure what the difference is, maybe that the username has a dash character in it? The username looks like "1234-123456789012a"

I've tried adding the account as administrator in Cluster Manager, but the REST API still doesn't work.

I looked at the logs, I don't see anything logged when I use the dedicated account. If I make an invalid call using my personal account I do get an error message in there.

Edit: I tried uninstalling and installing HPC Pack 2019 (pre update) and it's still doing the same thing. So I have one grid that works with the dedicated account and one that doesn't.
vipullag-MSFT 24,206 Reputation points Microsoft Employee

2022-12-19T05:35:39.627+00:00

@Scott Day

Thanks for responding back and sharing more details.

If you are using Basic Auth for an AD user you need to enter both the domain name and user name for -u option for curl, like "domain\user". And, make sure that you pay attention to the "\": you should consider the shell escape character "\", and type "\" for such shell environment, like Linux Bash.
Scott Day 101 Reputation points

2022-12-19T20:51:51.23+00:00

Hi,

I've tried using the domain\user format, as well as user@ad.domain.com form.

I can also recreate the problem using the hpc web portal, I can log in using my personal account, and I can log into a different head node using my account or this dedicated account, but if I use the dedicated account on the second head node it says "Invalid username or password!"

I've attached an image which hopefully explains the situation.
vipullag-MSFT 24,206 Reputation points Microsoft Employee

2023-01-10T09:59:08.207+00:00

@Scott Day

Apologies in delayed response on this.

Do you mean the "dedicated account" can login on head node 1 but cannot login on head node 2, while your "personal account" can login on both head nodes? If so, what's the difference between head node 1 and 2 for the "dedicated account"? If you open the Cluster Manager on head node2, can you see the "dedicated account" as an hpc pack user?

Please note that the "dedicated account" has to be a valid windows user, as well as an hpc pack user, on each node.

Hope that helps.
Scott Day 101 Reputation points

2023-01-18T19:21:01.99+00:00

Hi, yes that's correct.

I can't think of any substantial difference between the head nodes. I tried a fresh image of windows server 2019 and it still does the same.

Both nodes have the "dedicated account" as users in HPC.
Scott Day 101 Reputation points

2023-01-20T16:12:34.4966667+00:00

I traced the issue to the "Deny Log On Locally" security policy setting. The node that worked did not enforce this policy, and the dedicated account had this set.

Why does HPC require local log on? (rather than batch log on for example)

Is there a way to change this?
vipullag-MSFT 24,206 Reputation points Microsoft Employee

2023-02-14T06:43:30.2866667+00:00

Hello Scott Day

Apologies for delay in response.

By default, HPC Pack requires the Log On Locally right for user authentication to submit jobs and run tasks on the cluster. There is a job env ‘HPC_JOBLOGONTYPE’ to alter to batch log on when running tasks on the compute node, however there is no easy way to change how user authentication on the head node when submitting jobs.

Hope this helps.