@JustusNBB Thanks for the question. The reason for this nav property being of type DirectoryObject (the base type for pretty much all objects in the directory services part of Microsoft Graph) is that a manager could also be an organizational contact. Please see https://learn.microsoft.com/en-us/graph/api/user-post-manager?view=graph-rest-1.0&tabs=http#request-body.
Currently we have to use a base type in this kind of schema declaration - I don't think there's a (simple) way to indicate that this navigation property can be 2 or more specific types.
Hope this helps,
Dan.