Which version of SharePoint you are using?
Per my test, in the modern library, SharePoint opens a document in a new tab by default. In the classic library, SharePoint opens a document in the same tab by default.
It you want to SharePoint opens a document in the same tab, you could vote this suggestion in the User Voice.