question

If you're building a Windows Forms app then note that when the main window closes the app terminates. The main window is the first form that is opened. It sounds like you initially created the transaction window and then added the login form. But the app sees the main window as the first form you had (transaction). Go to your app's startup code (or search for where your transaction window class is used). Somewhere is a call to create and display that form and it is probably being passed to Application.Run. Change it to create an instance of your login form instead.

To prevent the app from closing when the login form goes away (the startup window) you'll need to adjust the behavior. VB actually has better support for this through its custom application class. Set the ShutdownStyle to only close after all forms are closed. If you're not using that base class for some reason then you can implement the equivalent logic as discussed here.

To provide assistance with your current code, consider sharing the code via a GitHub repository and if so make sure you do not include sensitive information.

I have a Microsoft TechNet article which discusses performing login with and without a database with a GitHub repository which can be cloned to try out the examples.

In all the code samples the login form is set in project properties, the login form is hidden given proper credentials to show the main form which closes the app via Application.ExitThread().

Code for no database login form

 Public Class LoginForm
     Private userID As String = "spicyramen06"
     Private password As String = "96619"
     Public Shared attempts As Integer = 0
     Private Sub LoginButton_Click(sender As Object, e As EventArgs) Handles LoginButton.Click
    
         attempts += 1
    
         If attempts >= 4 Then
             LoginButton.Enabled = False
             MessageBox.Show("Attempts exceeded")
             Exit Sub
         End If
         If Not String.IsNullOrWhiteSpace(UserNameTextBox.Text) AndAlso Not String.IsNullOrWhiteSpace(PasswordTextBox.Text) Then
             If UserNameTextBox.Text = userID AndAlso PasswordTextBox.Text = password Then
                 My.Application.UserName = UserNameTextBox.Text
                 Dim f As New MainForm
                 Me.Hide()
                 f.ShowDialog()
             Else
                 Controls.OfType(Of TextBox).ToList().ForEach(Sub(tb) tb.Text = "")
                 MessageBox.Show("Invalid logn")
             End If
         End If
     End Sub
    
     Private Sub CancelButton_Click(sender As Object, e As EventArgs) Handles CancelButton.Click
         Close()
     End Sub
 End Class

Main form

 Imports System.ComponentModel
    
 Public Class MainForm
     Private Sub MainForm_Closing(sender As Object, e As CancelEventArgs) Handles Me.Closing
         Application.ExitThread()
     End Sub
    
     Private Sub MainForm_Shown(sender As Object, e As EventArgs) Handles Me.Shown
         Text = $"Welcome [{My.Application.UserName}]"
     End Sub
 End Class

With a database dependent on requirements and if a database is used it can be more complex as explained in the TechNet article.