Put the entire name of the DL in single quotes, 'test1&test@mydomain.com'. Or "escape" the & symbol by adding ` in front of it. Or use a different identifier for the group, such as the GUID.
Import-CSV C:\temp\input.csv | ForEach {Add-DistributionGroupMember -Identity 'test1&test@mydomain.com' -Member $_.users -BypassSecurityGroupManagerCheck}