Frequently asked questions

Got questions? Check out the following FAQ for more information.

General FAQ

Can I install the extension on my Visual Studio Team Foundation Server instance instead of on an Azure DevOps instance?

No. The extension isn't available for downloading and installation for Visual Studio Team Foundation Server.

Do I have to run Microsoft Security Code Analysis with my build?

Maybe. It depends on the type of analysis tool. The source code might be the only thing that's required, or the build output might be required.

For example, Credential Scanner (CredScan) analyzes files within the folder structure of the code repository. Because of this analysis, you can run the CredScan and Publish Security Analysis Logs build tasks in a standalone build to get results.

For other tools like BinSkim that analyze post-build artifacts, the build is required first.

Can I break my build when results are found?

Yes. You can introduce a build break when any tool reports an issue or problem in its log file. Just add the Post-Analysis build task, and select the checkbox for any tool for which you want to break the build.

In the UI of the Post-Analysis task, you can choose to break the build when any tool reports either errors only or both errors and warnings.

How do the command-line arguments in Azure DevOps differ from those arguments in the standalone desktop tools?

For the most part, the Azure DevOps build tasks are direct wrappers around the command-line arguments of the security tools. You can pass as arguments to a build task anything you normally pass to a command-line tool.

Noticeable differences:

  • Tools run from the source folder of the agent $(Build.SourcesDirectory) or from %BUILD_SOURCESDIRECTORY%. An example is C:\agent_work\1\s.
  • Paths in the arguments can be relative to the root of the source directory previously listed. Paths can also be absolute. You get absolute paths either by using Azure DevOps Build Variables or by running an on-premises agent with known deployment locations of local resources.
  • Tools automatically provide an output file path or folder. If you provide an output location for a build task, that location is replaced with a path to our well-known location of logs on the build agent
  • Some additional command-line arguments are changed for some tools. One example is the addition or removal of options that ensure no GUI is launched.

Can I run a build task like Credential Scanner across multiple repositories in an Azure DevOps Build?

No. Running the secure development tools across multiple repositories in a single pipeline isn't supported.

The output file I specified isn't being created, or I can’t find the output file I specified

The build tasks filter some user input. For this question specifically, they update the location of the generated output file to be a common location on the build agent. For more information on this location, see the following questions.

Where are the output files generated by the tools saved?

The build tasks automatically add output paths to this well-known location on the build agent: $(Agent.BuildDirectory)_sdt\logs. Because we standardize on this location, all teams that produce or consume code-analysis logs have access to the output.

Can I queue a build to run these tasks on a hosted build agent?

Yes. All tasks and tools in the extension can be executed on a hosted build agent.

Note

The Anti-Malware Scanner build task requires a build agent with Windows Defender enabled. Hosted Visual Studio 2017 and later provide such an agent. The build task won't run on the Visual Studio 2015 hosted agent.

Although signatures can't be updated on these agents, signatures should always be less than three hours old.

Can I run these build tasks as part of a release pipeline as opposed to a build pipeline?

In most cases, yes.

However, Azure DevOps doesn't support running tasks within release pipelines when those tasks publish artifacts. This lack of support prevents the Publish Security Analysis Logs task from running successfully in a release pipeline. The task instead fails with a descriptive error message.

From where do the build tasks download the tools?

Build tasks can download the tools' NuGet packages from the Azure DevOps Package Management feed. Build tasks can also use Node Package Manager, which must be preinstalled on the build agent. An example of such installation is the command npm install tslint.

What effect does installing the extension have on my Azure DevOps organization?

Upon their installation, the security build tasks provided by the extension becomes available to all users in your organization. When you create or edit an Azure Pipeline, these tasks are available from the build-task collection list. Otherwise, installing the extension in your Azure DevOps organization has no effect. The installation doesn't modify any account settings, project settings, or pipelines.

Does installing the extension modify my existing Azure Pipelines?

No. Installing the extension makes the security build tasks available for addition to your pipelines. You're still required to add or update build definitions, so that the tools can work with your build process.

Task-specific FAQ

Questions specific to build tasks are listed in this section.

Credential Scanner

What are common suppression scenarios and examples?

Here are details of two of the most common suppression scenarios.

To suppress all occurrences of a given secret within the specified path

The hash key of the secret from the CredScan output file is required as shown in the following sample.

    {
        "tool": "Credential Scanner",
        "suppressions": [
        {
            "hash": "CLgYxl2FcQE8XZgha9/UbKLTkJkUh3Vakkxh2CAdhtY=",
            "_justification": "Secret used by MSDN sample, it is fake."
        }
      ]
    }

Warning

The hash key is generated by a portion of the matching value or file content. Any source-code revision can change the hash key and disable the suppression rule.

To suppress all secrets in a specified file or to suppress the secrets file itself

The file expression can be a file name. It can also be the basename part of a full file path or a file name. Wildcards are not supported.

The following examples show how to suppress the file <InputPath>\src\JS\lib\angular.js

Examples of valid suppression rules:

  • <InputPath>\src\JS\lib\angular.js - suppresses the file in the specified path

  • \src\JS\lib\angular.js

  • \JS\lib\angular.js

  • \lib\angular.js

  • angular.js - suppresses any file with the same name

      {
          "tool": "Credential Scanner",
          "suppressions": [
          {
              "file": "\\files\\AdditonalSearcher.xml", 
              "_justification": "Additional CredScan searcher specific to my team"
          },
          {
              "file": "\\files\\unittest.pfx", 
              "_justification": "Legitimate UT certificate file with private key"
          }
        ]
      }      
    

Warning

All future secrets added to the file will also be suppressed automatically.

The following resources help you securely manage secrets and access sensitive information from within your applications:

For more information, see the blog post Managing Secrets Securely in the Cloud.

Can I write my own custom searchers?

Credential Scanner relies on a set of content searchers that are commonly defined in the buildsearchers.xml file. The file contains an array of XML serialized objects that represent a ContentSearcher object. The program is distributed with a set of well-tested searchers. But you can implement your own custom searchers too.

A content searcher is defined as follows:

  • Name: The descriptive searcher name to be used in Credential Scanner output files. We recommended you use the camel-case naming convention for searcher names.

  • RuleId: The stable opaque ID of the searcher:

    • A Credential Scanner default searcher is assigned a RuleId value like CSCAN0010, CSCAN0020, or CSCAN0030. The last digit is reserved for potentially merging or dividing searcher groups via regular expressions (regex).
    • The RuleId value for a customized searcher should have its own namespace. Examples include CSCAN-<Namespace>0010, CSCAN-<Namespace>0020, and CSCAN-<Namespace>0030.
    • A fully qualified searcher name is the combination of a RuleId value and a searcher name. Examples include CSCAN0010.KeyStoreFiles and CSCAN0020.Base64EncodedCertificate.
  • ResourceMatchPattern: Regex of file extensions to check against the searcher.

  • ContentSearchPatterns: An array of strings containing regex statements to match. If no search patterns are defined, all files matching the ResourceMatchPattern value are returned.

  • ContentSearchFilters: An array of strings containing regex statements to filter searcher-specific false positives.

  • MatchDetails: A descriptive message, mitigation instructions, or both to be added for each match of the searcher.

  • Recommendation: The suggestions-field content for a match using the PREfast report format.

  • Severity: An integer that reflects the severity level of an issue. The highest severity level has the value 1.

    XML showing Credential Scanner setup

Roslyn Analyzers

What are common errors when using the Roslyn Analyzers task?

The project was restored using a wrong Microsoft.NETCore.App version

The full error message:

"Error: The project was restored using Microsoft.NETCore.App version x.x.x, but with current settings, version y.y.y would be used instead. To resolve this issue, make sure the same settings are used for restore and for subsequent operations such as build or publish. Typically this issue can occur if the RuntimeIdentifier property is set during build or publish but not during restore."

Because Roslyn Analyzers tasks run as part of compilation, the source tree on the build machine needs to be in a buildable state.

A step between your main build and Roslyn Analyzers steps might have put the source tree into a state that prevents building. This extra step is probably dotnet.exe publish. Try duplicating the step that does a NuGet restoration just before the Roslyn Analyzers step. This duplicated step might put the source tree back in a buildable state.

csc.exe can't create an analyzer instance

The full error message:

"'csc.exe' exited with error code 1 -- An instance of analyzer AAAA cannot be created from C:\BBBB.dll : Could not load file or assembly 'Microsoft.CodeAnalysis, Version=X.X.X.X, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified."

Ensure your compiler supports Roslyn Analyzers. Running the command csc.exe /version should report a version value of 2.6 or later.

Sometimes a .csproj file can override the build machine's Visual Studio installation by referencing a package from Microsoft.Net.Compilers. If you don't intend to use a specific version of the compiler, remove references to Microsoft.Net.Compilers. Otherwise, make sure the version of the referenced package is also 2.6 or later.

Try to get the error-log path, which is specified in the csc.exe /errorlog option. The option and path appear in the log for the Roslyn Analyzers build task. They might look something like /errorlog:F:\ts-services-123_work\456\s\Some\Project\Code\Code.csproj.sarif

The C# compiler version isn't recent enough

To get the latest versions of the C# compiler, go to Microsoft.Net.Compilers. To get your installed version, run csc.exe /version at a command prompt. Ensure that you reference a Microsoft.Net.Compilers NuGet package that is version 2.6 or later.

MSBuild and VSBuild logs aren't found

The Roslyn Analyzers build task needs to query Azure DevOps for the MSBuild log from the MSBuild build task. If the analyzer task runs immediately after the MSBuild task, the log won't yet be available. Place other tasks between the MSBuild task and the Roslyn Analyzers task. Examples of other tasks include BinSkim and Anti-Malware Scanner.

Next steps

If you need additional assistance, Microsoft Security Code Analysis Support is available Monday to Friday from 9:00 AM to 5:00 PM Pacific Standard Time.