How to Onboard a Managed Reference Project

This doc describes the various steps for APEX engagement PMs to set up a managed reference docset in OPS.

Note

This process is evolving as we move toward self-service, automated reference onboarding. In the meantime, only APEX engagement PMs can obtain the required permissions. Content and product teams should work through their assigned engagement PM to request reference onboarding.

Prerequisites

To follows these steps, engagement PMs need:

The content or product team requesting the onboarding must supply:

  • One or more NuGet packages non NuGet.org for the product to document, including XML comments if available. NuGet packages are recommended for ease of setup and maintenance. OR
  • (NOT recommended) a set of DLLs and optional XML comments.
    • DLLs, XML files, and dependencies should be placed on a share to be accessed by the APEX engagement team.
  • The GitHub repo to add the managed reference docset. The engagement PM needs admin perms on this repo.
  • The name for the docset. This should include dotnet to indicate the type of reference.

Set up a CI Job (NuGet)

The following steps only apply if you are using public NuGet packages, as recommended. If you are using DLLs, see Set up a CI Job (DLLs), below.

Create Your NuGet Package Listing

If you are using NuGet as recommended, start by configuring the package listing. Assuming that your NuGet packages are already on NuGet.org, you can create a new file in the mrefconfig repository, in the master branch:

  1. Go to https://apidrop.visualstudio.com/binaries/_git/mrefconfig?path=%2Fpackageconfig&version=GBmaster&_a=contents.
  2. Create a new file and name it {productname}-pac.csv. This will be the file used to group packages for your CI job.
  3. In the file, on a new line, enter the moniker base, package ID on NuGet.org, and comma-separated versions that need to be supported, like this:
ef,EntityFramework,6.2.0-beta1,6.1.3
Note

The version of the product onboarded should not be included in the moniker that is placed before the package ID. The full moniker will be created at build-time, combined with individual required versions, as specified in the CSV.

There should be one moniker per package. For moniker naming guidelines, see Reference Versioning & Monikers.

If you need more packages as part of the same managed reference docset, add each on a new line (one package per line). Always combine multiple versions of one package in the same entry.

Configure the CI Job (NuGet)

Click here to go to the Build section of the binaries project in the APIDrop instance. If you need write perms, contact the DOCS Product Team.

Notice the build definitions that already exist:

Warning

It is extremely important that you do not edit or remove existing build definitions.

Click + New to create a new build definition.

new definition

We have created a new template that you can leverage instead of building every step from scratch - simply scroll to the bottom of the template list, until you reach Custom and select Managed Reference Onboarding (NuGet):

new nuget mref definition

Rename your definition to {productname}-ECMAXML.

Save the definition under \ with the Default Agent Queue set to Hosted.

Setting Required Parameters

There are several parameters you need to set inside the job to make sure that it runs properly. First, select the Extract NuGet Packages step in the build definition view.

Here, in the Arguments section, you will need to point the job to the CSV file that you created earlier, that outlines your package details. Replace INSERT_PACKAGE_FILE_HERE.CSV with the name of the CSV file.

In addition to this, depending on what flavor of the .NET platform your package targets, you will need to set the -f argument to a recognizable NuGet Target Framework Moniker (TFM). A complete list of those can be obtained on the TFM page.

Note

For most cases, the TFM you will need is net47, which implies that you are working with the .NET Framework 4.7 (or earlier) APIs.

You do not need to modify any parameters from the job.

Setting Build Variables

Under certain circumstances, you already have content onboarded and will want to make sure that we augment the existing ECMAXML content with the new release. To do that, you will need to modify some build variables (inside the Variables section in the build definition view):

build variables

The following are needed:

  • GitHubBranch - the branch in which the content resides, referenced through the documentation repo that you are using.
  • GitHubCustomEmail - IGNORED. This parameter will be used when we automate checkins.
  • GitHubLocation - the ID of the GitHub repository, in the format {github_account}/{repo}, such as MicrosoftDocs/visualstudio-docs-dotnet-api-pr.
  • GitHubToken - the GitHub Personal Access Token (you can generate one in the Account Settings) that has repo-level access to the repository in question. Make sure to set it to a secret variable, to avoid token leaks and potential account compromises.
  • GitHubUsername - the username through which you are trying to access the documentation repo. It must have read permissions.
  • InRepoLocation - relative to the root, location of the documentation XMLs with the FrameworksIndex folder. Make sure to append a trailing \ to the path, such as dotnet\xml\.
  • NeedsToChangeContent - boolean value that determines whether the build job needs to change existing content (from an existing GitHub repo) or just generate new content on each run.

Building the Content

Once you set all the variables, select Save & Queue from the top action bar:

Queue New Build

Depending on the number of packages, the build can take anywhere from 5 minutes to 2 hours. Default timeout for the build is set to 60 minutes, so if it takes more than that, it will error out.

Note

If your build requires more than 1 hour to complete, make sure to remove the timeout limit in the Options view: Build Timeout

Once the build completes, the newly generated content will be made available as a build artifact that can be downloaded from VSTS:

Build Artifact

The content of the ZIP needs to be checked-in to the content repository by you - we are working on automating that step as well.

Note

We moved away from checking in files to an Azure Storage blob container, to centralize the build resources.

Set Up a CI Job (Local Libraries)

In some situations, partner teams might not be in a position to ship NuGet packages, and instead rely on a set of managed binaries. This is not the advised way to onboard content on docs.microsoft.com, since it adds a lot of overhead in terms of manual maintenance and organization, but can still be used when there is a good business justification.

Create Local DLL Copy

Go to https://apidrop.visualstudio.com/_git/binaries?_a=contents&path=%2F&version=GBmaster and clone the repo to your local machine.

In your local clone, add a folder structure as follows:

ROOT
+---docset name
|   +---{moniker}
|   +---dependencies
|   |   +---{moniker}

For example, for Dynamics 365 v9 Unified Service Desk, you might have:

ROOT
+---dynamics
|   +---dynamics-usd-9
|   +---dependencies
|   |   +---dynamics-usd-9

Assuming that the product team already gave you a location from which you can copy a set of DLLs and their dependencies, copy the DLLs to reflect and XML comment files, if available, to the {moniker} folder in the docset root.

Next, if there are dependency assemblies that core DLLs rely on, copy those to the dependencies/{moniker} folder and commit your changes.

When you return to the binaries repo in the browser, you will see your new folders, like this:

Provisioning the CI Job

Click here to go to the Build section of the binaries project in the APIDrop instance.

We have created a new template for the build definition, named Managed Reference Onboarding (Local) - you can see if if you click on + New in the Build view:

Name your definition {productname}-ECMAXML - that way, we keep it consistent with other build definitions. Make sure that you use the Hosted build agent queue, when asked.

Save the definition under \.

Set Correct Job Variables

Next, you need to make sure to set the correct local build variables:

  • GitHubBranch - the branch in which the content resides, referenced through the documentation repo that you are using.
  • GitHubCustomEmail - IGNORED. This parameter will be used when we automate checkins.
  • GitHubLocation - the ID of the GitHub repository, in the format {github_account}/{repo}, such as MicrosoftDocs/visualstudio-docs-dotnet-api-pr.
  • GitHubToken - the GitHub Personal Access Token (you can generate one in the Account Settings) that has access to the repository in question. Make sure to set it to a secret variable, to avoid token leaks and potential account compromises.
  • GitHubUsername - the username through which you are trying to access the documentation repo. It must have read permissions.
  • InRepoLocation - relative to the root, location of the documentation XMLs with the FrameworksIndex folder. Make sure to append a trailing \ to the path, such as dotnet\xml\.
  • NeedsToChangeContent - boolean value that determines whether the build job needs to change existing content (from an existing GitHub repo) or just generate new content on each run.
  • VstsToken - REQUIRED - needed to clone the binaries repository. You can create alternate credentials by visiting the account settings page.
  • VstsUsername - REQUIRED - needed to clone the binaries repository. Same as VstsToken, you can create it on the same account settings page.
  • LocalBinaryLocation - REQUIRED - the name of the docset folder in the binaries repo root.

Once you set up the VstsToken, make sure to make it secret, to avoid data leaks, and add it as an argument to the Generate XML Documentation step, like:

-VstsToken $(VstsToken)

Building the Content

Once you set all the variables, select Save & Queue from the top action bar:

Queue New Build

Depending on number of binaries, and associated doc XML files, the build might take some time. Default timeout for the build is set to 60 minutes, so if it takes more than that, it will error out.

Note

If your build requires more than 1 hour to complete, make sure to remove the timeout limit in the Options view: Build Timeout

Once the build completes, the newly generated content will be made available as a build artifact that can be downloaded from VSTS:

Build Artifact

The content of the ZIP needs to be checked-in to the content repository by you - we are working on automating that step as well.

Note

We moved away from checking in files to an Azure Storage blob container, to centralize the build resources.

Add Monikers in the OPS Portal

  1. Go to https://ops.microsoft.com/#/monikers.
  2. Click Add New.

Add new monikers corresponding to the package list you created for the CI job.

For example, this...

ef,EntityFramework,6.2.0-beta1,6.1.3

... becomes this:

Moniker Name       Moniker Display Name
ef-6.2.0-beta1     Entity Framework 6.2.0-beta1
ef-6.1.3           Entity Framework 6.1.3

Add Monikers to the API Browser

Send an email to Mike Sampson with your new moniker list.

Set up the docset

  1. At the repo root in GitHub, add a folder called dotnet.
  2. In the OPS Portal, provision a docset on the dotnet folder.
  3. Give the docset a descriptive name, such as <technology>-dotnet.
  4. Choose the appropriate them (Docs for docs.microsoft.com).
  5. Add a temporary base URL, generally the same as the docset name. This will later be changed on the backend to add your docs under the dotnet path.
  6. Choose your tenant, such as C+E.
  7. Click Show More Settings.
  8. Select the Product Family, such as VS.
  9. Select to enable APIScan (should be enabled for all new mref).

Here's an example. Numbers correspond to the steps above.

After provisioning runs, you will have the following repo structure:

REPOROOT
|   .gitignore
|   .openpublishing.build.ps1
|   .openpublishing.publish.config.json
|   LICENSE
|   LICENSE-CODE
|   README.md
|   ThirdPartyNotices
|   index.md
|
+---dotnet
|   |   docfx.json
|   |   TOC.md

Add the ECMAXML

  1. Fork and clone the repo.
  2. In your local clone, add an xml folder under docset.
  3. Copy the entire contents of the ECMAXML build folder that you downloaded from Azure Storage.

The resulting structure should be like this:

REPOROOT
| ...
|
+---dotnet
|   |   docfx.json
|   |   TOC.md
|   +---xml
|   |   +---FrameworksIndex
|   |   +---Namespace.Example.One
|   |   +---Namespace.Example.Two
| ...

Add Cross Reference Zip Files

To link to APIs outside the docset, you will need to one or more cross reference zip files to map the xref links to the published URLs. For example, to link to the .NET Framework documentation currently published to MSDN, add msdn.4.5.2.zip. The repo structure now looks like this:

REPOROOT
| ...
|
+---dotnet
|   |   docfx.json
|   |   TOC.md
|   +---xml
|   |   +---FrameworksIndex
|   |   +---Namespace.Example.One
|   |   +---Namespace.Example.Two
|   +---msdn.4.5.2.zip
| ...

For additional zip files, contact Alice Wang's team with pointers to the documentation sets you need to link to.

Configure docfx.json

Overwrite the default docfx.json with the following:

{
  "build": {
    "content": [
      {
        "files": [
          "**/*.md",
          "**/*.yml"
        ],
        "exclude": [
          "api/overwrites/**",
          "**/toc.yml",
          "**/obj/**",
          "**/includes/**",
          "README.md",
          "LICENSE",
          "LICENSE-CODE",
          "ThirdPartyNotices"
        ]
      },
      {
        "files": [
          "**/toc.yml"
        ],
        "src": ".",
        "dest": ""
      }
    ],
    "resource": [
      {
        "files": [
          "**/*.png",
          "**/*.jpg"
        ],
        "exclude": [
          "**/obj/**",
          "**/includes/**"
        ]
      }
    ],
    "overwrite": {
      "files": [
        "api/overwrites/**/*.md"
      ]
    },
    "xref": [],
    "externalReference": [],
    "globalMetadata": {
      "apiPlatform": "dotnet",
      "author": ""
    },
    "fileMetadata": {},
    "template": [],
    "dest": "",
    "lruSize": 0
  }
}

The reference specific fields are as follows:

Field Description Example
api/overwrites/** Tells the build to ignore overwrite files, if used. These are optionally used to apply metadata at the type level. Keep the default.
dest For the first dest, under content, specify the TOC output file. This file will be generated at build time. The name should be similar to the docset name but with underscores and _toc at the end. vsts_dotnet_toc
overwrite Specifies the location for optional overwritre files. Keep the default.
xref Indicate any xref zip files required to cross link to other repos. We'll come back to this step later! msdn.4.5.2.zip
apiPlatform Add global metadata to indicate the type of reference. For .NET, always use dotnet.
author Specifies the default author for the docset.
dest For the second dest, toward the end of the file, use the docset name plus -api. vsts-dotnet-api

Other customizations, such as additional global or file metadata, can also be added, just like for other docset. See DocFX Documentation for more information.

Here's an example:

Configure .openpublishing.build.config.json

Add the following custom sections to this file:

"customized_template_paths": [
        "_dependentPackages/memberpage.plugins/content"
      ],
      "customized_tasks": {
        "docset_prebuild": [
          "_dependentPackages/ECMA2Yaml/tools/Run.ps1",
          "_dependentPackages/splitToc/tools/run.ps1",
          "_dependentPackages/ECMA2Yaml/tools/DiffFolder.ps1"
        ],
        "docset_postbuild": [
          "_dependentPackages/ECMA2Yaml/tools/PostBuild.ps1"
        ]
      }
"template_folder": "_themes",
   "dependent_packages": [
    {
      "id": "Microsoft.DocAsCode.ECMA2Yaml",
      "nuget_feed": "https://www.myget.org/F/op/api/v2",
      "path_to_root": "_dependentPackages/ECMA2Yaml",
      "target_framework": "net45",
      "version": "latest"
    },
    {
      "id": "memberpage.plugins",
      "nuget_feed": "https://www.myget.org/F/docfx/api/v2",
      "path_to_root": "_dependentPackages/memberpage.plugins",
      "target_framework": "net45",
      "version": "latest"
    },
    {
      "id": "SplitToc",
      "nuget_feed": "https://www.myget.org/F/op/api/v2",
      "path_to_root": "_dependentPackages/splitToc",
      "target_framework": "net45",
      "version": "latest"
    }
  ],
  "ECMA2Yaml": {
    "SourceXmlFolder": "dotnet/xml",
    "OutputYamlFolder": "dotnet/api",
    "Flatten": true
  },
  "DiffFolder": [
    "dotnet/api"
  ],
  "toc_path_need_to_split": "dotnet/api/toc.yml"

Check in Your Local Changes

When all these steps are done in your local clone, create a pull request and check the build log. Fix any issues before merging into the upstream repo, especially if it is already live.

Note

It is recommended that you always do this through a pull request, so that any problems are detected early - there is a chance that the mdoc process missed some specific code pieces, or it might have been misconfigured in the CI job, therefore it's a good idea to double check before making the repo check-in.

Change Base URL (Backend)

To change the base URL to the standard .NET path, https://docs.microsoft.com/en-us/dotnet/api/, send email to Zhiliang Xu.

Other things to document

  • CAPS migration process.
  • Process for updates, including when docset contains authored content!