Perform an offline deployment of a SQL Server big data cluster

This article describes how to perform an offline deployment of a SQL Server 2019 big data cluster (preview). Big data clusters must have access to a Docker repository from which to pull container images. An offline installation is one where the required images are placed into a private Docker repository. That private repository is then used as the image source for a new deployment.

Prerequisites

  • Docker Engine 1.8+ on any supported Linux distribution or Docker for Mac/Windows. For more information, see Install Docker.

Note

SQL Server big data clusters is first available as a limited public preview through the SQL Server 2019 Early Adoption Program. To request access, register here, and specify your interest to try SQL Server big data clusters. Microsoft will triage all requests and respond as soon as possible.

Load images into a private repository

The following steps describe how to pull the big data cluster container images from the Microsoft repository and then push them into your private repository.

Tip

The following steps explain the process. However, to simplify the task, you can use the automated script instead of manually running these commands.

  1. First, log in to the Microsoft Docker registry with the docker login command. Use the username and password that Microsoft provided to you as part of the Early Adoption Program.

    docker login private-repo.microsoft.com -u  <SOURCE_DOCKER_USERNAME> -p <SOURCE_DOCKER_PASSWORD>
    

    Tip

    These commands use PowerShell as an example, but you can run them from cmd, bash, or any command shell that can run docker. On Linux, add sudo to each command.

  2. Pull the big data cluster container images by repeating the following command. Replace <SOURCE_IMAGE_NAME> with each image name. Replace <SOURCE_DOCKER_TAG> with the tag for the big data cluster release, such as ctp3.0.

    docker pull private-repo.microsoft.com/mssql-private-preview/<SOURCE_IMAGE_NAME>:<SOURCE_DOCKER_TAG>
    
  3. Log in to the target private Docker registry.

    docker login <TARGET_DOCKER_REGISTRY> -u <TARGET_DOCKER_USERNAME> -p <TARGET_DOCKER_PASSWORD>
    
  4. Tag the local images with the following command for each image:

    docker tag private-repo.microsoft.com/mssql-private-preview/<SOURCE_IMAGE_NAME>:<SOURCE_DOCKER_TAG> <TARGET_DOCKER_REGISTRY>/<TARGET_DOCKER_REPOSITORY>/<SOURCE_IMAGE_NAME>:<TARGET_DOCKER_TAG>
    
  5. Push the local images to the private Docker repository:

    docker push <TARGET_DOCKER_REGISTRY>/<TARGET_DOCKER_REPOSITORY>/<SOURCE_IMAGE_NAME>:<TARGET_DOCKER_TAG>
    

Big data cluster container images

The following big data cluster container images are required for an offline installation:

  • mssql-appdeploy-init
  • mssql-monitor-fluentbit
  • mssql-monitor-collectd
  • mssql-server-data
  • mssql-hadoop
  • mssql-java
  • mssql-mlservices-pythonserver
  • mssql-mlservices-rserver
  • mssql-monitor-elasticsearch
  • mssql-monitor-influxdb
  • mssql-security-knox
  • mssql-mlserver-r-runtime
  • mssql-mlserver-py-runtime
  • mssql-controller
  • mssql-portal
  • mssql-server-controller
  • mssql-monitor-grafana
  • mssql-monitor-kibana
  • mssql-service-proxy
  • mssql-app-service-proxy
  • mssql-ssis-app-runtime
  • mssql-monitor-telegraf

Automated script

You can use an automated python script that will automatically pull all required container images and push them into a private repository.

Note

Python is a prerequisite for using the script. For more information about how to install Python, see the Python documentation.

  1. From bash or PowerShell, download the script with curl:

    curl -o push-bdc-images-to-custom-private-repo.py "https://raw.githubusercontent.com/Microsoft/sql-server-samples/master/samples/features/sql-big-data-cluster/deployment/offline/push-bdc-images-to-custom-private-repo.py"
    
  2. Then run the script with one of the following commands:

    Windows:

    python deploy-sql-big-data-aks.py
    

    Linux:

    sudo python deploy-sql-big-data-aks.py
    
  3. Follow the prompts for entering the Microsoft repository and your private repository information. After the script completes, all required images should be located in your private repository.

Install tools offline

Big data cluster deployments require several tools, including Python, mssqlctl, and kubectl. Use the following steps to install these tools on an offline server.

Install python offline

  1. On a machine with internet access, download one of the following compressed files containing Python:

    Operating system Download
    Windows https://go.microsoft.com/fwlink/?linkid=2074021
    Linux https://go.microsoft.com/fwlink/?linkid=2065975
    OSX https://go.microsoft.com/fwlink/?linkid=2065976
  2. Copy the compressed file to the target machine and extract it to a folder of your choice.

  3. For Windows only, run installLocalPythonPackages.bat from that folder and pass the full path to the same folder as a parameter.

    installLocalPythonPackages.bat "C:\python-3.6.6-win-x64-0.0.1-offline\0.0.1"
    

Install mssqlctl offline

  1. On a machine with internet access and Python, run the following command to download all off the mssqlctl packages to the current folder.

    pip download -r https://private-repo.microsoft.com/python/ctp-2.3/mssqlctl/requirements.txt
    
  2. Download the requirements.txt file.

    curl -o requirements.txt "https://private-repo.microsoft.com/python/ctp-2.3/mssqlctl/requirements.txt"
    
  3. Copy the downloaded packages and the requirements.txt file to the target machine.

  4. Run the following command on the target machine, specifying the folder that you copied the previous files into.

    pip install --no-index --find-links <path-to-packages> -r <path-to-requirements.txt>
    

Install kubectl offline

To install kubectl to an offline machine, use the following steps.

  1. Use curl to download kubectl to a folder of your choice. For more information, see Install kubectl binary using curl.

  2. Copy the folder to the target machine.

Deploy from private repository

To deploy from the private repository, use the steps described in the deployment guide, but use a custom deployment configuration file that specifies your private Docker repository information. The following mssqlctl commands demonstrate how to change the Docker settings in a custom deployment configuration file named custom.json:

mssqlctl cluster config section set -c custom.json -j "$.spec.controlPlane.spec.docker.repository=<your-docker-repository>"
mssqlctl cluster config section set -c custom.json -j "$.spec.controlPlane.spec.docker.registry=<your-docker-registry>"
mssqlctl cluster config section set -c custom.json -j "$.spec.controlPlane.spec.docker.imageTag=<your-docker-image-tag>"

The deployment prompts you for the docker username and password, or you can specify them in the DOCKER_USERNAME and DOCKER_PASSWORD environment variables.

Next steps

For more information about big data cluster deployments, see How to deploy SQL Server big data clusters on Kubernetes.