How to implement CI/CD on HDInsight ?

Federico Sardo 91 Reputation points
2023-07-26T15:51:14.8133333+00:00

Hi,

I need recommendations about how to apply CI/CD in HDinsight.

I want to create "dev" and "prod" cluster. One Spark and one Hive IQ each. Let's imagine:

my-cluster-spark-dev

my-cluster-hiveiq-dev

my-cluster-spark-prod

my-cluster-hiveiq-prod

Suppose that I am testing a flow and for that I created a hive table in dev. After running some test, everything goes well, and I need to deploy the create table script in the hive production cluster.

How can I achieve that? We are totally allowed to use Azure Devops.

Thanks,

Fede

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
199 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Federico Sardo 91 Reputation points
    2023-08-09T12:24:53.9233333+00:00

    Hi SatishBoddu-MSFT

    I think that you pointed me to a document about how to create a cluster with Ifra as code. (We allready did that).

    My question was related about how can I move , for instance, a pyspark script, or a hive DDL script from dev cluster to prod cluster.