Episode

Ask the Expert: Smart Data Pipelines to Azure: Ingesting and migrating data the DataOps way

Ask the Expert

Ingesting and migrating data from a source system or legacy Hadoop stack to Azure platforms such as Synapse, ADLS and HDInsight may seem easy. But optimizing the data for a given Azure platform requires real data engineering. And operationalizing the data pipelines to run non-stop requires smart data pipelines that embed DataOps. Learn how StreamSets, a Microsoft startup program member, has helped customers speed their adoption of Azure cloud with its DataOps platform for smart data pipelines. Ready to learn more? https://streamsets.com and https://aka.ms/AzureMarketplaceStreamSets

Chapters

  • 00:00 - Introduction
  • 01:25 - What is StreamSets? What are some of the gaps you see in existing ETL tools?
  • 05:13 - What are some of the biggest trends and issues with modern data applications?
  • 08:56 - What's the breath of Azure technology supported by StreamSets? How to integrate with Azure Synapse?
  • 11:51 - How do you see customers leveraging both these Azure Data Factory pipeline capabilities alongside StreamSets?
  • 17:12 - What other Azure services or products does StreamSets agree with?
  • 18:06 - How do you see DevOps and DataOps practices fitting into these data workflows and data pipelines? Why should developers care about these tools?
  • 21:37 - How does StreamSets remains GDPR compliant?
  • 24:39 - How quickly can StreamSets deploy a data pipeline? How is the deployment process?
  • 26:37 - Does deploying workloads to Azure require rewrites?
  • 27:10 - What are the typical target customers?
  • 27:42 - Closing Notes