question

WozniakJoanna-0585 avatar image
0 Votes"
WozniakJoanna-0585 asked WozniakJoanna-0585 answered

Pipeline does not run new data

Hi -
I created and published a pipeline that pulls data from an Azure SQL table, processes, models and then appends the output to an Azure SQL table. The Azure SQL table is updated with new data every day or two. In my script, I want to model on data that has been added two days before today with the following script:

from datetime import date, timedelta
yesterday = date.today() - timedelta(days=2)
yesterday.strftime("%Y-%m-%d")
print(yesterday)

keep data that is 2 days ago only

data_prior = data[data['MatterOpenDate'] == str(yesterday)]
print(data_prior.head())

while True:
answer = data_prior.empty
if answer == False:
print('Continue Process')
break
elif answer == True:
print('Empty dataset')
run.complete()
exit()

When I first ran my pipeline it worked great. I published this experiment, etc. and created a reoccurring schedule to run once a day every day.

BUT the schedule continues to run the exact same data as the original run even when there is new data being uploaded. Why and what do I need to do for the script to run 'naturally' as written?

Thank you

azure-data-factoryazure-sql-databaseazure-machine-learning
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @WozniakJoanna-0585 and welcome to Microsoft Q&A.

To better assist you, could you help clarify what service or compute you are using?
Currently the question is tagged with Data Factory and Machine Learning. Is your pipeline a Machine Learning pipeline, or a Data Factory pipeline?
Is this code being run on Synapse notebook, or Databricks notebook or Batch compute or some Machine Learning service or something else?

0 Votes 0 ·

@WozniakJoanna-0585 I have not heard back from you. Are you still facing the issue?

0 Votes 0 ·

1 Answer

WozniakJoanna-0585 avatar image
0 Votes"
WozniakJoanna-0585 answered

I solved it, thank you.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.