question

RohitKulkarni-6062 avatar image
0 Votes"
RohitKulkarni-6062 asked RohitKulkarni-6062 commented

S4 TABLE MERGE

Hello Team,

We have implemented the S4 Tables data into sql server tables .As of now we are truncating the data in SQL server and we are doing full load data from S4 to SQL Server. At the later stage the data in S4 will increase in millions of records at that stage the loading of Full Data into SQL server will be time consuming. So is there any method we can get only latest data from S4 and load only latest data into SQL server Tables. So that we can reduce the time consuming of the pipeline running.

Please advise

RK

azure-data-factory
· 6
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

<< Reply 1/2>>
Hello @RohitKulkarni-6062,
Thanks for the question and using MS Q&A platform.
As we understand the ask here is hyow to do an incrementalk load vs a full load , please do let us know if its not accurate.
When you refer S4 , I am assuming that the S/4 HANA please correct me if I am wrong .
<<contd below >>

0 Votes 0 ·

<< Reply 2/2>>
Looking that document I see that ADF do support query for SAP . https://docs.microsoft.com/en-us/azure/data-factory/connector-sap-hana?tabs=data-factory#sap-hana-as source , so i suggest this
1. Read the sink ( SQLin this case ) read the primary key and get the max value .You can get this using a Lookup activity
2. Create a query using a dynamic content with a filter > then the max value which you got from Step 1 . We will use this in the Source of copy activity .

Thanks
Himanshu

0 Votes 0 ·

As of now we are using the same technic we are loading the data from S4 Tables to SQL server Tables


Example :

S4 TABLE: 21-04-2022 : 10 Records
SQL SERVER :21-04-2022 : 10 Records is loaded

S4 TABLE: 22-04-2022 : 4 Records
SQL SERVER :22-04-2022 : In Current scenario I am truncating the current table data and loading again then it will load : 14 records

But i require only

21-04-2022 :data(10 rows) will be present in the table.
22-04-2022 :Need to load only todays data ( 4 rows ). Without truncate the previous date data

Total : 10+ 4 : 14 rows

Day to Day need to upload only the current days data into the tables without truncate the previous date data


Thanks

RK




0 Votes 0 ·
Show more comments

But We are using the source as S4..We can't get Max value from the S4 tables

0 Votes 0 ·

0 Answers