I'm trying to run a pipeline (built via Python SDK), and I want the repo to be snapshotted, because the different modules are called inside the pipeline scripts. The source folder I pass to the pipeline has the following structure:
repo
- src
- data
- script_step1.py
- script_step2.py
- script_step3.py
- .gitignore
I want the data folder to be ignored in the snapshot, so in my gitignore file I have al line with written /data/*, but when I try to run the pipeline I get an error that tells me that the snapshot is too big (I checked the size of the other stuff except for data folder and it is very little)
I'm not understanding why that happens.
Thanks a lot in advance
