Download and Delete from Source every 5 minutes (Ubuntu)
-
Hey all.
As some of you may have noticed I've been asking a bunch of linux questions, trying to move jobs from windows and simply learn....so here's my next one.
I currently have this software I use on windows that will download files from a server, then delete the source, every 5 minutes (however it could take hours to finish)
Now from quickly googling I believe I would be doing a Cron job to...rsync?
Is that my best option?How would you Ubuntu experts handle this.
Thanks!
-
Cron would be the way to do this, absolutely.
You could schedule it for 5 minutes after the download completes to delete the files.
Might I ask why you would download a file so rapidly and delete it so soon? What processing are you trying to run on this file?
-
Its just moving files from location 1 to location 2
So essentially whenever a file appears (my random number was 5 minutes) to download the file from source and delete it.
Remote source moves file into location
Ubuntu server see's a new file
Downloads File
Deletes
RepeatBut as its downloading the files, new ones could appear so it couldnt just do a blanket delete, would have to be specific
-
So cron could do this, but I've never seen it done directly with Cron.
What you would do is have a script that gets run via cron say, ever 2 minutes (cron is a task scheduler). That script would check for any new files, copy them down, and delete the source.
I'm not sure if you could schedule cron to check for changes and run on a "change schedule"
-
@DustinB3403
Would there be a way to script it in that it checks, if it sees the file the same (source vs remote) that it deleted?Just trying to think of the best way to do it so it doesnt just delete the whole folder without getting them all)
-
This sounds like a good use case for Rsync. Setup a cron job to run rsync every X amount of minutes.
-
Well you'd have to have the script check for any new files, and compare the source vs the destination.
So you could have it check every 30 seconds or whatever schedule you'd want.
-
@Jstear said in Download and Delete from Source every 5 minutes (Ubuntu):
This sounds like a good use case for Rsync. Setup a cron job to run rsync every X amount of minutes.
rsync would work perfectly for this as well, cron is for scheduled tasks.
-
No problem using rsync with a ftp I assume?
-
There shouldn't be any issues.
-
Alright think I've got it
So mount my drives with fstab and then just a simple
rsync --remove-source-files -options /path/to/src/ /path/to/dest
-
I believe the function you want is --delete-after . But otherwise, yeah that should work.Nope ignore me remove-source-files would be better.
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
@DustinB3403
Would there be a way to script it in that it checks, if it sees the file the same (source vs remote) that it deleted?Yes, that would not be too hard. MD5 hash stored in a text file to compare against would work. But you need an MD5 on the remote site to be able to compare against. Otherwise, it needs to download it to check the hash.
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
No problem using rsync with a ftp I assume?
Rsync and FTP are different protocols. The term "using rsync with FTP" is meaningless. Imagine saying "using email with a phone call." What does that even mean?
Rsync and FTP are competing options. You can only use the technologies that the site you are downloading from provides. If they only offer FTP, you can only use FTP, it's that simple.
-
Let's start with the source, what is the actual source of this data. And why do you need it every five minutes? This is a weird task and I think we need to understand the real goals to really know what is and isn't a reasonable approach.
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
Alright think I've got it
So mount my drives with fstab and then just a simple
rsync --remove-source-files -options /path/to/src/ /path/to/dest
Seems that this would be a good option.
-
I'm trying to mount the drive under /etc/fstab but I cant seem to get it to work.
I have my local windows drive mounted, just cant get the remote Ubuntu drive mounted.
I'm assuming my problem child will either be firewall or port, does anyone know what port mounting the drive takes on Ubuntu?
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
I'm trying to mount the drive under /etc/fstab but I cant seem to get it to work.
I have my local windows drive mounted, just cant get the remote Ubuntu drive mounted.
I'm assuming my problem child will either be firewall or port, does anyone know what port mounting the drive takes on Ubuntu?
What is the mounting protocol in question?
-
Trying to mount using /etc/fstab
So I'm doing...
//XXX.XXX.XXX.XXX/var/www/completed /mnt/ftp cifs username=root,password=PASSWORD,iocharset=utf8,sec=ntlm,dir_mode=0777,file_mode=0777 0 0
but I'm getting error code 115
So I'm wondering if there's a certain port that needs to be opened up, or a typical thing that has to be done when connecting to remote computers (both ubuntu 14.04)
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
Trying to mount using /etc/fstab
So I'm doing...
//XXX.XXX.XXX.XXX/var/www/completed /mnt/ftp cifs username=root,password=PASSWORD,iocharset=utf8,sec=ntlm,dir_mode=0777,file_mode=0777 0 0
but I'm getting error code 115
So I'm wondering if there's a certain port that needs to be opened up, or a typical thing that has to be done when connecting to remote computers (both ubuntu 14.04)
Can you run the mount commands by hand and have it work?
