we use this kind of FTP in some applications. Besides editing by hand the FTP file and command file, you can have your program create a simple text file, name it as a batch file, have the program also create the command file.
On the first run log into the FTP site and don't do ANY gets. Instead do ls command to list the FTP site files (cd first on FTP site if you need to). lcd will determine where the ls will goto on your local drive (put a file name after ls so it stores locally)
quit FTP
read the directory of the FTP site, to determine if there are any files you want to process. We went as far as checking a database table of already-processed file names, so we'd skip downloading or processing anything we had already got from FTP site (files named with date in name rather than replaced). When you DO transfer a file, the LS text file contains the date the file was posted on the FTP site and you can capture this to log files if needed, i.e. date and time a customer posted file to FTP sites, handy to avoid blame if they were late, etc.
this way you don't need to process files every day or get them overwritten, instead any files not run will run whenever your prog runs.
After you've determined what files need to be run, write a second batch file that has same commands but writes a get to the command text file for each file you want to download.
Then your prog calls second time ftp,
gets the files
and does whatever you want to do with them.
At end of run if you do whatever you want to do successfully, write to database table that you have processed this file. that way next run won't run it again.
We generally had the FTP go to a local staging directory then have the program do a directory of this staging area, read thru it and ask user to select a file that needed processing. only files that had not been run show up. Also having a local staging area means you can just pop a file in that directory if you need to do a test run for the later process and know FTP works but not have to have the slowdown of FTP xfer of large files.
Sounds complicated but takes program only a few seconds and is good pattern for error free transfer.
If going the other way and doing puts, you can use the 'ls' to ensure the file actually made it up to the FTP site and branch to an error catch or trap if needed.
While the above was for production files at a company, I also use something similar to get my raw access log files from the web sites I created. This doesn't need the 'ls' as the file name is always the same on the website. However they reset the file periodically so I have it setup to load every hour. I have the command file created each time and append the date plus time to the download file.
Then I read thru this text file, generate a unique key for each record based on time and date of access and write to a web site access database, skipping duplicates I wrote previously, and steering robot accesses (based on 4 octets of the IP address, and if not present 1st 3 octets) from a lookup table compared to the ip in the record, and if present writing a robots stats file instead. I have to do this as not all robots identify themselves in the UA and progs like awstats or analog, etc don't separate them out.
