General discussion

Locked

automate download of plain text file

By easyprit ·
I need a way to create a summary of 10 or so log files that are updated daily on the local interoffice website. The text files have a static name on the site.
The constraints that make this a tougher problem is that I can't buy software (being a minion not a boss) and can't use IIS. Hopefully open source stuff like free perl or even shell scripts to help me do this would be awesome. The data that is parsed gets written to an excel file. Is there a way to write a macro to either parse the webpage or download the files and then parse them? So VB macro or perl script or something else that helps...

This conversation is currently closed to new comments.

10 total posts (Page 1 of 1)  
| Thread display: Collapse - | Expand +

All Comments

Collapse -

by jc2it In reply to automate download of pla ...

It sounds like you want to do this on a computer running the Windows OS. That will be more difficult because the DOS prompt doesn't have the tools that a Unix Shell has.

This is a difficult question to answer because we do not know what the log files contain and what you are pulling out of them to make your summary.

A scripting language like perl would be ideal, but you will probably need to be administrator to run or install it.

Lets assume you only want to pull out the previous day's entrys to the log file. You will need a method for finding the date and time the log entry was made. You can write a perl script to do this, but it might be time consuming to write a new script for each different log file.

What would be easier would be to make a historical copy of the log file each day and then append the lines that are different the next day to the summary. Then delete the old copy and create a new one. You could do this easily with the diff command on most Unix computers. This would work on each log file regardless of date format. I believe you can use the fc command in dos to compare the files and then you could Concatenate them with the >> operator.

Excel is not a problem as you can open a comma or tab separated value file in that application.

The following example uses FC which puts extra info into the summary, but this could easily be removed with a perl script or an Excel macro.

So a MS-DOS batch file might look like this:

REM Batch file to compare logs and save
REM differances
REM Run once per day at midnight for daily report
REM Schedule with the DOS AT command
REM 11-4-2005

fc log1 C:\logtemp\log1 >> C:\logtemp\summary
fc log1 C:\logtemp\log2 >> C:\logtemp\summary
...
fc log1 C:\logtemp\log10 >> C:\logtemp\summary

REM in order for the above to work this must be
REM done before this batch file is first run
copy /Y log1 C:\logtemp\log1
copy /Y log2 C:\logtemp\log2
...
copy /Y log10 C:\logtemp\log10

Collapse -

by easyprit In reply to

The problem was I didn't ahve the files on hard disk, I needed a way to get them to a file so that I can do performance analysis on the numbers using excel. Turns out excel does support something that can be used to do this.

Collapse -

by sgt_shultz In reply to automate download of pla ...

i like to prototype this from the keyboard if possifble.
let say you have a drive mapped (U:) to webserver place with text files, know the file names, and are running windows xp
first we prototype:
Start/run/cmd <enter>
echo off
u:
copy file1.txt+file2.txt>summary.txt

the only thing you need is the copy command with the append switch between the filenames (the plus sign) and with the output redirected (the right arrow) to your new summary filename.
try it from the keyboard with test files on your local box.
see how it works
then map a drive and get it working on the production box
then put those lines in a batch file and schedule them in windows scheduler.
your mileage will vary depending on your os and tools and reason for doing this.
you could probably do it all from excel eventually, in one step.
why do you feel you need perl or a scripting language?

Collapse -

by sgt_shultz In reply to

boy that was sure smart of them to enable those smiley faces in a tech forum. sheesh.
here is something else you might like
Using FTP Batch Scripts
http://support.microsoft.com/default.aspx?scid=kb;en-us;96269

Collapse -

by easyprit In reply to

Batch scripts will help me with another automation. This will cut down my monday morning routine work where I back up the data to a ftp server. Although I want a way to automate "ftp>mput avgresp.txt" and then confirming... I can figure that out someother way.

Collapse -

by easyprit In reply to automate download of pla ...

all files are plain text files, are static, have the same name every day (eg http://xxxxx.xxx/reports/revcredit.temp2), and contain the timestamp, servername and jvm name for exceptions for only previous day seperated by spaces. It would have been immensly simple to do if files were on a network drive. One word: bureaucracy. So we don't have access other than the website
I can easily write a program to parse and do content manipulation. What I really need help with is a means to get this content off the http site.

Collapse -

by easyprit In reply to automate download of pla ...

Those batch scripts gave me an idea. I just mailed the admin who "owns" the log files, asking him to give me a username/password. Hopefully he'll help me out. In the mean while I'd appreciate it if you guys can find a way to do this easily over http as this program will have to be mobile enough to run on different files. It doesn't have to be comletely automated/scheduled. I actually welcome a mouse click before starting the whole process of downloading all the files.
I spent quite a bit of time over at the rfc websites and learnt more about other things than my problem.
Thanks again both you guys

Collapse -

by easyprit In reply to automate download of pla ...

I have written a code a while ago for blackberry device when I worked for them, that code I believe I used winsockets to connect and download new games and midis as soon as they were posted to the internal website. I polled the website every hour (since it's always on and no restrictions for usage for employees). Anyway is there a way I can do the same without a wireless device. I can write Java or C++ code and have my other script execute the program along with file names as parameters. I don't care about the footprint, and I don't care about the amount of code I have to write. I'd like to learn this for myself and so I'm willing to put in a weekend to develop something like this.

Collapse -

by easyprit In reply to automate download of pla ...

Solved it... Using excel to import the text files, writing the beautifully formatted data straight to the appropriate tabs. did I mention it works bee-uuu-tifully.
For those of you who come wandering this way looking for a solution, take a look at the WebFormatting Property in Visual Basic for MSExcel. Once you have the web contents on a spreadsheet, you can do whatever you want with the data including write it back to a text file locally. Here's the vbscript to import the data
Sub main()
testingThisBull "CPM.temp2", "Sheet1"
End Sub
Sub testingThisBull(FileName As String, SheetName As String)

Dim importFile As String
Sheets(SheetName).Select
importFile = "URL;http://esmweb.xxx/reports/" & FileName
Set shFirstQtr = Workbooks(1).Worksheets(1)
Set qtQtrResults = shFirstQtr.QueryTables.Add(Connection:=importFile, Destination:=shFirstQtr.Cells(1, 1))
qtQtrResults.Refresh

'Do all the parsing, writing to the files over here
Worksheets(1).QueryTables(1).CancelRefresh 'removes linking
'clean the spreadsheet and you're ready to call this again with another set of params
ActiveSheet.Range("A1", Cells(200, 5)).ClearContents
End Sub

Collapse -

by easyprit In reply to automate download of pla ...

This question was closed by the author

Back to Windows Forum
10 total posts (Page 1 of 1)  

Related Discussions

Related Forums