Question Locked

TEXT PARSING HELP PLEASE

0 helpful
5 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

Okay,

We are soon going to have a 2,600 page text file that reports configuration information on a cyber machine. We are going to have close to 500 machine reports for a total of 1.3 million pages on configuration data. I need a program specifically tailored to text parsing or recommendations on how to approach this. We are going to be importing this into a DB.

I have plenty of parsing experience through my engineering class, but not with this much data and not in a non-development environment.

Any help is sincerely appreciated.

5 replies

by older mycroft

So much so that users sometimes become worried more by the page-count than by the content, which is responsible for the filesize.

A standard printed Bible has 687 pages of body copy, printed in a particularly tiny typeface - but the filesize is relatively small at SIX MEGABYTES - that's all, just 6MB.

So when you start considering your upcoming 2,600 page text file, just think of it as 4 Bibles. I doubt if your text file will be printed in 6pt type. ;)

I'll leave you to estimate the filesize.

by winthrop.polk

Filesize is about 8MB. If I were to export all the relevant information to another text file, it would probably be less than 1MB. The problem is that it isn't really formated for import export (i.e. demiliters are not constant. I need to have a parser that is very flexible. For example, I will need to do the following types of psydocode:

1. Find "Operating System", store info after ":"

2. Find all "TCP" or "UDP" and store associated info

3.etc

by slayer_

Grab a line of text, or all if you need, split them by the delimiter, then sort through them. I wrote a function in VB6 for this if you want.

Public Function GetVarFromFile(ByVal iVarIndex_Base0 As Integer, Optional bVerbose As Boolean) As Variant
On Error GoTo FileError
Dim slineTemp As String
Dim sArr() As String
Dim varr As Variant
Dim dTemp As Double
Dim d As Double
Dim dVarCount As Double
If ReadAccessAllowed = True Then
If LineSaftey = True Then
'Line Input #iFileNumber, sLine
' If Left(sline, 1) = Chr(34) And Right(sline, 1) = Chr(34) Then
' slineTemp = Mid(sline, 2, Len(sline) - 2)
' Else
' slineTemp = sline
' End If

'Determine the array size
d = 0
dVarCount = 0
ReDim sArr(0 To dVarCount) As String

'check the left of the line, if its a quote, the first var is a string
'otherwise, first var is a number

For d = 1 To Len(sLine)
If Mid(sLine, d, 1) = Chr(34) Then
ReDim Preserve sArr(0 To dVarCount) As String
dTemp = InStr(d + 1, sLine, Chr(34))
If dTemp = 0 Then
sArr(dVarCount) = Mid(sLine, d)
Else
sArr(dVarCount) = Mid(sLine, d + 1, dTemp - d - 1)
End If

d = InStr(InStr(d + 1, sLine, Chr(34)), sLine, ",")
dVarCount = dVarCount + 1
Else 'means its number or boolean
ReDim Preserve sArr(0 To dVarCount) As String
dTemp = InStr(d + 1, sLine, ",")
If dTemp = 0 Then
sArr(dVarCount) = Mid(sLine, d)
Else
sArr(dVarCount) = Mid(sLine, d, dTemp - d)
End If
d = InStr(d + 1, sLine, ",")
dVarCount = dVarCount + 1
End If
If d = 0 Then 'means no more
d = Len(sLine)
End If
Next d

slineTemp = sArr(iVarIndex_Base0)
varr = CVar(slineTemp)
Select Case VBA.VarType(varr)
Case vbBoolean
GetVarFromFile = CBool(varr(iVarIndex_Base0))
' If vArr(iVarIndex_Base0) = "#TRUE#" Then
' GetVarFromFile = True
' Else
' GetVarFromFile = False
' End If
' Case vbInteger
' GetVarFromFile = CInt(varr)
' Case vbDouble
' GetVarFromFile = CDbl(varr)
' Case vbSingle
' GetVarFromFile = CSng(varr)
' Case vbLong
' GetVarFromFile = CLng(varr)
Case vbString
If VBA.IsNumeric(varr) = True Then
GetVarFromFile = CDbl(varr)
Else
If Left(CStr(varr), 1) = Chr(34) And Right(CStr(varr), 1) = Chr(34) Then
GetVarFromFile = Mid(CStr(varr), 2, Len(CStr(varr)) - 2)
Else
GetVarFromFile = CStr(varr)
End If
End If
Case Else 'what to do if not a known type
GetVarFromFile = varr
End Select

Else
Call Error("The first line has not been read", bVerbose)
End If
Else
Call Error("File not opened in read mode", bVerbose)
End If
Exit Function
FileError:
Call Error(Err.Description, bVerbose)
End Function

~~~~~~~~~~~~~~~~
This is part of a class I call my file manager. All written in VB6 for these sort of purposes. I however do not know what language you are using. Or what exactly you intend to parse with, and parse for.

This discussion is closed to new replies.