Discussion on:

Parse and process HTML with WebBrowser

Using the WebBrowser control worked well for me, ut it might not be ideal for everyone since it loads all of IE in the process. If you have a different way of doing this, I'd love to hear about it!

J.Ja

Justin, I am wondering why regex is not a good way to do this. I'm not disagreeing, just looking to improve.

I've been doing the following.
1. Download web page with webclient.
2. HTML goes into string.
3. Run HTML through regex class to get rid of tags.
4. Find my matches with regex.

Any input is appreciated, is it slow, are regexs inflexible? They are a pain to write but there are some regex developers that help. Just looking to improve. Thanks!

Eric -

Good question. Here are the problems with using regex:

* Writing a regex that covers all the basics is a pain in the neck.

* People often write HTML in a way that defies logic and even the HTML spec itself, but continues to work in browsers; I'd rather let Microsoft (or Mozilla, or whoever) worry about those parsing rules that try to handle them all!

* Difficult to extract attributes with regex.

By the by, wondering if you are the same Eric Lamey that I went to high school (JMHS) with?

J.Ja

These are the subroutines and some comment sections it should work with some tweaking. The delay and check to see page done sub routines were the keys to knowing that the page had loaded. Just past into VB module in a new spreadsheet and have fun the URLs are from years ago and you need to put in your own

Dim MyBrowser As SHDocVw.InternetExplorer
Dim webpage As HTMLDocument
'Dim B As MSHTML.HTMLBody
Dim B As MSHTML.HTMLFrameElement


Dim browser2 As MSHTML.HTMLDocument

Dim Z As MSHTML.HTMLGenericElement

Dim Wtime As Integer


Dim AUTOLOG As Hyperlink
Dim A As MSHTML.HTMLAnchorElement



Sub GetPage()


'Obtain Web Information for processing
'Paste hyperlink in worksheet for autologin
'With Worksheets(1)
' .Hyperlinks.Add Anchor:=.Range("Z1"), _
' Address:="http://www.neco.navy.mil/biz_opps/search_edi.cfm", _
' ScreenTip:="NECO Web Site Open", _
' TextToDisplay:="NECO"
' 'assign hyperlink to variable autolog
' Set AUTOLOG = Range("Z1").Hyperlinks(1)
' Columns("A:A").ColumnWidth = 20
' Columns("B:B").ColumnWidth = 11
'End With

'Open web page
GetWeb
' Delay for remote server error (document needs time to fully load)
' Replace all web delay cycles with Browser event download complete

Wtime = 3
Delay


'Do Until Left(peekat2, 4) = "Done"
'peekat2 = MyBrowser.StatusText
'Loop


capture
'*******
End Sub

Sub capture()
pageaddress = LocationURL
Set webpage = MyBrowser.Document
For Each A In webpage.all
Count = Count + 1
'If A.innerHTML = "LINK" Then
Range("A" & Count) = A.tagName
Range("B" & Count) = A.innerHTML
Range("C" & Count) = A.innerText
Range("D" & Count) = A.href
Range("E" & Count) = A.ID



'For Each B In webpage.all
'counter = counter + 1


' Range("F" & Count) = B.tagName

' Range("G" & Count) = B.outerHTML
' Range("H" & Count) = B.innerHTML
' 'Range("I" & Count) = B.Link
' Range("J" & Count) = B.childNodes
' 'Range("K" & Count) = B.children
' Range("L" & Count) = B.childNodes
' Range("M" & Count) = B.Document


' Next B
' End If


Next A

'Set browser2 = MyBrowser.Document
'look = browser2.frames.Length
'For p = 1 To look
'Set webpage = browser2.frames(p)

'Range("A" & Count) = ("FRAME # " & p)
'Count = Count + 1
' For Each A In webpage.all



' Count = Count + 1
' 'If A.innerHTML = "LINK" Then
' Range("A" & Count) = A.tagName
' Range("B" & Count) = A.innerHTML
' Range("C" & Count) = A.innerText
' Range("D" & Count) = A.href
' Range("E" & Count) = A.ID
'Next A
'Next p



End Sub

Private Sub MyBrowser_DocumentComplete()
Dim pDisp As Double
Dim objDoc As HTMLDocument
Set MyBrowser = New SHDocVw.InternetExplorer

With MyBrowser
.Navigate URL:="http://www.alittlesomethingcatering.com"
.Top = 50 'set the browser in the top
.Left = 100 'left of the user's screen
.StatusBar = True 'display the status bar
.Visible = True
End With

pDisp = MyBrowser.HWND
MsgBox pDisp & " " & MyBrowser.Name

Set objDoc = MyBrowser.Document

' Display all of the HTML for the active Web page.


MsgBox "Entire HTML: " & vbCrLf & vbCrLf & objDoc.activeElement

With objDoc.body
MsgBox " outer HTML: " & vbCrLf & vbCrLf & .outerHTML
MsgBox " inner HTML: " & vbCrLf & vbCrLf & .innerHTML
End With


End Sub


Private Sub GetWeb()
' both statements work to open new explorer window
Set MyBrowser = New SHDocVw.InternetExplorer
'Set MyBrowser = CreateObject("InternetExplorer.Application")


With MyBrowser
'SiteAddress = "http://phoenix.ilsmart.com/FormsLogin.asp?/subscriber/default.asp"
SiteAddress = InputBox("Enter Url after the www.", "URL w/o header and .com trailer")
'ThePage = ("http://www." & SiteAddress & ".com")
.Navigate URL:=SiteAddress

.Top = 20 'set the browser in the top
.Left = 50 'left of the user's screen
.StatusBar = True 'display the status bar
.Visible = True
.Width = 500
.Height = 500



End With


Do Until Left(peekat2, 4) = "Done"
peekat2 = MyBrowser.StatusText
Loop
If peekat2 "Done" Then MsgBox ("Page didn't Load")
peekat2 = ""

End Sub

Function Delay()
newHour = Hour(Now())
newMinute = Minute(Now())
newSecond = Second(Now()) + Wtime
waitTime = TimeSerial(newHour, newMinute, newSecond)
Application.Wait waitTime
End Function

I've used HTML Agility Pack (http://www.codeplex.com/htmlagilitypack) with success.

I've used that as well, on another project, to rapidly strip a page of HTML where using WebBrowser would not be appropriate.

J.Ja

I had created the pages with my old xml writer (not MS-XMLWriter). I found a few problems on it, but by and large I got everything quite well. I had the ability to get any attribute of any element. All the content too.

I still have the stuff set up in case I need to crawl my old site (or any other site) again.

... with the way the average person writes HTML.

J.Ja

This all worked great for me until....

GRRRRRRR!

frame cross domain security issues. I never could get past this no matter what I did to security settings.

The web browser control DOES provide a nice dom model, which works on most sites. Frames with different domains???

If any of you have a solution to this, I am still interested in pursuing...

Using IE's COM Objects makes it child's play to extract anything from an HTML page, including navigating links to other pages. Plus, you don't have to waste disk space for the .NET framework, don't have to worry about Big Brother M$ watching over your shoulder.

http://www.autoitscript.com/

The AutoIt items looks interesting in and of itself, but your arguments in favor of it don't really hold much water. The .NET WebBrowser control is the IE COM object wrapped in .NET and there are methods to access it fully in COM to. Wasting disk space fot .NET? Hardly relevant, seeing as every Windows PC has .NET to begin with. And the Big Brother comment... not sure what you mean by that, but neither IE nor .NET report anything back to Microsoft...

J.Ja

I find the AutoIt COM interface much easier and intuitive than any other and the fact that it 'compiles' into a relatively small, standalone executable is another advantage.

My PCs do NOT have .Net on them. I refuse to have anything to do with it. I have yet to find anything I can't do in straight C or, if in a hurry, with AutoIt.

How do you KNOW that the framework doesn't call home? You can't even install it without allowing it to connect to the Internet.

I would be incredibly shocked if you have Windows PCs without at least one version of the .NET framework. For one thing, .NET comes with all post-XP Windows versions (http://blogs.msdn.com/astebner/archive/2007/03/14/mailbag-what-version-of-the-net-framework-is-included-in-what-version-of-the-os.aspx). Secondly, huge swaths of applications install it, from printer drivers to little utilities.

In terms of "calling home", you absolutely CAN install it without being connected to the Internet. You just need to get the big package which doesn't download anything. Furthermore, the .NET Framework is something that anti-Microsoft folks love to pick at. Beleive me, if it was "calling home", someone would have noticed by now and there would have been a massive outrage over it. If you have proof that .NET "calls home" I would love to see it, because I'll be the first to blast Microsoft for that. I find it very ironic that you are afraid of .NET "calling home" while using Windows (which *does* "call home") and IE.

I'm going to guess that you are also opposed to the Java platform for the same reasons. Between no Java and no .NET, there are an awful lot of apps that won't install on your system. I really wonder what the advantage there is in a system that you are ignoring a huge number of applications on, due to a prejudice based on speculation.

J.Ja

It wouldn't be my first choice for anything but I always install it (the non-M$ version) on a PC.

Although I honestly don't like Microsoft's attitude, I have made a good living supporting it's products. My main objection is that .net enourages poor programming practices. I just don't understand why the framework takes up Hundreds of MBs when a C (even C++, even Java!) program that does exactly the same thing would only be a few MBs in size. Even with TB drives and multi-GB RAM, that seems wasteful to me and indicative of sloppy coding and sloppy reasoning.

But isn't the Framework more properly compared with the JRE? Both are large, diskwise.

If you mean the size of .NET applications, I can't comment, as I don't use them or have the Framework installed, either.

I thought the article title was Parse and Process HTML?!!! So, where's the code?? All you say here is that it can be done. Very helpful. What a waste of time!!!!!

You aren't able to find the WebBrowser control?

Let me take care of that for you.
http://msdn.microsoft.com/en-us/library/aa752041(VS.85).aspx

http://msdn.microsoft.com/en-us/library/w290k23d.aspx

http://msdn.microsoft.com/en-us/library/system.windows.forms.webbrowser.aspx

I can't imagine why you'd want the specific parsing code for J.J.'s specific project, if that's what you mean.

Now you've wasted my time. Satisfied?

Has anyone tried TextPipe Pro for this purpose?

Which WebBrowser are you talking about? System.Windows.Forms.WebBrowser or System.Windows.Controls.WebBrowser? They are quite different.

... System.Windows.Controls.WebBrowser doesn't appear to have a ReadyState property...

More than one way to skin a cat, it would seem...

WebBrowser.DocumentCompleted Event @ http://msdn.microsoft.com/en-us/library/system.windows.forms.webbrowser.documentcompleted.aspx
"Handle the DocumentCompleted event to receive notification when the new document finishes loading. When the DocumentCompleted event occurs, the new document is fully loaded, which means you can access its contents through the Document, DocumentText, or DocumentStream property."

WebBrowserReadyState Enumeration @ http://msdn.microsoft.com/en-us/library/system.windows.forms.webbrowserreadystate.aspx
"Complete The control has finished loading the new document and all its contents"

I didn't find the code, is it a joke?
pedrito rod
mr85@servidor.unam.mx

shouldve use html agility pack. no need to re-invent the wheel this has been done many a time and is called screen scrapping.. nice approach though. httprequest class would have returned the response / web page markup u required as well...