I’m trying to load a web page in Safari, and check whether it is fully loaded or not before getting its source. I do this by checking whether the text of the applicable document includes either text I know will be at the bottom of the page, or text that indicates a bad page was loaded, in which case I’m done. The problem is, the variable that I’m loading the text of the document into (“pageText”) apparently is undefined randomly in the behaviour (I load many of these pages sequentially). So I’m thinking it’s a timing issue - sometimes it works, sometimes it doesn’t, and if I add a delay in then it seems to work more reliably. I really want to go through them as quickly as possible, so the delay isn’t good. Here are the relevant two sections. The variables siteNumber, endOfChapterText, and noChapterText are all globals, and refer to what text I’m looking for to indicate the end of a page or no page.
I certainly won’t say no to suggestions on better ways to do this, but I’d like to figure out what’s going on here, too, for my future reference. Thanks for your time and help!
-Joe
-- given a URL, loads that page in Safari and makes sure that it is fully loaded before returning true if a chapter and false if not
on FullyLoadSafariURL(theURL)
tell application "Safari"
activate
make new document at beginning of documents
set URL of document 1 to theURL
end tell
return WaitForSafariText(item siteNumber of endOfChapterText, item siteNumber of noChapterText)
end FullyLoadSafariURL
-- waits for the front Safari page text to contain the given text
on WaitForSafariText(goodText, badText)
set pageText to ""
delay 2 -- wait to make sure pageText is defined to avoid run-time error
repeat -- not infinite because return from within this repeat
tell application "Safari" to set pageText to text of document 1
if pageText is not "" then
if (FindText(goodText, pageText)) then
return true
else if (FindText(badText, pageText)) then
return false
end if
end if
end repeat
end WaitForSafariText
-- returns true if theText is contained within sourceText
on FindText(FindText, sourceText)
set OldDelims to AppleScript's text item delimiters
set AppleScript's text item delimiters to FindText
set textItems to text items of sourceText
set AppleScript's text item delimiters to OldDelims
if ((count of textItems) is greater than 1) then
return true
else
return false
end if
end FindText
If you can work with HTML source, Talix, then Kim’s curl suggestion might be a good way to go. Nevertheless, in case you need text - or you’d…
The problem is that a 2-second delay may be long enough to load some pages - but not in every case. For this type of operation, you ideally want a dynamic delay, rather than a fixed one. That way, your script waits for just as long as it takes for the relevant conditions to be satisfied before continuing.
Near the beginning of your WaitForSafariText handler, your script sets the value of the variable pageText to an empty string:
set pageText to ""
However, a little later, Safari is told to set the value of pageText to the text of document 1:
tell application "Safari" to set pageText to text of document 1
And if the document doesn’t yet contain any text, the variable pageText will be left undefined - regardless of whether or not it had a value previously.
Even if we first check for the existence of a document’s text, it isn’t necessarily a reliable indicator that loading of the page is complete - since the text might still be only partially loaded.
So what method might be more robust?
I’ve found monitoring the document’s name to be a pretty effective approach:
to check_text for some_text at some_url
tell application "Safari"
make new document with properties {URL:some_url}
tell document 1
repeat while name starts with "Untitled" or name starts with "Loading "
delay 0.2
end repeat
some_text is in its text
end tell
end tell
end check_text
check_text for "RSS (active topics)" at "http://bbs.applescript.net/viewforum.php?id=2"
Using “curl” works loads better - thank you both very much! And I am happy to know that the reason the variable is undefined is that previous settings of it don’t carry through if the new value isn’t set.
I do actually use command-line scripts in the rest of my Applescript, and I’ve been just opening up the Terminal and using the “do script x” command. The command I’m using is “textutil”, which is very handy-dandy for combining text together. My next question would be thus: textutil doesn’t seem to work if you’re not in the folder with the files you’re working with (i.e., you can’t specify full paths to each file) - do you know of a way to use the “do shell script” instead of opening a Terminal window, if the script you’re running is dependant on the state from a previous script (e.g., a “cd” command)?
or you can use a logical AND (meaning that if the command fails then it will stop processing say if the documents folder didn’t exist for some reason or what not.
I’m hoping you who helped me are still subscribed to this topic… If not I’ll start a new one, as this is technically a different problem.
So I’m using the “curl” script now, but I’ve run into a problem for sites that require cookies to log in to view some pages. Is there any way to still use the “curl” script while being able to view those kinds of pages?
With the correct password, obviously. The ideal solution is to have the cookie read out of a file, but as far as I can tell Safari stores its cookies in a .plist file (in ~/Library/Cookies folder), which curl doesn’t know how to read. Feel free to tell me I’m wrong on that. But this should work as long as the information doesn’t change.
And I am happy to know that the reason the variable is undefined is that previous settings of it don’t carry through if the new value isn’t set.
If not I’ll start a new one, as this is technically a different problem.
But this should work as long as the information doesn’t change.