store url contents into a filemaker field

to all that can help

while it is more or less easy to create a field “URL” that can be used to later open the url in a browser I don’t know how to get the cotents (source code) of the so opened page into a field in filemaker.

how to download the contents of a specific URL into a field of filemaker ?
( is it possible to do it without that the browser opens )?

any help welcome

thx

linus

:oops:

This script should get you started. You need to plug your target URL into the first line.

set targetURL to "http://macscripter.net/index.html" -- <-- Change this.
set desktopFol to path to desktop
set fileDestination to (desktopFol as text) & "htmlSource"

tell application "URL Access Scripting"
	set downloadedSource to download targetURL to file fileDestination replacing yes with progress
end tell

set sourceText to read downloadedSource

The script should:

  1. Download the source to a file named ‘htmlSource’ and place it on the desktop. It will overwrite a file with the same name but this can be changed easily.

  2. Read the contents of the file into a variable named ‘sourceText’.

All you should need to do is add the command needed to place the source into the database. Something along the lines of:

tell application "FileMaker Pro"
set cell "Source" to sourceText -- <-- Replace "Source" with your field name
end tell

You can check the dictionary of ‘URL Access Scripting’ for more details about downloading the file. Of course the script should include some error checking but I’ll leave that to you. :wink:

URL Access Scripting works, but it needs to launch, and it has had som bugs (still cannot POST a form, although dictionary claims it can). You can use the command-line tool curl to do this easily, without creating a temporary file:

If your database of the frontmost file in FileMaker, the URL field is called
URL, and the Source field is called Source:


tell application "FileMaker Pro"
	set theURL to cell "URL" of current record of document 1

-- get the source as a string
set sourceHTML to (do shell script "curl " & quoted form of theURL & " | vis")
-- piping the curl result into vis strips out characters that would cause an error

	set cell "Source" of current record of document 1 to sourceHTML
end tell

If you are calling this from within a FileMaker Perform AppleScript script step, you don’t need the ‘tell’ and ‘end tell’ lines.

thx a lot, it is working and I’m a big step further

it seems that you are a professional so you might have also an answer to the following quesiton.

can I use a field of filemaker to put it into a search field of a fix url and then retrieve the result of the web-page (source code) into the source field ?

thx

linus

:rolleyes:

Whew! It took me a minute to read that and get what you wanted! :slight_smile:

I think you are asking if you can run a search using some web-based search engine from FileMaker. The answer is yes, although some are easier than others.

If you wanted to search Yahoo from a FileMaker field called WebSearch, you’d do the following:


set searchBase to "http://search.yahoo.com/bin/search?p="
set theSearchTerms to cell "WebSearch" of current record of document 1
 set theURL to searchBase & theSearchTerms

-- get the source as a string
set sourceHTML to (do shell script "curl " & quoted form of theURL & " | vis")
set cell "Source" of current record of document 1 to sourceHTML

A few important points:

  1. You need to find out what the base URL you need.
  2. Some sites don’t allow this (although it is hard to stop). Google recently made it difficult (there may be a way around, but you’re violating their “terms of service”) to download search results directly, without a web browser. Curl can mimic web browsers in sophisticated ways, but the Terms of Service are Google’s prerogative.
  3. You may need to encode multiple search terms in URL format (i.e. spaces become %20), although it often will work without this.
  4. The search engine you wish to use may use the POST method, rather than GET (using GET means the search terms are in the URL, using POST means they are attached apart from the URL). If the engine uses POST, you’ll have to put your search terms in an option to curl: --data (read the curl man pages for more detail, an look up “form post method” on the web)

Linus4200 wrote to me off-board, since he had some specific questions about his case. I’m not going to include the URL he is trying to reach, but he can add the extra info I sent him, if he likes. Amazon’s search box on their front page also uses POST, if you’d like to see an example.

His case is difficult, as it requires the POST method, not just GET (which means he cannot use URL Access Scripting on OS X, since POST broke in URLAS for OS X). Here’s the overview I sent him:

Hmm,

Yours is not the easy case, in case you had not figured that out yet. :slight_smile:

Maybe I should give a little background, so you know the framework.

HTTP is just a simple way for a computer to say “Hey, give me some information back”, as I’m sure you know. One method is to just send a URL to a server, and it then sends a bunch of ASCII (or binary) back to your computer, which is waiting for the response. Many forms work this way. The form is specified in the HTML source code of the page to use the method GET, which means that your web browser will take the info you typed into the fields (and any hidden INPUT fields in the source code), and tack that onto the end of the ACTION url for that form (ACTION is also specified in the HTML source code of the page). Then, the server gets all the fields you filled out (plus those hidden ones), determines what you want, and sends back a page (or other data in some cases).

However, sometimes there is just too much information to fit in an URL (or the website doesn’t want data sent in URLs). So, there is the POST method, which means that your request includes the fields as a separate part of your request to the web server.

The site you’re trying to access uses POST. Take a look at the source code of the page where your form is. Find the form tag you want (there can be more than one). See what the ACTION parameter equals (this may be a relative URL, which means you need to look at the URL of the page your are on). Also, see if the method is GET or POST. If it is POST, as in our example, you’ll need to attach the data separately from the URL. The command-line utility curl can do this using the --data option. URL Access Scripting is SUPPOSED to be able to attach form POST data, but that is broken in OS X (works in OS 9). That is the main reason I’ve given up on URLAS.

Look for the tags to get the fields’ names. You may also need tags, if a popup is used, or as well. Most will be tags, with a type of hidden, text, radio, or checkbox. The --data options needs a string (with single quotes around the string) that looks like this: ‘SOMEFIELDNAME=SOMETHING&ANOTHERNAME=SomethingElse’ and so on.

So, your command to curl may look like this:

curl 'http://www.SOMESITE.com/their_cgi' --data 'firstname=Dan&lastname=Shockley'

So, for Amazon.com, to search for Stephenson in All Products, you’d use (notice I did not include the big numeric part in the URL, since that was specific to my visit):

curl 'http://www.amazon.com/exec/obidos/search-handle-form/' --data 'url=index=aps&field-keywords=Stephenson'

Now, some caveats: you need to make sure you specify all the required hidden fields. Another issue is that you may get back a response saying that the info you want is at another URL. This can happen when the web server gets your request, and, instead of feeding it back immediately, caches it in a local file that you can retrieve with a simple GET request. When it responds this way, your web browser just goes there, so it looks to you as if it went directly there immediately. What really happened is that the web browser was smart enough to say “Oh, OK, the web server is telling me that what I want is somewhere else, so I’ll just ask for that now instead” and you see the desired information. To tell curl to mimic this behavior, you need to add the --location option. So now your curl command would look like this (Amazon.com does not need this):

curl 'http://www.SOMESITE.com/their_cgi' --data 'firstname=Dan&lastname=Shockley' --location

Now you’ll get the information you want.

Oh, I should mention that to make it much easier to see what happens when you post a form in your web browser, you can get the utility tcpflow. http://www.circlemud.org/~jelson/software/tcpflow/

To watch your Ethernet connection, use: sudo tcpflow -c -i en0
(you’ll need your Admin password)

To watch your Airport connection, use: sudo tcpflow -c -i en1
(you’ll need your Admin password)

Here’s a summary I posted on my own site (which is pretty sloppy right now): http://danshockley.com/formposting.html