Coercing curl "file list" into applescript list

Greetings. I am trying to run a shell script (using curl) to get a file list from an ftp server. I can easily get the file list, but once I have it it is quite difficult to manipulate the data in the file list. It’s formatted text, and I can not find a way to get the formatting OUT, so I can coerce it to a string, and then a simple list. I can read it as text, and use text references like ‘paragraph’, ‘word’, ’ character’, etc, but there should be a better way to evaluate each record, as using text references is incredibly inefficient and error-prone.

Here is a sample of what the text (var ‘curlResults’) looks like when pasted straight into a text field…

For example, if I use…

word -1 of paragraph 1 of curlResults

… on the text above, I get “administration”…which is good and what I want. But if I use the same reference on paragraph 3, I get “html”… which is BAD. I want to get “index.html” out of it, so I can make a nice list of all the files and directories. Likewise, I want to use the permissions information, the date, etc… all of the basic “important” file data as it might pertain to an ftp application, so I really want to use the entire file record list rather than just the file names list.

I would prefer to do something like…

set ftpList to (curlResults as list)
--> Some statements (like a repeat loop for each record) left out
set ftpPerm to item 1 of ftpList
set ftpDate to item -2 of ftpList
set ftpName to item -3 of ftpList

I have fiddled with many iterations of coersions to strings, text, list…have used all sorts of attempts at setting the delimiters and parsing the “list”…and have run out of things to try to find an answer. Just when I think I’ve successfully got the contents as a list, it spit’s out an error “NSNSUnknownKeyScriptError (7)”, which apple identifies as “an unidentified error occurred; indicates an error in the scripting support of a scriptable application, or of AppleScript Studio (or Cocoa) itself”.

Below is a bit of code reflecting what I think SHOULD work, but as I said it’s throwing errors and I’m getting frustrated…

		(* This works fine, and spits out the var "curlResults" identified above *)
		set theCurlString to contents of text field "curlCommand" of window "main"
		set theUserPass to ("-u " & curlUser & ":" & curlPass & " ") as string
		set theCommand to ("curl " & theUserPass & (quoted form of theCurlString))
		try
			set curlResults to (do shell script theCommand)
		on error errorMessage number errorNumber
			set contents of text field "curlResponse" of window "main" to (errorMessage & "(" & errorNumber & ")")
			set curlResults to ""
		end try
		
		(* Something like this works, but doesn't always yield the correct results *)
		set curlResultsList to {}
		
		set theCount to count paragraphs of curlResults
		repeat with theIndex from 1 to theCount
			set curlResultsList to (curlResultsList & (word -1 of paragraph theIndex of curlResults))
		end repeat

		(* This does not work. I want a comma-separated list of file records *)
		set oldDel to AppleScript's text item delimiters
		set AppleScript's text item delimiters to (ASCII character 13) --> or "(ASCII character 10)"
		set curlResultsList to (curlResults as string) as list
		set AppleScript's text item delimiters to oldDel

		--> Then I want to further process each record/line as a list here

Note: some of this code I rewrote by hand because my test code was a huge mess. If you cut and paste code I claim “works” and it does not, it’s probably my fault for not typing it in correctly…sorry. :slight_smile: As I said, I can not successfully make each record (or the entire “list” of records) into a true applescript list. Any help or solutions are greatly appreciated. Thanks,
j

I found this in the documentation. You can just change the delimiter to return.


set AppleScript's text item delimiters to {":"}
get last text item of "Hard Disk:CD Contents:Release Notes" 
returns the result "Release Notes" .


set AppleScript's text item delimiters to {return}
get last text item of "Hard Disk:CD Contents:" & return & Release Notes" 
returns the result "Release Notes" .

I didn’t test it, but i’m sure it’ll work.

If only it were that simple. :wink: If you check the character (the return), it’s actually a carriage return (ascii 13) not a new line. It doesn’t matter though, I’ve tried MANY different variations and haven’t found any way of getting applescript to recognize the contents as data which can be coerced to a list. I can’t even get it to separate the file list at the spaces or a letter ‘e’. None of it works. Apple’s 10.2 release notes page comments on a bug that was supposedly fixed in which some text could not be coerced to a list, but I do not know if that applies to this case.

Really what I need is someone who’s done this same thing, or someone who wants to sit and try some other ideas on a test project to figure out a way. Any thoughts are welcome.
j

why not pipe the records through one more phase: say perl!

set yourCommand to …
set myPattern to “^(.{10})sd+sd+ssitesd+s(w+sd+sd+)s(.*)$”
set myReplace to “$1:$2:$3”
set myPerlScript to “| perl -pe 's/” & myPattern & “/” &myReplace&“/;'”
do shell script yourCommand & myPerlScript

the regex (myPattern) may need some tweaking based on your needs when it’s done set text item delimeters to “:” and you’re there…

The thought of having to use perl for something so trivial makes me…want…to…puke. :o

Yeah, I thought of that too, but I didn’t even want to go there unless ABSOLUTELY necessary. I figured with all of applescript’s power it could hack through a bit of overgrown text and I could find the treasure without incident. But NOOOO, it had to be a hard thing I tried to figure out. :smiley:

I’ll go perl if I have to, but a simple coersion to a list would be so much simpler and prettier…no?

j

Just use a space as your TID:

Jon


[This script was automatically tagged for color coded syntax by Convert Script to Markup Code]

I wouldn’t have suggested it if you weren’t already calling out to use curl… and since you are calling out, perl is probably faster and tidier than jon8’s answer involving collecting text items… (no offense) :smiley: And besides, there are far worse things than a little perl golf in your methodology.

(alright, I’ll give you that the regexen sitting in the applescript isn’t very tidy… nor is my solution obfoo or golfy… but it’s fewer lines than the AS… :wink: )

While no offense was taken, to be fair, my code could be optimized and the result is a more useful record than a string that needs to be further parsed using the PERL GREP. Also, in your GREP string, since you’re putting this into AppleScript, you need to escape all of your backslashes. So, my optimized code (and there’s probably room for more improvement) and your properly escaped code look like:

Jon


[This script was automatically tagged for color coded syntax by Convert Script to Markup Code]

Thanks for your input, guys, but they didn’t work for me or didn’t do quite what I’m trying to achieve. Since I’m really a perl hacker and not a perl guru I didn’t really know what to change to make grep work for me. Any follow-ups with modifications that will streamline this are welcome.

I’m beginning to suspect that it was not my code at all, but a bug or ‘conflict’ with trying to place a list into a text view/scroll view, rather than a plain text field. Oftentimes, if I set the contents of a text field to my list, it would work, whereas the same code received errors when used with a text view. In my exploration I found that even MENTIONING a list in the presence of a text view gave me errors. ???!!! Unfortunately my code evolved so much in trying to figure out the problem that my original code I think may have worked is no longer alive. :cry:

Anyways…I have found a (crude) way to get the job done. It is certainly not pretty, and I am sure it can be done differently and more efficiently, but with all of the trouble I’ve had I’m going to use this until I’m handed a better way or am forced to find one. :wink:

The code below will ask for the ftp directory, user name, and password from application text fields, and then get the contents of that directory and display them NICELY in a table view. It took me many days to figure this out (a testament to my adolescence in applescript & perl, no doubt :lol:) but I hope this will help someone else at some point and perhaps it will show the full extent of what I’m trying to do.

		set theDataSource to data source of table view "curlDataContents" of scroll view "curlData" of window "main"
		set theCurlString to contents of text field "curlCommand" of window "main"
		set theUserPass to ("-u " & curlUser & ":" & curlPass & " ") as string
		set theCommand to ("curl " & theUserPass & (quoted form of theCurlString))
		set contents of text field "curlCommandIssued" of window "main" to theCommand  --> displays what's being sent to curl/shell
		
		try
			set curlResults to do shell script theCommand
		on error errorMessage number errorNumber
			set contents of text field "curlResponse" of window "main" to (errorMessage & "(" & errorNumber & ")")  --> tells what error curl threw
			set curlResults to ""
		end try

		set curlResults to (curlResults as text)
		set theCount to count paragraphs of curlResults

		repeat with theIndex from 1 to theCount
			set thePerlScript to ("$theVariable="" & (contents of paragraph theIndex of curlResults) & "";
(@theList)=split(/ /, $theVariable);
@newList=();
foreach $listItem (@theList) {if ($listItem ne ""){push (@newList,$listItem)};};
print "@newList";")

			set perlOutput to do shell script ("perl  -e " & quoted form of thePerlScript)
			
			set oldDelimiter to AppleScript's text item delimiters
			set AppleScript's text item delimiters to " "

			set resultsList to text items of (perlOutput as string) as list --set resultsList to items of (perlOutput as string) as list
			set theFile to (text item 9 of resultsList)
			set theFileDate to ((text item 6 of resultsList) & " " & (text item 7 of resultsList) & ", " & (text item 8 of resultsList))
			set theFilePerm to (text item 1 of resultsList)

			set AppleScript's text item delimiters to oldDelimiter
			
			tell theDataSource
				set theRow to make new data row at the end of the data rows of theDataSource
				set contents of data cell "file" of theRow to theFile
				set contents of data cell "date" of theRow to theFileDate
				set contents of data cell "perm" of theRow to theFilePerm
			end tell
		end repeat
		

After all of this I get a nice table with three columns… Folder_Or_File | Jan 1, 2003 | drwxr-x-r-x.

Well, I saw some junk as I reread my post, so I’ve already cleaned up the code a bit. For some reason this is the only code I’ve got to work in 3 days, so either I’ve been drinking too much or all this 100º AZ heat is goin’ to my head (or both).

Take care,
j

That seems like so much work when this would appear to do the job (assuming the output looked like that you originally posted):

Jon


[This script was automatically tagged for color coded syntax by Convert Script to Markup Code]

Jon…that works NEARLY identical to mine. As I said, I know there are ways to streamline this, but there are two things that I see that immediately stand out.

The text is uniquely formatted so that it looks pretty when viewed in a plain text view. It is automatically padded with spaces to fill gaps so it lines up in fixed width fonts. Because of this, the space(s) between actual text values can be varied depending on digit size of the text items around them. For instance, consider the following two lines…

drwxr-xr-x 2 32092 site 4096 Jul 18 2003 cgi-bin
-rw-r--r-- 1 32092 site 1810 Aug 1 2003 index.html

What you don’t see here, because of the html formatting, is that there are actually multiple spaces between each text item that are inserted to fill the space of digits that aren’t there. Note the difference between Jul 18 and Aug 1 below. I have inserted @'s to show where the actual spaces are in the curl results. When the date is only a single digit, it places a space there to fill the gap. When splitting the text items this causes there to be another empty text item, which offsets the count of all subsequent items in the paragraph…

drwxr-xr-x@@@@2@32512@@@@site@@@@@@@@@4096@Jul@18@@2003@cgi-bin
-rw-r--r--@@@@1@32512@@@@site@@@@@@@@@1810@Aug@@1@@2003@index.html

Because there are all the extra spaces, the “nothing” in between each is technically a space so you must use something like…

set end of parsed_curlResults to {("" & (text item -1 of p)), ("" & (text items 18 thru 21 of p)), (text item 1 of p)}

…rather than the version of the same line you posted. Because of this, the first line returns as “Jul 18 2003”, but the second line only returns “Aug 1”, because it’s reading the blank value between the @@'s as an empty text item. As you can see, there are many extra ‘invisible’ spaces, so trying to read them by text item id could become a real problem, even if only one digit changes early in the string. Also, because many of the fields change with variations between servers, the alignment of the text items will change constantly, too.

So, from what I can tell the perl line is actually required. Perl makes quick work of turning the original record into a clean list separated by only one space between items. I don’t think that applescript can do it as quickly, and certainly not as cleanly as perl. And, as I said, I bet there is a way to make perl (using grep, map?, etc.) do this even more compactly…but I’m not that good. 8)

I didn’t realize until this post that the extra spaces were not there because of the transition to html, so sorry to misguide you. Perhaps this will even further clarify things.

The append code and the streamlining of the delimiters is a welcome change, and will find it’s way into my code eventually. Thanks again.

Cheers,
j

Ah. How about just removing the extra spaces first:

Jon


[This script was automatically tagged for color coded syntax by Convert Script to Markup Code]

Or how about this fun little combo solution:

Jon


[This script was automatically tagged for color coded syntax by Convert Script to Markup Code]

Now that’s what I was looking for! I knew there was an easy way, but my style is BRUTE FORCE, not clean and elegant. :rolleyes: I just couldn’t figure out how to do that without a huge mess of code. Thanks a bunch Jon…you have my respect.

As far as your “fun little combo” goes… I now feel like I know NOTHING about perl. :?

j