Lists from text file to TableView and back

Hi all,

I deleted my previous thread on this because of the poor description.

I have a data file in text format, space delimited, with return at eol.¨There are 1000 records in the file, and 15 fields in each line, all with numbers, some with decimals. There is a header line that labels the fields, I have trimmed out of the data.

I have successfully read in the data to theData, cleaned it up and converted the lines of data into a list of lists as follows:¨¨{{“item1”,“item2”,“item3”,“item4”,“item5”,“item6”},{“item1”,“item2”,“item3”,“item4”,“item5”,“item6”}}¨¨
-there are actually 15 items in each row’s list

Stuck on populating the tableview.¨

In the tutorials (http://macscripter.net/viewtopic.php?id=24756) I see there is a way to load the data into the Tableview from a list of lists as I now have, using a DataSource (described in the AS Studio tutorial) further saying that the table will actually build itself setting up the needed number of columns in addition to populating the rows. But I cannot seem to find the way to do it.¨

Here is the quote from the MS post:
" Beginning with AppleScript Studio 1.4, we can now set the content property of a table view to a list containing values, a list of lists containing values, or a list of records containing fields and values. Doing so will automatically create a data source (if it does not already exist) for the targeted table view, and insert and populate the appropriate number of data rows and columns, based on the data specified."

Is there a way to load a list of lists as content directly into a tableview in ASOC?

Many thanks!

The easiest way is to change your list of lists into a list of records, with the record labels matching the columns, and using bindings.

If you want to use a list of lists, you’ll need to declare a datasource for the table, and it must contain at least two handlers: numberOfRowsInTableView_ and tableView_objectValueForTableColumn_row_. See NSTableViewDataSource for details.

Even with a datasource, using a list of records will probably be easier.

Thanks Shane,

Using your recommendation, how would I add the data label and colon (i.e.- “day: item1”) to all 15,000 items without a major hit in performance?

If I could do that, I would think your Florida (free plug!) sample would make sense as a model for the data viewing and manipulation in my table view, but I have never added a prefix to 1000 lists of 15 items each lol without something going bang! One would think that the labels of the TableView could be somehow be prepended to each list item to form a record easily.

Also, once processed, I need to get these items all back together in a neat and tidy space separated, linefeed delimited text file, so if I make them into records I need to un-make them :slight_smile: I’ve found none of that capability so far. Probably a given for experienced programmers but should be documented somewhere.

What do you define as a major hit? Something like this shouldn’t take more than a few seconds:

set theData to x -- text you've read in
set listOfrecords to {} -- to hold list of records
set oldDelims to AppleScript's text item delimiters
set AppleScript's text item delimiters to {" "}
repeat with i from 1 to count of paragraphs of theData
	set oneRecord to paragraph i of theData
	set end of listOfrecords to {col1:text item 1 of oneRecord, col2:text item 2 of oneRecord, col3:text item 3 of oneRecord, col4:text item 4 of oneRecord, col5:text item 5 of oneRecord, col6:text item 6 of oneRecord, ...}
end repeat
set AppleScript's text item delimiters to oldDelims

If that’s too slow, it could be done using ASObjC.

Reversing the process should be easy enough: coerce the records to lists, then use text item delimiters again.

uhhh, you’re absolutely right. … Because I was using a variant of the loop you suggested in processing theData before to get extra spaces out of the text, and it had a bug, that “performance hit” was already at 2 minutes until I found the error I mde in the loop after seeing your post. Instead of setting the end of the list I was copying to it. Now the preprocessing takes about 3 seconds on the list, and your code which I integrated to it, only extends it out a further 3 or 4 seconds.

I will set up the tableview today and post back my code. I think you not only helped me find a solution to coercing lists to records, but also fixed a bug I had that I thought was just processing overhead. :smiley: silly me!

Well, I made slow progress today mainly because of a couple surprises which took me straight back to the overhead issue related to performance mentioned above.
Using the AS Editor for sanity while working this out - First I realized that working on my sample file was not a good idea, when checking the live files they ranged in size from the 1500 record (acutally lines ) that I was basing my project on, to the maximum file size with just over 44,000 lines in it. Also, because it was a unix file with linefeeds, and also had thousands of extra spaces in the records, making space delimited importing almost impossible, so I had previously read the entire file to a variable, then used a repeat to clean up the records using “words of” which worked well.

So the repeat code now looked like this:


repeat with i from 1 to count of fileContents
	set oneRecord to words of text item i of fileContents
	set end of listOfrecords to  {col1:text item 1 of oneRecord, col2:text item 2 of oneRecord, col3:text item 3 of oneRecord, col4:text item 4 of oneRecord, col5:text item 5 of oneRecord, col6:text item 6 of oneRecord, col7:text item 7 of oneRecord, col8:text item 8 of oneRecord, col9:text item 9 of oneRecord, col10:text item 10 of oneRecord, col11:text item 11 of oneRecord, col12:text item 12 of oneRecord, col13:text item 13 of oneRecord, col14:text item 14 of oneRecord, col15:text item 15 of oneRecord, col16:text item 16 of oneRecord, col17:text item 17 of oneRecord}
end repeat

This worked extremely well on the 1500 record files, processing them in about 12 seconds on a 2.4 c2d iMac, and even up to 10,000 records it was taking around 2 minutes…but once I started using the 44,000 list file, I soon realized that the project could not go ahead as it was taking more than 30 minutes (the time limit I force quit the script) to process that huge list.

Just as I was about to give up on it, tonight I found this after many google searches, regarding referencing of lists in AS. (Halfway down the page) - search the page using “bigList” http://developer.apple.com/mac/library/documentation/AppleScript/Conceptual/AppleScriptLangGuide/reference/ASLR_classes.html

This changed everything! First I have posted a sample of the file here with data converted to animal names: http://www.wikiupload.com/download_page.php?id=192398 (if there was a way to host it on this site please let me know as I don’t know how long the sample will be available from a file host site and I think this is a great example for others in the same situation)

The above file takes me 33minutes 32 seconds to read using the above code (2012 secs)
Now, after a straight read of the file into variable fileContents, I set up the references and the repeat code looks like this, a variant of Shane’s great advice above.


set listOfrecordsref to a reference to listOfrecords
set fileContentsref to a reference to fileContents
repeat with i from 1 to count of fileContents
	set oneRecord to words of text item i of fileContentsref
	copy {col1:text item 1 of oneRecord, col2:text item 2 of oneRecord, col3:text item 3 of oneRecord, col4:text item 4 of oneRecord, col5:text item 5 of oneRecord, col6:text item 6 of oneRecord, col7:text item 7 of oneRecord, col8:text item 8 of oneRecord, col9:text item 9 of oneRecord, col10:text item 10 of oneRecord, col11:text item 11 of oneRecord, col12:text item 12 of oneRecord, col13:text item 13 of oneRecord, col14:text item 14 of oneRecord, col15:text item 15 of oneRecord, col16:text item 16 of oneRecord, col17:text item 17 of oneRecord} to the end of listOfrecordsref
end repeat

This processes the same file in THREE seconds (3) on my system. To check that I actually had the records processed (because I didn’t believe it for a number of retries!.. I simply performed a set operation to another variable in the editor at the end of the script and it showed as a result(and took a very very long time!. Logging does the same thing and much faster but you won’t see the quote characters that are in the records. This threw me off for a while until I figured it out.

Note that I also returned to the “copy to the end of the listOfrecords” method as well as using the “set x to a reference of the list” as shown in the developer page examples. Doing this to the oneRecord, due to its small size, had no impact on performance, but using referencing on the fileContents AND listOfrecords, both of which grew exponentially through the repeat’s iterations, reduced processing time by a factor of 400! That is absolutely remarkable.

So, now that I’m back in business I will get this ported into the ASOC project and continue tomorrow to hopefully wire everything into the TableView. Hope this helps others. And of course please feel free to add your advice to make the code better or point out any issues.

EDIT: the code optimization above regarding references only worked in the AS editor and not in XCode with the exception of the listOfrecordsref code in the repeat. See my post below.

How naive of me to think that what works in Applescript Editor would work in Applescript in Xcode lol.

After trying for hours to port the code into Xcode I find that the references optimization above only works with the "listOfrecords at the end of the repeat loop. However using just that reference still processes the "animal.txt file from the above post in 14 seconds on my system, so still happy with the result.

Here is the revised code that runs after the user selectect the file. It then loads it into the TableView that I modified from Shane’s Florida sample code.


	set listOfrecordsref to a reference to my listOfrecords
	repeat with i from 1 to count fileContents
		set oneRecord to words of text item i of my fileContents
		copy {col01:text item 1 of oneRecord, col02:text item 2 of oneRecord, col03:text item 3 of oneRecord, col04:text item 4 of oneRecord, col05:text item 5 of oneRecord, col06:text item 6 of oneRecord, col07:text item 7 of oneRecord, col08:text item 8 of oneRecord, col09:text item 9 of oneRecord, col10:text item 10 of oneRecord, col11:text item 11 of oneRecord, col12:text item 12 of oneRecord, col13:text item 13 of oneRecord, col14:text item 14 of oneRecord, col15:text item 15 of oneRecord, col16:text item 16 of oneRecord, col17:text item 17 of oneRecord} to end of listOfrecordsref
	end repeat
	set my theData to listOfrecordsref

Works beautifully, thanks Shane. I wish there was a way to get it as efficient as before, but once there’s some documentation or more experience with ASOC maybe I can revisit this.

I also tried to implement a determinate progress indicator in the repeat loop in 2 ways, first as a bar but there was no way that I could update the value of the bar within the loop as it read the 20,000 records in the sample. The bar just sat at idle. The second way was to animate the spinning indicator, and doing this before the loop worked stopping it after completion. However, the user feedback was lacking in that I could not update a corresponding label to show the number of records processed.

The indicators are bound with the appropriate attributes to corresponding properties in the project, showing max values, min, the value, animate and hidden values.

Here is the code I tried for the bar indicator:


set my hideProgress to false
set my animateProgress to true
set my progressBarmaxval to (count of fileContents)
repeat with i from 1 to count fileContents
	set oneRecord to words of text item i of my fileContents
	copy {col01:text item 1 of oneRecord, col02:text item 2 of oneRecord, col03:text item 3 of oneRecord, col04:text item 4 of oneRecord, col05:text item 5 of oneRecord, col06:text item 6 of oneRecord, col07:text item 7 of oneRecord, col08:text item 8 of oneRecord, col09:text item 9 of oneRecord, col10:text item 10 of oneRecord, col11:text item 11 of oneRecord, col12:text item 12 of oneRecord, col13:text item 13 of oneRecord, col14:text item 14 of oneRecord, col15:text item 15 of oneRecord, col16:text item 16 of oneRecord, col17:text item 17 of oneRecord} to end of listOfrecordsref
	set my progressBarval to progressBarval + 1
end repeat
set my theData to listOfrecordsref
set my animateProgress to false
set my hideProgress to true

Any advice is much appreciated. If this can’t be done then I will go the spinning indicator route and hopefully find a way to update a label from with in a repeat loop. … and for now I learned that it’s much better to work in the Xcode environment than switching between the Editor and back as there are real differences in the way the Applescript works, like the variants of referencing above, between the two.

Try:

   set my progressBarval to my progressBarval + 1

Thanks Shane. No, it doesn’t function at all. The progress bar doesn’t even unhide now… unless I make it indeterminate.

Also Xcode auto formats the line, on save, to:

			
set my progressBarval to (my progressBarval) + 1

Are you setting progressBarval to an initial value anywhere?

Shane, yes, I set it in “application will finish launching”.

Below I’ll post the proof of concept code now, again based on Shane’s Florida sample code for TableView.

All there is in IB as of now is the Tableview and a progress indicator bound to the appropriate variables. I have set the code so it will work with the Animals.txt file from my above post so folks can use it to test.

The code runs both the handlers on launch. The tableview loads really fast, just not as fast as it did in the AS Editor, loading the animals file (20,000 records- 340,000 fields) in 12 seconds on my machine. Imac 2.4 g, then it writes theData variable to disk.

As of now the big challenges before moving on and actually doing stuff with the data and finishing the interface, are the progress indicator which is now appearing on completion of the read from file full at 100% (I think the delay helps this) but still not initializing and animating correctly, and of course the write to disk needs to be modified to coerce the records to a list. I have spent a number of hours on the the write to disk handler which finally works, but have not been successful with any of the documentation I have read both in this forum and elsewhere with legacy solutions re: coercing theData to a list AND be able to write to the file and put it back in the same format I left it lol.

The table view looks excellent and is very user friendly. It’s the mechanics of ASOC absent of documentation that drives me crazy. Seems like every AS command has been changed in just enough of a way to not function at all unless some obscure change is made to it. :o


property parent : class "NSObject"
	property NSView : class "NSView" of current application
	
	property theData : missing value --  the main property; array controller binds its content array to this

	property fileContents : {}
	property listOfrecords : {}

	property hideprogressBar : missing value
	property animateprogressBar : missing value
	property progressBarmaxval : missing value
	property progressBarminval : missing value
	property progressBarval : 0.0
	
	on applicationWillFinishLaunching_(aNotification)
		-- set some initial data
		set my theData to {}

		set my hideprogressBar to false
		set my animateprogressBar to true
		set my progressBarval to 0.0

		readLogtodata()
		delay 3
		writeFile()

		set my animateProgress to false
		set my hideprogressBar to true
		display dialog ("Done!")
	end applicationWillFinishLaunching_
	
	on applicationShouldTerminate_(sender)
		return (current application's NSTerminateNow) as integer
	end applicationShouldTerminate_
	
	on readLogtodata()
		set my hideprogressBar to false
		set my progressBarval to 1
		set my animateprogressBar to true

		set listOfrecords to {}

		set theFile to (choose file with prompt "Select a file to read:" of type {"TEXT", "txt"} as list)
		open for access theFile
		set fileContents to (read theFile using delimiter {return, ASCII character 10})
		close access theFile

		set listOfrecordsref to a reference to my listOfrecords -- this speeds things up in the repeat by 100x +

		set my progressBarmaxval to (count of fileContents)
		repeat with i from 1 to count fileContents
			set oneRecord to words of text item i of my fileContents
			set end of listOfrecordsref to {col01:text item 1 of oneRecord, col02:text item 2 of oneRecord, col03:text item 3 of oneRecord, col04:text item 4 of oneRecord, col05:text item 5 of oneRecord, col06:text item 6 of oneRecord, col07:text item 7 of oneRecord, col08:text item 8 of oneRecord, col09:text item 9 of oneRecord, col10:text item 10 of oneRecord, col11:text item 11 of oneRecord, col12:text item 12 of oneRecord, col13:text item 13 of oneRecord, col14:text item 14 of oneRecord, col15:text item 15 of oneRecord, col16:text item 16 of oneRecord, col17:text item 17 of oneRecord} -- to end of listOfrecordsref
			set my progressBarval to (my progressBarval) + 1
		end repeat
		set my theData to listOfrecordsref 
               -- would have set theData in the repeat, but it doesn't load into the tableview unless set all at once
               -- tableview won't load dynamically as the repeat is running, but does once set later
	end readLogtodata
	
	on writeFile()
		try
			set thenewFile to open for access ((path to desktop) & "test.txt") as text with write permission
			set eof of thenewFile to 0
			write theData to thenewFile as list starting at eof 
                       -- this just dumps AS code into the file along with the records, needs work
			close access thenewFile
		on error
			try
				display dialog ("error")
				close access thewriteFile
			end try
		end try
	end writeFile

any assistance is always appreciated

So what triggers readLogtodata()?