looking for a better way.

Hello all, Im looking for some advice or help here. I have a script that works on a watched folder bases. A image goes in, the applescript reads a comma separated document that has 3 fields (newname, original name, and new path). My script takes the file name of the image in the hotfolder. Finds all of the times that that name is used in column 2 of my csv file (original name) from this, I have to duplicate the document to a new location (new path) and give it a new name (newname). Example, 1 image could be duplicated 20 times with 20 different final names and locations.

Problem. Currently I am doing this using this subroutine to read in the file.


set ExcelFileContents to my readfile(excelFile)


on readfile(unixPath)
	set theText to (open for access (unixPath))
	set txt to paragraphs of (read theText)
	close access theText
	return txt
end readfile

and then I am using a repeat to find what to do.


repeat with aRow in ExcelFileContents
		set FinalImageName to ""
		set AppleScript's text item delimiters to ","
		set TheColumns to every text item of aRow
		set partNumber to item 1 of TheColumns as string
		set UsedImage to item 2 of TheColumns as string
		set NewLocation to item 3 of TheColumns as string
		set AppleScript's text item delimiters to ""
		if (UsedImage as string) = fileName then -- This means we found the match in the data. start the work
			set FinalImageName to partNumber & TheExtension as string
			-- kick it down to the subroutine for processing
			my makeFoldersandDuplicateFileRename(inInputFile, FinalImageName, NewLocation, RootFolder)
		end if
	end repeat



Im finding this to be a very slow process. My excel files that are coming in are now topping 9000 rows and its taking for ever to process. I dont know much about shell scripts but I was wondering if there is a faster way to do this?

thanks

The sluggishness is most certainly coming from searching and filtering the Excel data. Have you considered using a different method to store your data, like SQLite? It is on your Mac, and you can execute queries that will return your necessary data much faster and more efficiently than Excel.

If you find the GUI of Excel easier to work with, it would not be difficult to set up a script that would sync your Excel with an SQLite database whenever necessary.

I would love to use a different method like sqllite. But I would have no idea where to start. Also, this still has to be applescript based. So it would have to be a applescript sending the queries or do shell scripts. Although I said that this is a watched folder script, its actually part of a workflow that has a “run applescript” module. So that is the only “language” choice that I have here.

Thanks

I wrote a tutorial on SQLite that might be a good beginning

Thanks, Ill have a look at it

Running into some problems. Would appreciate any help possible. Im trying to build the empty database. And then populate it while I repeat thru the paragraphs of the csv file that I read in using the subroutine.


set excelFile to alias "Macintosh HD:Users:peki:Desktop:SampleDATA2_orig.csv"
set ExcelFileContents to my readfile(excelFile)


-- set up db name and headers
set loc to space & "~/desktop/TestDB.db" & space
set head to "sqlite3" & loc & quote
set tail to quote
set tblName to "Harley"
set newTbl to "create table " & tblName & "(partNumber, UsedImages, NewPath); "
tell application "Finder"
	if file "TestDB.db" of desktop exists then
	else
		do shell script head & newTbl & tail
	end if
end tell
repeat with aRow in ExcelFileContents
	set FinalImageName to ""
	set AppleScript's text item delimiters to ","
	set TheColumns to every text item of aRow
	set partNumber to item 1 of TheColumns as string
	set UsedImage to item 2 of TheColumns as string
	set NewLocation to item 3 of TheColumns as string
	set AppleScript's text item delimiters to ""
	set newData to "insert into mods values ('" & partNumber & "', " & UsedImage & "', " & NewLocation & "'); "
	do shell script newData & tail
end repeat

on readfile(unixPath)
	set theText to (open for access (unixPath))
	set txt to paragraphs of (read theText)
	close access theText
	return txt
end readfile


I need to replace the ‘ray’, ‘barber’, ‘usa’; " with my variables (partnumber, usedImage, NewLocation) is this possible? Im not sure how to escape the quotes properly.

thanks

Yes.


set d1 to "insert into mods values(" & quoted form of partnumber & ", " & quoted form of usedImage & quoted form of NewLocation & "); "

Cheers,

Craig

Im getting close :slight_smile: but im getting a error “SQL error: no such table: mods”.

It is creating the TestDB.db file on my desktop.

set excelFile to alias "Macintosh HD:Users:peki:Desktop:SampleDATA2_orig.csv"
set ExcelFileContents to my readfile(excelFile)

-- set up db name and headers
set loc to space & "~/desktop/TestDB.db" & space
set head to "sqlite3" & loc & quote
set tail to quote
set tblName to "Harley"
set newTbl to "create table " & tblName & "(partNumber, UsedImages, NewPath); "
do shell script head & newTbl & tail

repeat with aRow in ExcelFileContents
	set AppleScript's text item delimiters to ","
	set TheColumns to every text item of aRow
	set partNumber to item 1 of TheColumns as string
	set UsedImage to item 2 of TheColumns as string
	set NewLocation to item 3 of TheColumns as string
	set AppleScript's text item delimiters to ""
	set newData to "insert into mods values(" & quoted form of partNumber & ", " & quoted form of UsedImage & quoted form of NewLocation & "); "
end repeat

-- First, the same path, head and tail we used before:
set loc to space & "~/desktop/TestDB.db" & space
set head to "sqlite3 -line" & loc & quote -- the "-line" option outputs the column data and heading one line at a time - useful for parsing the output for particular data items.
set tail to quote
set All to "select * from mods; " -- the "*" means get all rows. columns are separated by pipes ("|") in the result.
set modsTable to do shell script head & All & tail
set title to "So far, the table contents are: " & return & return
display dialog title & modsTable buttons {"Done"} default button 1 with icon 1

Here is a Library I wrote a few months ago after I read through Adam’s tutorial to get a feel for sqlite3.
THIS HAS NOT BEEN THOROUGHLY TESTED!!

PLEASE LOOK THIS OVER THOROUGHLY BEFORE USE

That said, hopefully this will give you a head start and something
you can modify to your needs.

Cheers,

Craig


--#################################################################################
--#                                                                               #
--#     SqliteClass.applescript                                                   #
--#                                                                               #
--#     author:   Craig Williams                                                  #
--#     created:  2008-07-17                                                      #
--#                                                                               #
--#################################################################################
--#                                                                               #
--#     This program is free software: you can redistribute it and/or modify      #
--#     it under the terms of the GNU General Public License as published by      #
--#     the Free Software Foundation, either version 3 of the License, or         #
--#     (at your option) any later version.                                       #
--#                                                                               #
--#     This program is distributed in the hope that it will be useful,           #
--#     but WITHOUT ANY WARRANTY; without even the implied warranty of            #
--#     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the             #
--#     GNU General Public License for more details.                              #
--#                                                                               #
--#     You should have received a copy of the GNU General Public License         #
--#     along with this program.  If not, see <http://www.gnu.org/licenses/>.     #
--#                                                                               #
--#################################################################################

(*
folder is created in ~/Library/Application Support/database_name

@property = folder_name
@property = database_name

@function = create_db(table_name, column_names_array)
@function = sql_insert(table_name, the_values)
@function = sql_update(table_name, the_fields, the_values, search_field, search_value)
@function = sql_addColumn(table_name, col_name)
@function = sql_select(column_names_array, table_name, search_field, search_value)
@function = sql_select_all(column_names_array, table_name)
@function = sql_select_all_where(table_name, search_field, search_value)
@function = sql_delete(table_name, search_field, search_value)
@function = sql_delete_every_row(table_name)
@function = sql_delete_table(table_name)
*)


property folder_name : "NameOfDBFolder"
property database_name : "NameOfDB"


-- I create comma sep quoted string to enter into sqlite db
on returnCommaSepQuotedString(the_array)
	set return_string to ""
	if length of the_array > 1 then
		repeat with i from 1 to count of the_array
			set this_item to item i of the_array
			set return_string to return_string & "'" & this_item & "', "
		end repeat
		return text 1 thru -3 of return_string as string
	else
		--return "'" & (item 1 of the_array) & "'"
		return item 1 of the_array
	end if
end returnCommaSepQuotedString


-- RETURN FILE PATH, HEAD, TAIL
on createFolderReturnFilePathHeadTail()
	set support_folder to (path to application support from user domain)
	tell application "Finder"
		set folder_path to (support_folder & folder_name) as string
		if not (exists folder folder_path) then
			make folder at support_folder with properties {name:folder_name}
		end if
	end tell
	set file_path to support_folder & folder_name & ":" & database_name & ".db" as string
	set file_path to quoted form of POSIX path of file_path
	set loc to space & file_path & space
	set head to "sqlite3" & loc & quote
	set tail to quote
	return {file_path, head, tail}
end createFolderReturnFilePathHeadTail


-- CREATE DB
on create_db(table_name, column_names_array)
	set column_names_string to my returnCommaSepQuotedString(column_names_array)
	set {file_path, head, tail} to my createFolderReturnFilePathHeadTail()
	set sql_statement to "create table if not exists " & table_name & "(" & column_names_string & "); "
	--display dialog head & sql_statement & tail as string
	--return
	do shell script head & sql_statement & tail
end create_db


-- INSERT INTO DB
on sql_insert(table_name, the_values)
	set {file_path, head, tail} to my createFolderReturnFilePathHeadTail()
	set the_values to my returnCommaSepQuotedString(the_values)
	set sql_statement to "insert into " & table_name & " values(" & the_values & "); "
	do shell script head & sql_statement & tail
end sql_insert


-- UPDATE DB
on sql_update(table_name, the_fields, the_values, search_field, search_value)
	set {file_path, head, tail} to my createFolderReturnFilePathHeadTail()
	
	repeat with i from 1 to count of the_fields
		set this_item to item i of the_fields
		set sql_statement to ("UPDATE " & table_name & " set " & this_item & "  = '" & item i of the_values & "' WHERE " & search_field & " = '" & search_value & "'; " as string)
		log sql_statement
		do shell script head & sql_statement & quote
	end repeat
end sql_update

-- ADD COLUMN
on sql_addColumn(table_name, col_name)
	set {file_path, head, tail} to my createFolderReturnFilePathHeadTail()
	set sql_statement to ("ALTER table " & table_name & " add " & col_name & "; " as string)
	do shell script head & sql_statement & quote
end sql_addColumn


-- SELECT
on sql_select(column_names_array, table_name, search_field, search_value)
	set {file_path, head, tail} to my createFolderReturnFilePathHeadTail()
	set column_names_string to my returnCommaSepQuotedString(column_names_array)
	set sql_statement to ("SELECT  " & column_names_string & "  FROM  " & table_name & "  WHERE " & search_field & " = " & search_value & "; " as string)
	set sql_execute to (do shell script head & sql_statement & quote)
	set sel_list to my tidStuff(return, sql_execute)
	--set sel_list to my tidStuff(return, sql_execute)
	--log sel_list
	return my sel_list
end sql_select


-- SELECT ALL
on sql_select_all(table_name)
	set {file_path, head, tail} to my createFolderReturnFilePathHeadTail()
	set sql_statement to ("SELECT * FROM  " & table_name & " ; " as string)
	set sql_execute to (do shell script head & sql_statement & quote)
	return my tidStuff(return, sql_execute)
end sql_select_all

-- SELECT ALL WHERE
on sql_select_all_where(table_name, search_field, search_value)
	set {file_path, head, tail} to my createFolderReturnFilePathHeadTail()
	set sql_statement to ("SELECT * FROM  " & table_name & " WHERE " & search_field & " = " & search_value & " ; " as string)
	set sql_execute to (do shell script head & sql_statement & quote)
	return my tidStuff(return, sql_execute)
end sql_select_all_where

-- DELETE ONE ROW
on sql_delete(table_name, search_field, search_value)
	set {file_path, head, tail} to my createFolderReturnFilePathHeadTail()
	set sql_statement to ("DELETE FROM " & table_name & " WHERE " & search_field & " = " & search_value & " ; " as string)
	log sql_statement
	do shell script head & sql_statement & quote
end sql_delete

-- DELETE EVERY ROW
on sql_delete_every_row(table_name)
	set {file_path, head, tail} to my createFolderReturnFilePathHeadTail()
	set sql_statement to ("DELETE * FROM " & table_name & "; " as string)
	do shell script head & sql_statement & quote
end sql_delete_every_row

-- DELETE TABLE
on sql_delete_table(table_name)
	set {file_path, head, tail} to my createFolderReturnFilePathHeadTail()
	set sql_statement to ("DELETE  " & table_name & "; " as string)
	do shell script head & sql_statement & quote
end sql_delete_table

-- TURN STRING INTO LIST
on tidStuff(paramHere, textHere)
	set OLDtid to AppleScript's text item delimiters
	set AppleScript's text item delimiters to paramHere
	set theItems to text items of textHere
	set AppleScript's text item delimiters to OLDtid
	return theItems
end tidStuff

Craig, that was a huge help. My script is complete and running great. I do have a quick question. How would I get a count of all rows of my table using do shell script?

Thanks again for all your help.

Use sql_select_all and count the result.


set rows to count of (sql_select_all("myTable"))

Cheers,

Craig

Actually, that’s very inefficient, since you’re processing the whole search and results, just to get a count.

An almost immediate method is:

(do shell script head & "select count(*) from myTable;" & quote)

Tom
BareFeet


Comparison of SQLite GUI apps for Mac OS X:
http://www.tandb.com.au/sqlite/compare/

It looks fairly comprehensive. Here are my initial thoughts:

You have used this routine for managing a list of values and also for managing a list of column names. Values should be quoted with the single quotes, but only if they are text values, and you need to allow for single quotes that may actually appear in the value by escaping it as two single quotes (ie ‘’, not ").

Column names, however, should use a double quote (not to be confused with two single quotes). Sometimes you can get away with swapping single for double quotes, but it is the wrong syntax and leads to problems.

Here are some handlers for each:


on returnCommaSepQuotedStringValues(the_array) -- use for values in rows (eg in sql_insert), not for column names
	set quoteChar to "'" -- single quote
	returnCommaSepQuotedString(the_array, quoteChar)
end returnCommaSepQuotedStringValues

on returnCommaSepQuotedStringEntities(the_array) -- use for column names (eg in create_db), table names, not values
	set quoteChar to quote -- AppleScript's reserved word for a double quote: "
	return returnCommaSepQuotedString(the_array, quoteChar)
end returnCommaSepQuotedStringEntities

on returnCommaSepQuotedString(the_array, quoteChar)
	set oldDelimiters to AppleScript's text item delimiters
	-- Find any text values, quote them and escape any quote characters with quotes:
	repeat with item_ref in the_array
		set item_value to contents of item_ref
		if class of item_value is text then
			if item_value contains quoteChar then
				set AppleScript's text item delimiters to quoteChar
				set parsedList to text items in item_value
				set AppleScript's text item delimiters to (quoteChar & quoteChar)
				set item_value to parsedList as text
			end if
			set item_value to (quoteChar & item_value & quoteChar) as text
			set contents of item_ref to item_value
		end if
	end repeat
	-- Join all the values together as a string, separated by commas:
	set AppleScript's text item delimiters to ", "
	set return_string to the_array as text
	set AppleScript's text item delimiters to oldDelimiters
	return return_string
end returnCommaSepQuotedString

I think there’s an error in this:

You need to use the SQL command “DROP TABLE” to delete/drop a table, rather than “DELETE”, which is just for rows.

All of your sql_select routines return the data through your tidStuff handler, which breaks the text into a list where each item is a row of data:

However, this does not allow for returns that might be in the actual data and it doesn’t split each row into a list of items/columns. I’ve posted an alternative method here:
http://macscripter.net/viewtopic.php?pid=97447#p97447

Hope this helps,
Tom
BareFeet


Comparison of SQLite GUI apps for Mac OS X:
http://www.tandb.com.au/sqlite/compare/

Thanks for all the input on this Tom. I haven’t looked at it since the afternoon I put it
together. Your input is very valuable to those who will be using sqlite3 from AppleScript. :slight_smile:

Cheers,

Craig