Parse a list of files based on extensions and process accordingly

Now that is interesting, but I am not convinced, that it will always work, with every object model, and if the property’s datatype is a string up front, then I prefer to use an explicit get anyway.

Hello!

I have stopped using Finder to delete files.

I also think I should be able to circumvent some of the list operations, to make it faster, using specialized handlers for this particular task


(* rewritten to assure finder  doesn't crash or becomes unresponsive *)
set theFolder to ((path to documents folder from user domain as text) & "Week01:")

tell application "Finder"
	set pxFolder to quoted form of POSIX path of (theFolder as alias)
	
	set folToDel to (name of every item of its folder theFolder whose class is folder)
	
	set filesToDel to (get name of its every file of its folder theFolder whose name extension is in {"", "jpg", "tiff", "png"})
end tell

set pxfolList to mkPxFileListString for folToDel
do shell script "cd " & pxFolder & "  ; rm -fR " & pxfolList

set pxfileList to mkPxFileListString for filesToDel

do shell script "cd " & pxFolder & "  ; rm -f " & pxfileList

-- get a list of what is left in the folder 

tell application "Finder" to set {theNames, theExtensions} to (get {name, name extension} of its every file of folder theFolder)
set ct to (get count theNames)
set theList to {}
set i to 1
repeat ct times
	set end of theList to {item i of theNames, item i of theExtensions}
	set i to i + 1
end repeat

-- get a list of every file name with a name  extension that is 7 chars long and isn't convert
set of7list to _filterL(theList, offending7s, 2)

set filesToDel to {}
set ct to get count of7list
set i to 1
repeat ct times
	set end of filesToDel to item 1 of item i of of7list
	set i to i + 1
end repeat

set pxfileList to mkPxFileListString for filesToDel

do shell script "cd " & pxFolder & "  ; rm -f " & pxfileList

tell application "Finder"
	set theNames to (get name of its every file of folder theFolder) -- :)
	set theNames to (sort theNames by name)
end tell
set {tids, AppleScript's text item delimiters} to {AppleScript's text item delimiters, return}

set theNames to theNames as text
set AppleScript's text item delimiters to tids

writeFile(theNames, (theFolder & "Files_to_Check.txt"))

to mkPxFileListString for aFilelist
	local pxfileList
	set pxfileList to {}
	repeat with i from 1 to (get count aFilelist)
		copy quoted form of item i of aFilelist to end of pxfileList
	end repeat
	local tids
	set {tids, AppleScript's text item delimiters} to {AppleScript's text item delimiters, space}
	
	set pxfileList to pxfileList as text
	set AppleScript's text item delimiters to tids
	return pxfileList
end mkPxFileListString


to writeFile(theData, theFile)
	(*For writing a file.  Handles situations where the new file may be
	 shorter than the original file, since AppleScript's write command doesn't 
	 reset EOF to the new data length.*)
	--returns boolean success (true=success)
	--Assumes: theFile is a file path string and file exists.	
	try
		-- open file
		open for access (theFile) with write permission
		copy the result to theFile_ID
		
		-- Set the file length to zero
		set eof theFile_ID to 0
		
		-- Write our message
		write theData to theFile_ID
		
		-- close the file
		close access theFile_ID
		return true
		
	on error
		try
			close access theFile_ID
		end try
		return false
	end try
end writeFile

on _filterL(L, crit, itemNo)
	-- © Matt Neuburg AppleScript The Definitive Guide Second Edition.
	-- reworked to work on lists of lists, finding criteria in chosen item of the sublist
	script filterer
		property criterion : crit
		property itemnum : itemNo
		on _filter(L)
			if L = {} then return L
			if criterion(item itemnum of item 1 of L) then
				return {item 1 of L} & (_filter(rest of L))
			else
				return _filter(rest of L)
			end if
		end _filter
	end script
	return filterer's _filter(L)
end _filterL

on offending7s(x)
	local ct
	set ct to count x
	if ct = 7 and x is not in "convert" then return true
	return false
	
end offending7s


The power of xargs & find :slight_smile: the following example removes all the directories in a given directory.

set theFolder to quoted form of text 1 thru -2 of (POSIX path of (path to desktop folder))
do shell script "find " & theFolder & " -type d -depth 1 -print0 | xargs -0 rm -rf"

The folder can’t have a trailing slash and I’m using null-terminated string (C-string) so I don’t have to quote the paths.

Hello!

This is tested, and it seems to work.

It is not the solution I hoped for, as the list optimization is dubious, but at least the call to finder to regenerate the lists are avoided! I also think that the file extension for is empty, so I test for the extensions, in the filenames, like Nigel does. :slight_smile:

Edit:
Added properties and the usage of my to speed up stuff. I also see I can concatenate the delete operation,
speeding things up more.


(* rewritten to assure finder  doesn't crash or becomes unresponsive *)

property folToDel : missing value
property theNames : missing value

set AppleScript's text item delimiters to ""
set theFolder to ((path to documents folder from user domain as text) & "Week01:")
set folToDel to missing value
tell application "Finder"
	set pxFolder to quoted form of POSIX path of (theFolder as alias)
	try
		set folToDel to (get name of every item of its folder theFolder whose class is folder)
	end try
	
	set my theNames to (get name of its every file of folder theFolder)
	
end tell

if folToDel is not missing value then
	set pxfolList to mkPxFileListStringL for folToDel by 0
	do shell script "cd " & pxFolder & "  ; rm -fR " & pxfolList
end if

set theRefList to {}
repeat with i from 1 to (get count theNames)
	set end of theRefList to (a reference to item i of my theNames)
end repeat

set filesToDel to _filter(theRefList, unwantedXts)

set pxfileList to mkPxFileListStringL for filesToDel by 0

repeat with i from 1 to (get count filesToDel)
	set contents of item i of filesToDel to missing value
end repeat

set theNames to my theNames's text

set theRefList to {}
repeat with i from 1 to (get count theNames)
	set end of theRefList to (a reference to item i of my theNames)
end repeat


do shell script "cd " & pxFolder & "  ; rm -f " & pxfileList

-- get a list of what is left in the folder 


-- get a list of every file name with a name  extension that is 7 chars long and isn't convert
set filesToDel to _filter(theRefList, offending7s)

set pxfileList to mkPxFileListStringL for filesToDel by 0

repeat with i from 1 to (get count filesToDel)
	set contents of item i of filesToDel to missing value
end repeat

do shell script "cd " & pxFolder & "  ; rm -f " & pxfileList

set theNames to my theNames's text

quickSort(my theNames, 1, (count my theNames))

set {tids, AppleScript's text item delimiters} to {AppleScript's text item delimiters, return}

set theNames to my theNames as text
set AppleScript's text item delimiters to tids

writeFile(my theNames, (theFolder & "Files_to_Check.txt"))

to mkPxFileListStringL for aFilelist by itemNo
	local pxfileList
	set pxfileList to {}
	if itemNo = 0 then
		repeat with i from 1 to (get count aFilelist)
			copy quoted form of item i of aFilelist to end of pxfileList
		end repeat
	else
		repeat with i from 1 to (get count aFilelist)
			copy quoted form of item itemNo of item i of aFilelist to end of pxfileList
		end repeat
	end if
	local tids
	set {tids, AppleScript's text item delimiters} to {AppleScript's text item delimiters, space}
	
	set pxfileList to pxfileList as text
	set AppleScript's text item delimiters to tids
	return pxfileList
end mkPxFileListStringL


to writeFile(theData, theFile)
	(*For writing a file.  Handles situations where the new file may be
	 shorter than the original file, since AppleScript's write command doesn't 
	 reset EOF to the new data length.*)
	--returns boolean success (true=success)
	--Assumes: theFile is a file path string and file exists.	
	try
		-- open file
		open for access (theFile) with write permission
		copy the result to theFile_ID
		
		-- Set the file length to zero
		set eof theFile_ID to 0
		
		-- Write our message
		write theData to theFile_ID
		
		-- close the file
		close access theFile_ID
		return true
		
	on error
		try
			close access theFile_ID
		end try
		return false
	end try
end writeFile

-- http://macscripter.net/viewtopic.php?id=38978
on quickSort(theList, theLeft, theRight)
	set i to theLeft
	set j to theRight
	set v to item ((theLeft + theRight) div 2) of theList -- pivot
	repeat while (j > i)
		repeat while (item i of theList < v)
			set i to i + 1
		end repeat
		repeat while (item j of theList > v)
			set j to j - 1
		end repeat
		if (not i > j) then
			tell theList to set {item i, item j} to {item j, item i} -- swap
			set i to i + 1
			set j to j - 1
		end if
	end repeat
	if (theLeft < j) then quickSort(theList, theLeft, j)
	if (theRight > i) then quickSort(theList, i, theRight)
end quickSort

on _filter(L, crit)
	-- © Matt Neuburg AppleScript The Definitive Guide Second Edition.
	script filterer
		property criterion : crit
		on _filter(L)
			if L = {} then return L
			if criterion(item 1 of L) then
				return {item 1 of L} & _filter(rest of L)
			else
				return _filter(rest of L)
			end if
		end _filter
	end script
	return filterer's _filter(L)
end _filter

on unwantedXts(x)
	set t to reverse of every character of x as text
	if t starts with "gpj" or t starts with "ffit" or t starts with "gnp" then return true
	return false
end unwantedXts

on offending7s(x)
	local t
	set t to reverse of every character of x as text
	if (offset of "." in t) is 8 and x does not end with "convert" then return true
	return false
	
end offending7s

My previous post was showing how you can properly remove files. The code you’re using I’m not fond of, because when cd fails any file can be removed on your HD. For instance:

do shell script "cd ~Desktop ; ls"

because of an typo (but also something else could go wrong) it will fail but won’t trow an error, instead it will list my HD. With do shell script I avoid cd as much as possible and will use absolute paths as much as possible. There are some cases when you can/need use a cd command. But I want to execute the second command only if the first was successful, so I need something like this:

do shell script "cd ~Desktop && ls"

Now an error will occur when trying to changing the current working directory and the second command is not executed. This is a more safe way of working. You could even return an alternative or default to avoid errors or do error handling yourself

do shell script "cd ~Desktop && ls || echo [NULL]"

. except, unfortunately, that it deletes all the subfolders, JPEGs, PNGs, etc. in the folder and logs everything that’s left into the text file. :frowning:

Chris’s requirements were to ignore folders, JPEGs, TIFFs, and PNGs; to delete files with seven-character name extensions (except for those with “.convert”) or with no extensions at all; and to write to a text file the names of files with five, six, eight, or more characters in their name extensions. (Actually, he also mentioned any unignored files which hadn’t been handled above. Your script may be right in that respect and mine wrong. I’ll have to look at it again tomorrow ” when I’ve rebuilt my test folder. ;))

Hello!

Well, I did read his requirments as deleting every subfolder, and understood it to be with contents.

I’m sorry about ruining your testfolder! Well, I think also I forgot to remove the files without any extension.

(I can’t seem to get that he wanted to keep the contents of the subfolders though.)

I’ll have to look at it tomorrow as well, its late… I’m thinking of leaving Finder be, as I don’t need a list of file extensions anyway! I’ll also merge the two filter operations and remove the deleted items in one go, to speed things up. I can’t see how it will help to sort the list by extension, nor length of extension in this particular case.
Only to sort the filnames, to make it easier to check them with the contents of the folder afterwards.

Edit
Today I’ll use Timemachine after I have constructed the test folder, -again! :wink:

Hello! :frowning:

I used the reference trick, to just keep all within a list, when declaring properties, and later on setting contents of the extracted list to missing value, it broke! It seems like I am doing a privilege violation or such.

So I removed the my references, in order for basis to work, slowing things down.

I have however merged the operations, so the overall speed should be faster than it was up front. The commandline is constructed under the assumption, that the bash commandline can be around 54.000 characters long, which will leave me with the possibility of deleting 400 files each with a file name length of 135 characters, before I get into trouble.


(* rewritten to assure finder  doesn't crash or becomes unresponsive 
optimized!*)

set theFolder to ((path to documents folder from user domain as text) & "Week01:")

-- Deleting subfolders
set folToDel to missing value

tell application "Finder"
	set pxFolder to quoted form of POSIX path of (theFolder as alias)
	try
		set folToDel to (get name of every item of its folder theFolder whose class is folder)
	end try
end tell

if folToDel is not missing value then
	set pxfolList to mkPxFileListStringL for folToDel by 0
	do shell script "cd " & pxFolder & "  ; rm -fR " & pxfolList
end if

-- Starts processing unwanted files
set theNames to every paragraph of (do shell script "cd " & pxFolder & "  ; ls -1 |sort -f")

set theRefList to {}
repeat with i from 1 to (get count theNames)
	set end of theRefList to (a reference to item i of theNames)
end repeat

set filesToDel to _filter(theRefList, unwantedFiles)

set pxfileList to mkPxFileListStringL for filesToDel by 0

ignoring application responses
	do shell script "cd " & pxFolder & " ; {  rm -f " & pxfileList & " ; } & "
end ignoring

-- Cleaning up before showing what to check
repeat with i from 1 to (get count filesToDel)
	set contents of item i of filesToDel to missing value
end repeat

set theNames to theNames's text
-- Constructing the file
set {tids, AppleScript's text item delimiters} to {AppleScript's text item delimiters, return}

set theNames to theNames as text
set AppleScript's text item delimiters to tids

writeFile(theNames, (theFolder & "Files_to_Check.txt"))

to mkPxFileListStringL for aFilelist by itemNo
	local pxfileList
	set pxfileList to {}
	if itemNo = 0 then
		repeat with i from 1 to (get count aFilelist)
			copy quoted form of item i of aFilelist to end of pxfileList
		end repeat
	else
		repeat with i from 1 to (get count aFilelist)
			copy quoted form of item itemNo of item i of aFilelist to end of pxfileList
		end repeat
	end if
	local tids
	set {tids, AppleScript's text item delimiters} to {AppleScript's text item delimiters, space}
	
	set pxfileList to pxfileList as text
	set AppleScript's text item delimiters to tids
	return pxfileList
end mkPxFileListStringL


to writeFile(theData, theFile)
	(*For writing a file.  Handles situations where the new file may be
	 shorter than the original file, since AppleScript's write command doesn't 
	 reset EOF to the new data length.*)
	--returns boolean success (true=success)
	--Assumes: theFile is a file path string and file exists.	
	try
		-- open file
		open for access (theFile) with write permission
		copy the result to theFile_ID
		
		-- Set the file length to zero
		set eof theFile_ID to 0
		
		-- Write our message
		write theData to theFile_ID
		
		-- close the file
		close access theFile_ID
		return true
		
	on error
		try
			close access theFile_ID
		end try
		return false
	end try
end writeFile

on _filter(L, crit)
	-- © Matt Neuburg AppleScript The Definitive Guide Second Edition.
	script filterer
		property criterion : crit
		on _filter(L)
			if L = {} then return L
			if criterion(item 1 of L) then
				return {item 1 of L} & _filter(rest of L)
			else
				return _filter(rest of L)
			end if
		end _filter
	end script
	return filterer's _filter(L)
end _filter

on unwantedFiles(x)
	set t to reverse of every character of x as text
	if t starts with "gpj" or t starts with "ffit" or t starts with "gnp" then
		return true
	else if (offset of "." in t) is 8 and x does not end with "convert" then
		return true
	else
		return false
	end if
end unwantedFiles

The limit is 262144 for most darwin kernels, run the following code:

do shell script "getconf ARG_MAX"
--or
do shell script "sysctl -n kern.argmax"

Also you told another good reason to use my example in my previous posts. When removing folders my way you will never get in trouble even if you want to remove millions of files.

Hello!

I’m not sure which posts you are referring to, but please do post a link, as I am sure they should be interesting to read! :slight_smile:

One of those days, I’ll sit down and get a little bit more confident with the commands mentioned. I’m not a kernel builder! (getconf and syscntl)

Hello!

Actually, and honestly, I think all the deletion in my script is inferior to an ideal solution, the ideal solution would be a command line, that moves the files physically to the trash.

The reason for not using Finder for this, is that it sends to many events, in a to short time, and some laziness, as if I used Finder, then I’d rather close every window, that may show the contents of the folder.

So the original poster, is advised to use the Time Machine before running the script, if it ever might happen that some of the deleted items may have some interest, as with my solution, the files are gone forever!

So, I’ll upgrade my solution, during the afternoon, and move the items to ~/.Trash!

It’s in this topic:
An example how to remove multiple files
Here I said it’s not wise to use cd that way

Well for server software I prefer shell scripts over AppleScript I learned a lot here. Another reason is that most clients doesn’t use OS X servers anymore and steps over to Linux distributions; I’m forced to shell scripting. I’m fan of these command line interpreters, how fast they are and how piping and command substitution can be very useful. Through the years I’m very comfortable in the shell and when opening a shell’s overhead isn’t an issue I prefer shell over AS when not inter processing with applications, especially when text processing and file management.

Hello!

I am fond of the shell as well, for the same reasons, and the same exeptions. (But I prefer moving stuff to trash.)

I agree with you totally on the cd command I will make a string construct that should go like this as a replacment, under the hope that cd returns an errorcode when fed a non-existing directory: Edit It does!

do shell script "cd " & pxFolder & " &&  rm -fR " & pxfolList

For those not so well versed in shell, a command string is a string of commands separated by either && or || the && works like a short circuted and, in that if the error code of the command preceding that and returns with any other errorcode than 0, then the rest of the commandstring fails.

The || works like an or (inclusive ), the command after the || gets to be executed, no matter what error code the previous command returned with.

Thanks for pointing that out! :slight_smile: I see that my script don’t catch that error early, in the finder tell block, so I shall implent a test for it there, before running the script, I will still use the string construct as I may be liable to copy paste code in the future! :slight_smile:

As for the xargs construct; I really tried to used space separated qouted form of paths, fed to xargs with cat like this:

do shell script "cd " & pxFolder & " ; cat " & pxfileList & " |xargs rm -f " 

But I didn’t manage to make xargs work for some odd reason! (I got an error saying it was missing a quote.)

I do however like the way you use find to remove all subdirectories, I seldom use find nowadays, with locate, and mdfind, or just spotlight! :slight_smile: Consider your way snagged!.

(I really don’t understand how I failed to see that post yesterday, but I did! ) :frowning:

Hello!

I can’t seem to make this line of do shell script work!


        try
        do shell script "find " & pxFolder & " -type d -depth 1 -exec ditto -v \\{\\} ~/.Trash/\\{\\} \\;"
    on error e number n
        display dialog e & " : " & n
    end try

Both this

find . -type d -depth 1 -exec ditto -v \{\} ~/.Trash/\{\} \;

and this

find . -type d -depth 1 -exec ditto -v {} ~/.Trash/{} \;

works fine on the command line! the path is used in other incantations of find, and should be ok!

Edit
I found a work around that works!

do shell script "find " & pxFolder & " -type d -depth 1 -print0 | xargs -0 -I {} cp -r {} ~/.Trash"

It is still interesting to know why it fails! I even spelled out the full path to ditto with no relief!

Hello!

It works, it is fairly tested, and robust. you now set the property hasSkepsis to true, to send the files to the ~/.Trash

I consider myself finished with this, unless the OP has some comments, and of course if anybody should find any errors in this!


” © McUsr with good help and productive comments by DJ Bazzie Wazzie
property hasSkepsis : true
(* rewritten to assure finder  doesn't crash or becomes unresponsive 
optimized!*)
try
	set theFolder to ((path to documents folder from user domain as text) & "Week01:")
	set theTest to theFolder as alias
on error
	tell application "SystemUIServer"
		activate
		display dialog "The Folder : " & theFolder & " Does not exist.

You must edit the script!

Aborting." buttons {"Ok"} default button 1 with icon 2
	end tell
	error number -128
end try
-- Deleting subfolders
set folToDel to missing value

set pxFolder to quoted form of POSIX path of (theFolder as alias)

-- removing folders 
if hasSkepsis then
	try
		do shell script "find " & pxFolder & " -type d -depth 1 -print0 | xargs -0 -I {} cp -r {} ~/.Trash"
	end try
end if

ignoring application responses
	try
		do shell script "find " & pxFolder & " -type d -depth 1 -print0 | xargs -0 rm -rf"
	end try
end ignoring

-- Starts processing unwanted files
set theNames to every paragraph of (do shell script "cd " & pxFolder & "  && ls -1 |sort -f")

set theRefList to {}
repeat with i from 1 to (get count theNames)
	set end of theRefList to (a reference to item i of theNames)
end repeat

set filesToDel to _filter(theRefList, unwantedFiles)

set pxfileList to mkPxFileListStringL for filesToDel by 0

-- removing files
if hasSkepsis then
	do shell script "cd " & pxFolder & " && {  cp" & pxfileList & "  ~/.Trash ; } & "
end if

ignoring application responses
	do shell script "cd " & pxFolder & " && {  rm -f " & pxfileList & " ; } & "
end ignoring

-- Cleaning up before showing what to check
repeat with i from 1 to (get count filesToDel)
	set contents of item i of filesToDel to missing value
end repeat

set theNames to theNames's text
-- Constructing the file
set {tids, AppleScript's text item delimiters} to {AppleScript's text item delimiters, return}

set theNames to theNames as text
set AppleScript's text item delimiters to tids

writeFile(theNames, (theFolder & "Files_to_Check.txt"))

to mkPxFileListStringL for aFilelist by itemNo
	local pxfileList
	set pxfileList to {}
	if itemNo = 0 then
		repeat with i from 1 to (get count aFilelist)
			copy quoted form of item i of aFilelist to end of pxfileList
		end repeat
	else
		repeat with i from 1 to (get count aFilelist)
			copy quoted form of item itemNo of item i of aFilelist to end of pxfileList
		end repeat
	end if
	local tids
	set {tids, AppleScript's text item delimiters} to {AppleScript's text item delimiters, space}
	
	set pxfileList to pxfileList as text
	set AppleScript's text item delimiters to tids
	return pxfileList
end mkPxFileListStringL


to writeFile(theData, theFile)
	(*For writing a file.  Handles situations where the new file may be
	 shorter than the original file, since AppleScript's write command doesn't 
	 reset EOF to the new data length.*)
	--returns boolean success (true=success)
	--Assumes: theFile is a file path string and file exists.	
	try
		-- open file
		open for access (theFile) with write permission
		copy the result to theFile_ID
		
		-- Set the file length to zero
		set eof theFile_ID to 0
		
		-- Write our message
		write theData to theFile_ID
		
		-- close the file
		close access theFile_ID
		return true
		
	on error
		try
			close access theFile_ID
		end try
		return false
	end try
end writeFile

on _filter(L, crit)
	-- © Matt Neuburg AppleScript The Definitive Guide Second Edition.
	script filterer
		property criterion : crit
		on _filter(L)
			if L = {} then return L
			if criterion(item 1 of L) then
				return {item 1 of L} & _filter(rest of L)
			else
				return _filter(rest of L)
			end if
		end _filter
	end script
	return filterer's _filter(L)
end _filter

on unwantedFiles(x)
	set t to reverse of every character of x as text
	if t starts with "gpj" or t starts with "ffit" or t starts with "gnp" then
		return true
	else if (offset of "." in t) is 8 and x does not end with "convert" then
		return true
	else
		return false
	end if
end unwantedFiles

set targetFolder to quoted form of POSIX path of (path to trash folder)
set pxFolder to quoted form of text 1 thru -2 of POSIX path of (path to desktop folder)
do shell script "find " & pxFolder & " -type d -depth 1 -execdir ditto -v $(basename {}) " & targetFolder & "$(basename {}) ';'"

Something like this? The example copies every folder of your desktop to the trash

EDIT:

  • What I did different is set the working directory of ditto the the same directory as where the file is found (-execdir)
  • To get the file name I used basename (with command substitution)

the result is the for example I have a folder ‘Test’ on my desktop that the execution will be similar as this:

ARRGGHHH :mad:

Of course! Given a directory, I would have needed to use the basename. Oh well, I have worked around it, but it wasn’t more difficult than that -my fault! I bet that ditto won’t make the necessary intermediary folders.

It is alwasy good to have know the explanation to the cause! :slight_smile: Thanks! :slight_smile:

Edit

I’ll stick with cp -r now, as it is convenient for this purpose, but I’ll be sure to remember to use the basename function, and expansion or substitution of it herafter when dealing with find and ditto and paths to folders.

It is a very convenient combo, find and xargs, and some command.

Hi guys,

You are fantastic, so much scripting experience it’s awesome. It will take me quite some time to digest all this.

I’ll have to try quite a number of your ideas for an optimum folder cleaning script but, I better wait for the end of the Olympics to recover a large sleep deficit and get a clear head.

In addition, I already know a few scripts I can improve with some of your technics discussed here. Also, a few ideas for new scripts.

Many thanks again and Best Regards,
Chris