Baffling bug causing shell script rsync to skip archive

t.spoon · September 24, 2019, 1:48am

We’ve got a long, complicated handler in a library that, among other things, uses “do shell script” to run an rsync to move a file from a local temp folder to an NAS folder mounted via SMB.
Separately, we used to rename and make a backup copy of any existing file in the same path via Finder before moving the new file in via rsync.

(We are using rsync here because when we used to use Finder to perform the copy to the server, we used to end up with truncated files all the time for no apparent reason, and end up with files where, for other users, their computer would incorrectly report that they did not have permissions to save over the file. Switching from Finder to rsync to perform the file move fixed both these problems.)

We recently found out that having rsync perform the archiving of any pre-existing file in the same command as the move is WAY faster than having Finder do it. We now have some users working from abroad using SMB over a VPN with ≈250ms of lag and 16 hops, and the Finder move/rename operation often times out or fails, where the rsync archive command works instantly.

So I changed the script to issue the identical rsync command we developed in the terminal. It works in terminal. When run from Applescript inside our large handler, it moves the local file to the server correctly, but it does not perform the archive part of the command - no archive file is created.

Here’s an example of the rsync command:

To run this as a “do shell script,” I first create a variable of the entire text of the command (“shellLine”), then just run it via “do shell script shellLine.” I can “display dialog shellLine” immediately before running “do shell script shellLine,” copy and paste the dialog text into terminal, and run it. The identical line run in terminal moves the file from local to remote and then completes the archive operation. The line run from Applescript moves the file from local to remote and does NOT complete the archive operation.

So I dug into troubleshooting… last time I saw something like this, I had a new line character that terminal didn’t like that got corrected when copy & pasted into terminal, explaining the discrepancy in behavior.

So grabbing my lines that assemble the shell script line, putting them in a new script and assigning the variables, it works fine. It DOES do the archive:

set tempSavePath to "/Volumes/Hackintosh HD/Users/work/tmp/3_Back_DTG_767326_1068710.psd"
set serverFolderPath to "/Volumes/newtees/2016/S/Sandeep Gopal/326759/Masters/"
set saveFileName to "3_Back_DTG_767326_1068710.psd"

set shellLine to ("rsync --itemize-changes --whole-file --backup --backup-dir=../Art/Archive --suffix=\".\"$(date +\"%m-%d-%Y_%H-%M-%S.psd\")  \"" & tempSavePath & "\" \"" & serverFolderPath & saveFileName & "\"")
set transferResult to do shell script shellLine

So instead of copying that one line out of the function, I tried a stripped down version of the original (long) function:

set frontApp to application "Adobe Photoshop CS6"
export_to_server(frontApp, "3_Back_DTG_767326_1068710.psd", "/Volumes/newtees/2016/S/Sandeep Gopal/326759/Masters/", false)

on export_to_server(frontApp, saveFileName, serverFolderPath, noDialogMode) --(application, string including extension, string ending with "/", boolean)
	
	set tmpLocation to (path to home folder as text)
	set tempFolder to (tmpLocation & "tmp") as alias
	set tempSavePath to (POSIX path of tempFolder) & saveFileName as text
	
	
	considering application responses
		with timeout of 300 seconds
			try
				if (serverFolderPath contains "Masters") and (saveFileName does not contain "Proof") then
					set shellLine to ("rsync --itemize-changes --whole-file --backup --backup-dir=../Art/Archive --suffix=\".\"$(date +\"%m-%d-%Y_%H-%M-%S.psd\")  \"" & tempSavePath & "\" \"" & serverFolderPath & saveFileName & "\"")
					-- display dialog shellLine
					set transferResult to do shell script shellLine
				else
					set transferResult to do shell script ("rsync -aW --delete-before \"" & tempSavePath & "\" \"" & serverFolderPath & saveFileName & "\"")
				end if
			end try
		end timeout
	end considering
end export_to_server

Works fine.

Now, the real original function that inexplicably fails has got so many dependencies in it there’s no chance of me providing everything necessary for any of you to run the thing… but I’m praying that by looking at the code, somebody can give me some hint of what on earth could cause the identical rsync line to RUN but the archive part to fail, but only in this context?

Short of going through this line-by-line trying to delete each functionality and then re-run the script, I’ve got no idea how to proceed with troubleshooting here. I have no idea what can stop the identical shell line from completing all it’s functions in a different script context.

Any advice on how I could even begin to keep troubleshooting this would be much appreciated.

Thanks,

tspoon.


use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

set frontApp to application "Adobe Photoshop CS6"

export_to_server(frontApp, "3_Back_DTG_767326_1068710.psd", "/Volumes/newtees/2016/S/Sandeep Gopal/326759/Masters/", false)

on export_to_server(frontApp, saveFileName, serverFolderPath, noDialogMode) --(application, string including extension, string ending with "/", boolean)
	tell script "ROT Additions" to set debugMode to get_debug_flag()
	set directSaving to false --When false, saves to ~/tmp and moves the file from there to the server. When true, saves directly from the editor to the server.
	set recursionLayerTop to true
	if noDialogMode is "recurse" then
		set noDialogMode to true
		set recursionLayerTop to false
	end if
	
	set tmpLocation to (path to home folder as text)
	
	if directSaving is false then
		if debugMode is true then tell me to display notification with title "Creating TMP folder"
		try
			set tempFolder to (tmpLocation & "tmp") as alias
		on error
			do shell script "mkdir " & (POSIX path of (tmpLocation as alias)) & "tmp"
			set tempFolder to (tmpLocation & "tmp") as alias
		end try
	else
		set tempFolder to POSIX file serverFolderPath as alias
	end if
	set tempSavePath to (POSIX path of tempFolder) & saveFileName as text
	
	
	set oldDelimiters to AppleScript's text item delimiters
	set AppleScript's text item delimiters to "."
	set saveFormat to text item -1 of saveFileName as text
	set AppleScript's text item delimiters to oldDelimiters
	tell script "ROT Additions" to set saveFormat to convert_case("upper", saveFormat)
	
	considering application responses
		if debugMode is true then tell me to display notification with title "Locating target document for export"
		if (count of documents of frontApp) < 1 then error ("Tried to save a " & saveFormat & " file from " & name of frontApp & " while no documents were open.
		" & serverFolderPath & saveFileName)
		
		if debugMode is true then tell me to display notification with title "Clearing out target save location"
		try
			my force_smb_delete(POSIX file (serverFolderPath & "Thumbs.db" as alias)) --A PC can leave this while viewing, and it messes with accessibility.
		end try
		try --Clearing out in case the save failed last time:
			tell application "Finder" to delete (POSIX file tempSavePath as alias)
		end try
		try
			my force_smb_delete(POSIX file (serverFolderPath & saveFileName) as alias)
		end try
		try
			0.1
			do shell script "rm \"" & serverFolderPath & saveFileName & "\""
		end try
		
		-- do shell script "ls \"" & serverFolderPath & "\"" --Doesn't work.
		
		
		try --PS Save and Export commands split by Format:
			if debugMode is true then tell me to display notification with title "Exporting " & saveFormat & " file"
			if saveFormat is "PSD" then
				--all PS masters, all proof thumbnails, DTG production, 
				tell application "/Applications/Adobe Photoshop CS6/Adobe Photoshop CS6.app"
					set saveOptions to {class:Photoshop save options, embed color profile:true, save spot colors:true, save alpha channels:true, save annotations:true, save layers:true}
					tell the current document to save in (tempSavePath) as Photoshop format with options saveOptions
				end tell
			else if saveFormat is "JPG" and name of frontApp contains "Photoshop" then --art preview
				tell application "/Applications/Adobe Photoshop CS6/Adobe Photoshop CS6.app"
					set saveOptions to {class:save for web export options, web format:JPEG, quality:80}
					tell the current document to export in file (tempSavePath) as save for web with options saveOptions
					delay 0.5
				end tell
			else if saveFormat is "PNG" then
				tell application "/Applications/Adobe Photoshop CS6/Adobe Photoshop CS6.app"
					set dialogsHolder to display dialogs
					set display dialogs to never
					tell the current document to save in (tempSavePath) as PNG
					set display dialogs to dialogsHolder
				end tell
			else if saveFormat is "EPS" and name of frontApp contains "Photoshop" then --channels dcs eps
				tell application "/Applications/Adobe Photoshop CS6/Adobe Photoshop CS6.app"
					set saveOptions to {class:Photoshop DCS 2.0 save options, DCS:color composite, save spot colors:true, halftone screen:true, preview type:eight bit TIFF, encoding:ASCII}
					tell the current document to save in tempSavePath as Photoshop DCS 2.0 with options saveOptions
				end tell
			else if saveFormat is "AI" then --SP-AI master, CUT master
				tell application "/Applications/Adobe Illustrator CC 2018/Adobe Illustrator.app"
					set saveOptions to {class:Illustrator save options, save multiple artboards:false, compatibility:Illustrator 13}
					tell the current document to save in (tempSavePath) as Illustrator with options saveOptions
				end tell
			else if saveFormat is "JPG" and name of frontApp contains "Illustrator" then --art preview
				tell application "/Applications/Adobe Illustrator CC 2018/Adobe Illustrator.app"
					set saveOptions to {class:JPEG export options, antialiasing:false, artboard clipping:true, quality:80}
					tell the current document to export to tempSavePath as JPEG with options saveOptions
				end tell
			else if saveFormat is "PDF" then --DV production
				set tempHFS to tempFolder as text
				set tmpHFSSavePath to (tempHFS & saveFileName)
				tell application "/Applications/Adobe Illustrator CC 2018/Adobe Illustrator.app"
					tell the current document to save as pdf in tmpHFSSavePath
				end tell
			else if saveFormat is "EPS" and name of frontApp contains "Illustrator" then --SP-AI production
				tell application "/Applications/Adobe Illustrator CC 2018/Adobe Illustrator.app"
					set user interaction level to never interact
					set saveOptions to {class:EPS save options, compatibility:Illustrator 13, overprint:preserve, preview:transparent color TIFF}
					tell the current document to save in tempSavePath as eps with options saveOptions
					set user interaction level to interact with all
				end tell
			else if saveFormat is "SVG" then --CUT production
				tell application "/Applications/Adobe Illustrator CC 2018/Adobe Illustrator.app"
					tell the current document
						set saveOptions to {class:SVG export options, responsive svg:false}
						export to (tempSavePath) as WOSVG with options saveOptions -- (tearing on order 717943 fixed by adding "simplify nondestructive" to export production files)
					end tell
				end tell
				if debugMode is true then tell me to display notification with title "Waiting for asynchronous SVG save to finish"
				repeat with saveWaitIteration from 1 to 30 --Wait for save to finish - export apparently runs on an asynchronous thread to scripting.
					tell script "ROT Additions" to if file_is_corrupt(POSIX file tempSavePath as text) then
						delay 1
						log "Waiting " & saveWaitIteration & " seconds for saving..."
					end if
				end repeat
			else if saveFormat is "TIF" then --Storm and Breeze DTG
				tell application "/Applications/Adobe Photoshop CS6/Adobe Photoshop CS6.app"
					save current document in file tempSavePath as TIFF with options {class:TIFF save options, byte order:Mac OS, embed color profile:false, image compression:LZW, save alpha channels:true, save layers:false} appending lowercase extension
				end tell
			end if
			
			--tell script "ROT Additions" to if file_is_corrupt(POSIX file tempSavePath as text) then error ("CONTENTS OF SAVED " & saveFormat & " FILE UNREADABLE. POSSIBLE CORRUPTION.")
		on error errorMessage
			if noDialogMode then error "ERROR SAVING " & saveFileName & " FILE TO TMP: " & errorMessage
			tell frontApp
				activate
				display dialog "Error creating " & saveFormat & " file in the local temp folder:
" & errorMessage & "
in
" & tempSavePath & "
DO NOT CLOSE THE FILE. If running the script again produces the same result, save it manually.
Please tell Leif or Tom about this." buttons {"The file is fine, continue saving.", "Cancel and Retry"} default button 2
				if the button returned of the result is "Cancel and Retry" then error errorMessage
			end tell
		end try
		
		--delay 1 --stopgap to see if async saving threads might be reporting completion prematurely
		
		if directSaving is false then --Move exported file to server:
			-- with timeout of 300 seconds
			try
				if debugMode is true then tell me to display notification with title "Getting config data"
				tell script "ROT Additions" to set configData to get_config_data()
				
				delay 0.1
				if debugMode is true then tell me to display notification with title "Getting local file size"
				-- set origFileSize to (do shell script ("stat -f%z \"" & tempSavePath & "\""))
				
				tell application "System Events" to set userName to full name of current user as text
				
				if debugMode is true then tell me to display notification with title "Moving file to server using '" & networkFileTransferMethod of configData & "'"
				if networkFileTransferMethod of configData is "rsync" then
					if (serverFolderPath contains "Masters") and (saveFileName does not contain "Proof") then
						set shellLine to ("rsync --itemize-changes --whole-file --backup --backup-dir=../Art/Archive --suffix=\".\"$(date +\"%m-%d-%Y_%H-%M-%S.psd\")  \"" & tempSavePath & "\" \"" & serverFolderPath & saveFileName & "\"")
						-- display dialog shellLine
						set transferResult to do shell script shellLine
					end if
				end if
			end try
		end if
	end considering
end export_to_server

Shane_Stanley · September 24, 2019, 2:53am

The first thing I’d do would be to stop trying to quote paths yourself, and let quoted form of do it for you. I’d also add some code to check the file exists, with the name you expect, after the save/export from Photoshop.

t.spoon · September 24, 2019, 3:50am

Thanks Shane, will try those things in the morning.

Personally, I do use “quoted form of,” but this handler was by another coder. I’ll make the change.

I’ll try checking that the Photoshop save is complete. However, we keep background saves disabled in Photoshop, so the script should wait on the Photoshop save step until it’s complete. More to the point, the move of the local file saved by Photoshop to the server always works; it’s the move of the file that’s already on the server to the backup folder that fails. It strikes me as odd that behavior could be caused by the “do shell script” overlapping the Photoshop save.

But I’ll test both in the morning! I’m glad for any suggestion; I’m not discounting yours, I’m just trying to think it through.

Thanks,

tspoon.

t.spoon · September 24, 2019, 3:10pm

Grrr! As usual, it comes down to a retarded oversite, total facepalm.

To be extra sure an old copy of the file can’t persist on the server (so we don’t go to manufacturing with out-of-date files), the script deletes the file on the server before it copies the new one. The old Finder archive method occured before these delete command - archive, delete, move.

Combining the archive into the move operation, the server copy has already been deleted before the archive command, and there’s nothing to archive.

my force_smb_delete(POSIX file (serverFolderPath & saveFileName) as alias)
           do shell script "rm \"" & serverFolderPath & saveFileName & "\""

So, super “duh” on my part. Thanks for the suggestions though, Shane.

Tom.