UI Scripting - Cannot target buttons as an earlier OS?

Scripting Forums AppleScript | Mac OS X
#1

I REALLY need to beg for help here. I know very little about UI scripting. The script below once worked using Java on 10.5.8. But now on 10.8, it appears as if the UI elements are not being targeted as they once were. For example if, if I were to click UI button 1 of window 1 it is clicking the red button on the upper left corner. Can anybody shed some insight and how I can recode this to target the UI elements within the java window? I can attach a link to what the java window and its buttons look like if that would help?

And once again, thank you all.

tell application "System Events"
	tell application process "java"
		set frontmost to true
		set mainWindowName to ""
		set windowNumber to 0
		repeat while mainWindowName = ""
			set windowNumber to windowNumber + 1
			set windowName to title of window windowNumber
			if windowName begins with "Digital Asset Client" then set mainWindowName to windowName
		end repeat
		
		set text item delimiters of AppleScript to ""
		
		set userID to ""
		
		set testString to characters 38 thru 44 of mainWindowName as text
		
		set theCount to 1
		
		set testChar to character theCount of testString
		
		repeat while testChar ≠ " "
			set userID to userID & testChar
			set theCount to theCount + 1
			set testChar to character theCount of testString
		end repeat
		
		set userFile to ((path to current user folder as text) & ".currentuser")
		try
			set userData to open for access file userFile with write permission
			set eof userData to 0
			write userID to userData starting at eof
			close access userData
		on error
			try
				close access file userFile
			end try
		end try
		
		
		
		
		set ANA to "ana"
		delay 0.1
		perform action "AXRaise" of window mainWindowName
		
		
		----------------------
		----------------------
		----------------------
		-->>> the line below no longer works
		set value of attribute "AXFocused" of UI element 5 of window mainWindowName to true
		
		---- thel line above yields this error:
		--System Events got an error: Can't get UI element 5 of window "Digital Asset Client® - Logged in as jod - John Doe - RegC-GRX - null" of application process "java". Invalid index.
		keystroke tab using {shift down}
		delay 0.1
		keystroke ANA
		delay 0.2
		keystroke tab
		delay 0.1
		
	end tell
end tell
#2

Try to insert three extraneous instructions

    -->>> the line below no longer works
properties of ui elements of window mainWindowName
log result
error number -128
       set value of attribute "AXFocused" of UI element 5 of window mainWindowName to true

I asume that you will discover that the window doesn’t contain five UI elements.

Yvan KOENIG (VALLAURIS, France) vendredi 17 juillet 2015 20:39:56

#3

Thank you Yvan,
Adding your lines of code somewhat confirms what I was seeing. Using 10.8 it’s only detecting the red, yellow, green buttons (close, zoom, and minimize) buttons. Whereas in 10.5.8 it was detecting all of the UI elements within the window itself. which is what I want to target but no longer can?

Results of script run on a Mac running 10.5.8 (Nice Results)

get properties of every UI element of window “Digital Asset Client® - Logged in as jkr - Jeff R - Admin-1 - null” of application process “java”
{{size:{14, 16}, focused:false, description:“close button”, subrole:“AXCloseButton”, minimum value:missing value, enabled:true, role:“AXButton”, name:missing value, value:missing value, selected:missing value, class:button, title:missing value, help:missing value, position:{367, 572}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{14, 16}, focused:false, description:“zoom button”, subrole:“AXZoomButton”, minimum value:missing value, enabled:true, role:“AXButton”, name:missing value, value:missing value, selected:missing value, class:button, title:missing value, help:missing value, position:{409, 572}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{14, 16}, focused:false, description:“minimize button”, subrole:“AXMinimizeButton”, minimum value:missing value, enabled:true, role:“AXButton”, name:missing value, value:missing value, selected:missing value, class:button, title:missing value, help:missing value, position:{388, 572}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{400, 16}, focused:false, description:“text”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXStaticText”, name:“Asset Name:”, value:“Asset Name:”, selected:missing value, class:static text, title:“”, help:missing value, position:{365, 599}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{252, 27}, focused:true, description:“combo box”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXComboBox”, name:missing value, value:missing value, selected:missing value, class:combo box, title:missing value, help:missing value, position:{768, 618}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{101, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Print PDF”, value:missing value, selected:missing value, class:button, title:“Print PDF”, help:missing value, position:{397, 677}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{104, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Duplicate”, value:missing value, selected:missing value, class:button, title:“Duplicate”, help:missing value, position:{510, 677}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{93, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Rename”, value:missing value, selected:missing value, class:button, title:“Rename”, help:missing value, position:{626, 677}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{84, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Delete”, value:missing value, selected:missing value, class:button, title:“Delete”, help:missing value, position:{731, 677}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{164, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Instructions/Notes”, value:missing value, selected:missing value, class:button, title:“Instructions/Notes”, help:missing value, position:{827, 677}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{110, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Check Out”, value:missing value, selected:missing value, class:button, title:“Check Out”, help:missing value, position:{502, 648}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{75, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Find”, value:missing value, selected:missing value, class:button, title:“Find”, help:missing value, position:{624, 648}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{88, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Extract”, value:missing value, selected:missing value, class:button, title:“Extract”, help:missing value, position:{711, 648}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{75, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Info”, value:missing value, selected:missing value, class:button, title:“Info”, help:missing value, position:{811, 648}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{397, 28}, focused:false, description:“combo box”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXComboBox”, name:missing value, value:missing value, selected:missing value, class:combo box, title:missing value, help:missing value, position:{368, 617}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{258, 16}, focused:false, description:“text”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXStaticText”, name:“Asset Type:”, value:“Asset Type:”, selected:missing value, class:static text, title:“”, help:missing value, position:{765, 599}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{354, 16}, focused:false, description:“text”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXStaticText”, name:“Local Assets:”, value:“Local Assets:”, selected:missing value, class:static text, title:“”, help:missing value, position:{365, 719}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{152, 16}, focused:false, description:“text”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXStaticText”, name:“Location:”, value:“Location:”, selected:missing value, class:static text, title:“”, help:missing value, position:{719, 719}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{354, 149}, focused:false, description:“scroll area”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXScrollArea”, name:missing value, value:missing value, selected:missing value, class:scroll area, title:missing value, help:missing value, position:{365, 735}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{146, 22}, focused:false, description:“combo box”, subrole:missing value, minimum value:missing value, enabled:false, role:“AXComboBox”, name:missing value, value:missing value, selected:missing value, class:combo box, title:missing value, help:missing value, position:{722, 735}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{146, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:false, role:“AXButton”, name:“Check In”, value:missing value, selected:missing value, class:button, title:“Check In”, help:missing value, position:{874, 722}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{146, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:false, role:“AXButton”, name:“Discard”, value:missing value, selected:missing value, class:button, title:“Discard”, help:missing value, position:{874, 757}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{0, 0}, focused:false, description:“text”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXStaticText”, name:missing value, value:“”, selected:missing value, class:static text, title:“”, help:missing value, position:{871, 719}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{156, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Browse Hierarchy”, value:missing value, selected:missing value, class:button, title:“Browse Hierarchy”, help:missing value, position:{1034, 624}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{156, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Queries”, value:missing value, selected:missing value, class:button, title:“Queries”, help:missing value, position:{1034, 663}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{156, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Add Asset”, value:missing value, selected:missing value, class:button, title:“Add Asset”, help:missing value, position:{1034, 702}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{156, 29}, focused:false, description:“button”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXButton”, name:“Add Container”, value:missing value, selected:missing value, class:button, title:“Add Container”, help:missing value, position:{1034, 741}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{0, 0}, focused:false, description:“text”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXStaticText”, name:missing value, value:“”, selected:missing value, class:static text, title:“”, help:missing value, position:{1034, 594}, orientation:missing value, entire contents:{}, maximum value:missing value}, {size:{463, 17}, focused:false, description:“text”, subrole:missing value, minimum value:missing value, enabled:true, role:“AXStaticText”, name:“Digital Asset Client® - Logged in as jkr - Jeff R - Admin-1 - null”, value:“Digital Asset Client® - Logged in as jkr - Jeff R - Admin-1 - null”, selected:missing value, class:static text, title:missing value, help:missing value, position:{545, 570}, orientation:missing value, entire contents:{}, maximum value:missing value}}

Results of script run on a Mac running 10.8 (Undesireable Results)
get properties of every UI element of window “Digital Asset Client® - Logged in as jkr - Jeff R - Admin-1 - null” of application process “java”
→ {{description:“close button”, orientation:missing value, position:{931, 450}, class:button, role description:“close button”, accessibility description:missing value, focused:false, title:missing value, size:{14, 16}, value:missing value, help:missing value, enabled:true, maximum value:missing value, role:“AXButton”, entire contents:{}, subrole:“AXCloseButton”, selected:missing value, name:missing value, minimum value:missing value}, {description:“zoom button”, orientation:missing value, position:{971, 450}, class:button, role description:“zoom button”, accessibility description:missing value, focused:false, title:missing value, size:{14, 16}, value:missing value, help:missing value, enabled:true, maximum value:missing value, role:“AXButton”, entire contents:{}, subrole:“AXZoomButton”, selected:missing value, name:missing value, minimum value:missing value}, {description:“minimize button”, orientation:missing value, position:{951, 450}, class:button, role description:“minimize button”, accessibility description:missing value, focused:false, title:missing value, size:{14, 16}, value:missing value, help:missing value, enabled:true, maximum value:missing value, role:“AXButton”, entire contents:{}, subrole:“AXMinimizeButton”, selected:missing value, name:missing value, minimum value:missing value}, {description:“text”, orientation:missing value, position:{1110, 449}, class:static text, role description:“text”, accessibility description:missing value, focused:false, title:missing value, size:{464, 17}, value:“Digital Asset Client® - Logged in as jkr - Jeff R - Admin-1 - null”, help:missing value, enabled:true, maximum value:missing value, role:“AXStaticText”, entire contents:{}, subrole:missing value, selected:missing value, name:“Digital Asset Client® - Logged in as jkr - Jeff R - Admin-1 - null”, minimum value:missing value}}

#4

Maybe I can fake it out by using “click at {x,y}”?
But I never coded for this and don’t know how to adjust for a position if the palette moves around. The application I am using to register UI elements throws back different values when the window moves.

#5

Are you owning Xcode ? It’s free on mac App Store.

It contain an application named “Accessibility Inspector 4.0.app”

With it you will be able to identify every available UI elements.

In what you get under 10.8, are 3 buttons and a text item so my understanding is that the other UI elements are in an other window. My first thought was that they may be in a sheet but this one would be listed.

(a) If I read well, a block of instructions is duplicated at the beginning of your script

 tell application process "java"
       set frontmost to true
       set mainWindowName to ""
       set windowNumber to 0
       repeat while mainWindowName = ""
           set windowNumber to windowNumber + 1
           set windowName to title of window windowNumber
           if windowName begins with "Digital Asset Client" then set mainWindowName to windowName
       end repeat

I assume that it’s an error

(b) This piece of code let me think that you know that there are more than one window available.

May you try to run this code :

tell application "System Events"
   tell application process "java"
       set frontmost to true
     set nbWindow to count window
       repeat with i from 1 to nbWindow
          properties of window i
          log result
       end repeat
end tell
end tell

I assume that you will find the wanted UI elements in one of the set of properties which will be displayed.

Yvan KOENIG running Yosemite 10.10.4 (VALLAURIS, France) vendredi 17 juillet 2015 23:13:23

#6

Hi Yvan,
The results are as follows executing your snippet of code:

10.8 using a Java UI
Java version 1.8

your code displays:
(class:window, description:standard window, orientation:missing value, position:918, 137, role description:standard window, accessibility description:missing value, focused:false, title:Digital Asset Client® - Logged in as jkr - Jeff Reumann - Admin-1 - null, size:836, 323, value:missing value, help:missing value, enabled:missing value, maximum value:missing value, role:AXWindow, entire contents:, subrole:AXStandardWindow, selected:missing value, name:Digital Asset Client® - Logged in as jkr - Jeff Reumann - Admin-1 - null, minimum value:missing value)

10.5.8 using a Java UI
Java version 1.4

(class:window, size:836, 323, focused:false, description:standard window, subrole:AXStandardWindow, minimum value:missing value, enabled:missing value, role:AXWindow, name:Digital Asset Client® - Logged in as jkr - Jeff Reumann - Admin-1 - null, value:missing value, selected:missing value, title:Digital Asset Client® - Logged in as jkr - Jeff Reumann - Admin-1 - null, help:missing value, position:615, 569, orientation:missing value, entire contents:, maximum value:missing value)

I am using a tool called Accessibility Inspector that allows me to hover over the UI and it is displaying all the buttons. I have not dabbled with Xcode. This is the first day I have been on this 10.8 Mac, and it is just unfortunate that the Applescript code that once picked up the 5th UI element is no longer working. The Mac’s Accessibility preferences “Is” checked to “Enable Accessibility for assistive devices”. if that makes any difference?

And yes, there was redundant code. Removing it yields the same error on the same line:
“System Events got an error: Can’t get UI element 5 of window "Digital Asset Client® - Logged in as jkr - Jeff Reumann - Admin-1 - null" of application process "java". Invalid index.” number -1719 from UI element 5 of window “Digital Asset Client® - Logged in as jkr - Jeff Reumann - Admin-1 - null” of application process “java”

btw, I greatly appreciate you trying to help me out. Maybe it’s a limitation to UI scripting in general based on the OS?

#7

(1) I knew that removing redundant code will not solve the problem.
I was just trying to let you with a cleaner script.

(2) I’m a bit puzzled. What need for the loop scanning window names if there is only one window available?

(3) Accessibility Inspector is supposed to tell you to which item belong the UI element you hover above.

May you take a screenshot when Accessibility Inspector hover above such an item (cmd + shift + 3) and send it to my mailbox :
koenig yvan sfr fr

I apologize but I don’t know which application is creating the target window so there is no hope that I reproduce it here.

Yvan KOENIG (VALLAURIS, France) samedi 18 juillet 2015 12:12:58

#8

Thank you Yvan,
I sent you an email with two separate screen shots. I don’t know if it will help?

-Jeff