The past couple of days I dug as deep as possible into AppleScript documentation and code examples. As a result I managed to automate the user interaction with a third party application (on Mac OSX) that doesn’t come with a Dictionary / Library. Text Field, button, menu are straight forward and with the help of AppleScript and commands like UI elements I can identify the hierarchy of GUI elements.
However, this 3rd party application has a date picker as shown in the attached image.
Whereas the orange triangle icons are of class “button”, I have no idea what the date boxes are.
Besides UI inspectors and attempts of reverse engineering with the use of Automator “watch me do” I also tried AppleScript commands to return all UI elements of the respective window (type dialog) as well as the attribute AXChildren etc to identify the datepicker’s GUI representation.
Result: nothing ! No UI element.
Using the app “Script Debugger” with the “UI elements” command in the context of the window that I am analysing and the date picker activated (I.e. opened), returns me the tree of UI elements and there values, and I can see the date picker’s orange triangle icons representing buttons. But that’s it. There is no other UI element representing the table of days nor an menu etc.
Could you please enlighten me and guide me in the right direction?
I mean, if a user can click the dates in this date picker, there must be a representation, right?
No, unfortunately it’s a confidential business app not authorised to reveal more than I show in the attached 2 screenshots.
“Accessibility Inspector” shows more or less the same as “UIElementInspector.app” did for me before.
Screenshot #1 shows that the orange triangle icon is a button (it’s parent is the window)
Screenshot #2 shows when I hover one of the dates, e.g. Jan 10, 2023. The context being identified is the window itself (as you can see as well from the fact, that the parent is the application itself).
However, as a human user, you can click the date boxes.
Note: this is not a web application! It is an installed app.
Having > 10 years experience in web development, where every element is represented in the DOM, I apparently lack knowledge to understand how in this app visual elements can exist without being represented in the UI hierarchy.
Can anybody with an OSX app development background or automation background explain me if elements can be hidden from “inspector / scripting eyes” by design?
It would seem that the application isn’t using cocoa UI objects in its drawing of the calendar grid. You will most likely have to calculate coordinates to send a click to.
