How can I get the name of an opened document in an application?

First, you can examine the related application and see if it is “scriptable”. Drop it onto your script editor’s icon and examine its terminology. More probably, it will accept a syntax similar to this:

tell application "Scriptable App"
	set docName to name of window 1
end tell

If your target app is not scriptable, you may try UI Scripting, and a syntax similar to this:

tell application "System Events"
	tell process "Whatever"
		set docName to name of window 1
	end tell
end tell

If you don’t find the propper syntax, try inspecting the application UI hierarchy with tools as Apple’s UI Element Inspector or PreFab’s UI Browser.

Anyway, there are some circumstances where you won’t able to retrieve the name of some elements, because they were not designed to do so.