Greetings, wiredless… and welcome to macscripter.
I am assuming that you’ve dragged an nsmenu object from the palette in IB, and then connected it to your application/File’s Owner object as it’s dockMenu? Dock menu’s have been a source of frustration for scripters for some time. You’re not necessarily doing anything wrong, the connection to stand-alone menus is just buggy and error-prone. Even if you seem to have everything set up correctly, that connection is not honored and you will likely ALWAYS get that error.
You have a couple options to get around this. If you’re obj-c savvy, then you can program your whole menu in obj-c and it will work that way.
If you’re looking for a simpler, applescript method, try the following…
I’d recommend creating a new window to serve as a home to only your dock menu. Create a new window, and make sure it’s “visible at launch time” flag is NOT checked. This will ensure that the window is loaded at launch time, but does not ever show up on screen. Next, drag a popup button into the window. This will serve as your dock menu. Set up the popup button’s menu however you want it, and make all of the connections you normally would to your AS code. The last step is setting the popup button as the dock menu. In IB, in your MainMenu.nib window under instances, switch to outline view mode by clicking on the outline button (the one with the lines, just above the right scroll bar). Navigate through the object hierarchy and find your popup button, and then click on it’s disclosure triangle to expose the NSMenu object hidden inside of it. Then, drag from your NSApplication object down to the NSMenu in the popup button, and “dockMenu” should become a valid option for you in the connections pane of the info window. Select ‘dockMenu’ if it’s not the selected connection already, and then click ‘Connect’. Now save and rebuild the project and it should have your popup button’s menu in the dock menu, with no errors or problems. 
It is very important that although the window the popup button is in need never be shown, it can never be released (closed). ‘Closing’ a window is different than ‘hiding’ it, where a close command will remove it from the loaded objects memory and hiding it pretty much just moves it off screen. If the window is closed, either by it’s close button or programmatically, it’s connections will no longer be honored and the menu items in the dock will no longer respond. By using a dedicated window for this purpose, you can be relatively sure that the window will always be in memory, and unless you somehow leave a method for it to be closed it’s connections should be reliable. If you decide to use a window that the user has access to closing, make sure to issue a “hide” command to it rather than a close command whenever you want it to “go away”, otherwise it will be released.
Depending on the type of menu functions you want to provide, you may find that having the popup button set as a pulldown rather than a popup type is more appropriate. If set as a pulldown type, the items will not have a checkmark next to them in the dock menu showing which is currently selected. Remember also that a pulldown menu has a “hidden” menu item 1 for the title item, too, which WILL show up in the dock menu.
Good luck, and let me know if you need any further help or clarification…
j