How can I print a document using custom settings?

If the target application allows it (eg, Quark), just open its dictionary and explore it for printing settings.

It the target application isn’t even scriptable, you can still use Apple’s UI Scripting technology to manipulate menu items, popup buttons and so on. With the invaluable help of the UI Element Inspector or PreFab UI Browser, you can open the print dialog and browse all options.

More deeper, you could create presets and different defaults and make them available to the document using the shell’s tool defaults for scripting print defaults. Just go to your preferences folder and take a look into the files called “*”.