You don't seem to understand what "Sentry" or "Guard" is supposed to do. When idle, when either command is active, the ship(s) in question do nothing. When the given trigger wakes them up, they are ready to recieve orders. They take no actions until they recieve orders (I believe thats what they are intended to do, like in the civilization series).
The reason for this is, in DL original, when you finish giving orders to one ship, the game would automatically select the next ship (with remaining moves) for you. The "Sentry" and "Guard" commands give you a way to skip over certain ships until the given triggers are meet (like enemy ships in sensor range for the guard command).
...However, whether I try to order a fleet or an individual ship to "Sentry" duty, the check box is nullified after I click the "Turn" button and any foreign ships in range are ignored...
Yes they are ignored. You have not given any orders to the ship in question. They in no way are supposed to attack without the orders.
After giving it some thought, you seem to have the opposite problem I do. You seem to want some command, lets call it "Partol", which would automatically attack any ships that it can see. I on the other hand want a command, I call it "Fortify", which would cause a ship to remain idle and would have no triggers to wake it up (it would make it easier to protect starbases from attack).