Coup de grâce is a special kind of attack that is guaranteed to kill its target. A regular attack is (automatically) converted to a coup de grâce when the target is a helpless (paralyzed or sleeping) creature with four or fewer hit dice. This can be either a melee or a ranged attack, but the attacker must be within 30 feet of the target (the same range used for sneak attacks). While a coup de grâce cannot be applied to a player character, it can be made against associates (summons, familiars, etc.) of a player character.
When a coup de grâce is made, the attack is automatically successful, and the victim always dies from the attack. The automatic success means that the attack roll is always a natural 20, and miss chances (both concealment and attacker miss chances) are ignored. (There is no threat roll, though.) After the coup de grâce hits, the target dies. There is no specific damage dealt; the victim is simply killed.
Pen and paper
Coup de grâce as implemented in Neverwinter Nights differs significantly from the pen and paper (PnP) version. In PnP, a single coup de grâce replaces all attacks in the round (rather than just one attack) and provokes an attack of opportunity from threatening foes. The PnP coup de grâce is not limited by the hit dice of the victim; instead it allows a saving throw to avoid automatic death, with a DC based on the damage dealt (assuming the damage alone was not enough to kill). This save is made more difficult by the attack automatically being a critical hit (and a sneak attack, if applicable). If the creature is immune to critical hits, the PnP coup de grâce is avoided.
One other notable difference is that a PnP coup de grâce can only be made with a ranged weapon if the attacker is adjacent to the victim, while Neverwinter Nights allows the attacker to be up to 30 feet away.