In a stealth game the goal for the player is to avoid open conflict. A optimal play though is a ghost one where you leave no trace of your presence. No guards are harmed, no one detected you and all things are where they should be except the things you stole or the single target you killed.
Mechanics
Detection
At a most basic level a Line of sight check is used to verify if the agent can see the player. On top of that one can add support for darkness and shadows to make the player less visible if he stands in a dark area. When the player is noticed detections is often not instant but the agent will change it's alert level.
Alert Levels
Each agent has multiple levels between unaware and aware of the player. That way it can give of clues when it change level such as 'what was that' or 'must have been rats'. It also adds a small period for the player to hide better and avoids instant failures for the player.
Propagation
How the alert level or awareness of the player propagate to other agents when someone detects the player.
Disguises
By changing cloths the player can disguise as a member of a group (like an officer or a waitress) and get access to areas he was not allowed into before. Some agents might be able to see past your disguise if you spend to long time close to them.
Hiding with objects
The player can hide in boxes, storage lockers, beneath tables or behind a tapestry. Agents are not normally looking in these objects unless they are actively searching for the player in the area.
Games
Examining the Essentials of Stealth Game Design - 2014
Dishonored - 2012
Thief: The Dark Project - 1998