Sight

The basic way to detect if the agent see something is to use a raycast. To many raycast can get expansive so there are then some extra work to avoid the raycast if possible.

Filter

The first step is to only test for things that matter to the agent. Friendly targets can be unimportant for example and the agent only looks for hostile ones. Or if it is a stealth game it might only look for hostile and dead friendly targets. Things the NPC care about seeing is called perceptible object.

View Settings

Common setting for vision is a field of view and a sight range. If an object is outside that parameters no further test need to be performed.

Potentially visible set

The world database might already provide some sort of culling data for rendering and that can be used by the agent for quick rejects. If the area the agent is in can not possible see the target there is not need to test any further.

Queue Raycast

A final way to avoid to many raycast is to spread them out over time. Put the ones that need to be performed into a buffer and work off a number of them each frame.

Raycast

The final test is one or more raycast from the agent towards the target. If nothing else is hit along the way the target is visible. More then one test might be needed depending on the game. In a stealth game the player might be able to hide behind a crate so many rays need to be used to see if he is visible.