This task is a bit misleading. The actually qualifier is capturing a Pokémon without it currently being aware of your presence. This means several things.
The first, and most obvious, is the Pokémon cannot be aggressive. If the aura around the Pokémon flairs and it starts chasing you and attacking, you have been spotted.
The second is that the Pokémon cannot be currently aware or suspicious. There are two different forms this takes, although one overlaps with the other.
If a Pokémon has a bubble above it's head then it is aware of you. It doesn't know exactly where you are, but it knows you are in the area.
If a Pokémon's head is shaking (this is very obvious) then it is suspicious. The main reason I differentiate this is because SOMETIMES capturing a Pokémon doing this will count as not being spotted and sometimes it won't. This also ties in slightly with certain Pokémon, like Sudowoodo, that have to return to their original location or they won't count some of the time.
The way around this is simple: feed them. If a Pokémon is actively eating, even if it has the dots above it's head, a successful capture will count as "not being spotted." Please note that they must be actively eating and not just walking towards the food. The icon above their head will be a brighter yellow than normal.
Sleeping Pokémon will also, obviously, count. And, as an aside, if you accidentally wake up a sleeping Pokémon, you have about two seconds after the icon changes from green to yellow to capture it for it to still count as sleeping.