Sampling methods differ depending on the kinds of species you are trying to catalog. For example, the way you count plants and animals is different. Plants don't move, so you have as much time as you please to conduct the survey. Animals do move, so time is a factor to consider. The longer you take to count an animal the better chance you have of counting the same individual twice as it moves through its habitat. Of course this concern applies to animals that move relatively quickly - faster than the researcher. A snail in this context behaves more like plant. Scientists might also consult with locals, and look through historical records to determine if there are any other species their sampling methods might have overlooked.
Generally we use a number of characteristics and traits to categorize individuals into different species. Scientist look at DNA, genes, external and internal morphology, behaviour, and whether or not individuals can have sex and produce viable offspring. Scientists will go into the wild and collect as many specimens as they can. Then they will catalog all the species they find. After a while you keep collecting the same ones (the most common) and fewer new species are discovered. At this point you can develop a species richness curve which helps you figure out with pretty good accuracy, depending on your sampling methods and efforts, how many news species remain undiscovered in that habitat.
If you want to figure out how many individuals of species X there are then scientists again will take different approaches depending on the species in question (e.g. plants vs animals). For animals, one would ideally conduct surveys throughout the year - particularly before and after the reproductive season. This helps researchers construct life tables, understanding how many individuals are born in each year, and how many survive each year until sexual maturity. Here are four basic methods, and a final less well known sampling method that scientist use:
Total Counts: An expensive, time consuming method wherein each individual within the population is counted. These methods take a lot of effort and man power. So unless the population is very small, or very isolated (say to a small island) than this method is not very effective cost-benefit wise.
Incomplete Counts: Wherein only a part of the population is counted, and the rest is estimated. Different methods are used depending on the type of species. One common way is to take the habitat where you think the species is found and divide it into a grid, also known as quadrat sampling. You randomly choose a certain % of grids to survey and then extrapolate how many individuals you find in those grids to the larger area. Another way is to trace lines through the habitat - or transects - that you walk along counting all the individuals of your target species that you see. You then can extrapolate to the larger available habitat to get an estimate of the number of individuals. Another way is to get citizen scientists to report the number of individuals they see during a fixed period of time in a certain location. City christmas bird counts are a great example of this in sampling method action.
Indirect Counts: Sometimes animals are very difficult to see visually, so we use something else to tell us how many might occupy a habitat. Scat, prints, fur, counting dens or nests...These methods are less accurate because they don't really tell us how many individuals there (only a presence/absence) unless we have more information to go off of. For example, robins might lay 3-5 eggs per nest, by counting nests we might estimate the number of eggs...but what % of those hatch? and of those hatchings how many survive to fledgling stage? For that we need more information on their life-history.
Mark and Recapture: The classic mark and recapture but this only works for certain species (mostly fish and birds). Scientists catch a number of individuals, tag and release them, then recapture some a certain amount of time later. The proportion tagged can be used to determined the real number. So say you catch 100, and tag and release them, then a few months later you catch 100 and 5 have tags on them. You can assume about 5% of the population was tagged, so originally the 100 was 5% of population. Therefore there is about 2000 total. Some species do not respond well to being tagged. For example, the stress of tagging kills them, tagged individuals were more likely to be captured in the first place, and tags of course might fall off giving inaccurate results.
Illegal Markets: Finally for highly endangered but commercially valuable species (e.g. those whose pelts or horns would end up on the black market, or those species bound for the illegal pet trade) scientists might also try and estimate how many individuals are entering those markets by pretending to be potential buyers. They literally go to local markets and count how many gorilla hands are on display, or they test to see whether the rhino horn powder being sold is fake or real...if they count 100 genuine snow leopard pelts coming out of Pakistan each year it gives them an idea of how hard it is to find and kill them especially if that number is going up or down. Down suggests that either the pelt has lost value and demand has gone down (unlikely) or that the animal has become more rare (more likely).