It takes more calories to digest some foods than others, some of the food will never be completely digested and you'll poop out the calories. If you eat 100 calories of white bread, you get more calories than if you eat 100 calories of whole grain bread.

What you are saying is a bit misleading. The caloric content listed on the box doesn't refer to the total chemical energy in the food, but specifically to how much energy a person would get by metabolizing that food. To quote Wikipedia:

Food energy is chemical energy that animals (including humans) derive from their food and molecular oxygen through the process of cellular respiration.

For example, cellulose stores a lot of chemical energy (e.g. you can burn it), but because humans can't digest it, its caloric value for us is zero. As a result, as far as getting energy from food 100 calories is 100 calories is 100 calories. Where the food starts to matter a bit is that our body needs to put in energy to digest food, called the thermal effect of food (TEF). For mot foods the TEF is pretty low coming in at around 10%. Even for something like celery, the TEF only jumps to 20%. In other words, if you ate 100 kcal of celery your body would absorb 80 kcal, while for 100 kcal of bread your body would take in something closer to 90 kcal. This difference is not zero, but it's not really huge either.