Less then 10% of the South's population owned slaves, and the majority of soldiers serving in both armies were either poor conscripts or poor volunteers fighting for their home states.

And aside from that, no Sherman did not target "slavers", he targeted any civilian target that he deemed as helping the war's cause. That meant burning people's homes, farms, stealing people's foods and money and clothes, and burning down entire cities. Sherman ordered his men to execute many Confederate prisoners for no reason other then to show an example to trouble makers, and forced Confederate POWS and civilians as a source of labor to help his army's advance. Despite his talking about not committing crimes, his men got away with several dozen accounts of rape, theft and murder, ironically mostly directed towards slaves themselves. Sherman viewed slaves as nuisances and even had his troops use physical force to keep the slaves from following his army, even going so far as to force slaves to stay behind as he had his troops burn bridges across rivers since he calculated that the slaves would either go back or drown in the river trying to follow him (and many did).

If Sherman did half the things he did nowadays he'd be court-martialed and hung. If he were on stand at the Nuremberg Trials he would have been executed so quickly he wouldn't have been able to give a plea.