It wouldn't be abuse. It could be neglect but it depends on the circumstances and what's the kids disclose.

If she was unable or unwilling to consistently provide the children with food that would be problematic and reason for an investigation to transfer to an ongoing case depending on the circumstances. It would not go straight to removal. If the kids disclose that they are always hungry and mom refuses to feed them, you might remove if that's the least intrusive option available. An in home dependency is always preferred to removing children from the home as well.

As for the thing with the dad, the case manager would probably defer to family court. Social services is interested in child safety. If the kids are not in immediate or impending danger of serious harm, case managers usually refer for services and close the case.