You could write code that would follow this training strategy using a custom training loop just as you could in any other library.

However, I have similar questions/skepticism. I’d ask the following questions.

1) Are there more than two labels? In your problem, are the labels ordinal? If not, the mean of the labels has no meaning.

2) Are your labels also defined meaningfully, localized to the patch locations? If not, it’s very dubious to try to train patchwise and use the global label assignment for the image given only local context for each patch.

3) With your patch size, are you capturing enough spatial context to where a patch is likely to meaningfully contain the object of interest? If so, related to 1), say you have a very strong detection of dog at a patch and very low scores at other patches...seems like the mean of the soft max scores is the wrong reduction. Seems like max might be more appropriate.

4) Are you sure you aren’t really trying to solve an object detection or segmentation problem using a classification network? Seems like you might be.