The Kosovars would only be able to adversely posses (become legal owners of) the garden and barn if they have "open, notorious, visible, exclusive, and hostile" use of the garden and barn.

The Kosovars' use is indeed open, notorious, and visible, but it doesn't sound very exclusive or hostile.

You and your father-in-law should gather the evidence that the Kosovars' use of the garden and barn are not exclusive or hostile:

• gather evidence that you or your FIL use the garden and barn, too
• gather evidence that the Kosovars asked your FIL for permission
• ask the Kosovars to sign a $1 / year lease to use the garden and barn
  

See "Handling a Tenant's Abandoned Property in Wisconsin" for what to do if the Kosovars abandon junk on your FIL's property.