I thought it was a quaint southerner thing (tbh, still mostly do), until an older woman took extreme offense on my behalf hahaha. She got all serious and was like, "that is not your ol' lady!" She wasn't a total random stranger but we didn't really know her, she was staff at a campground we were staying at. I think she thought well of me in hindsight. That day, I googled a bit and did see there is a connotation of not getting married. My then-bf (now fiancé! lol) didn't know that, either. I was bothered by that definition and he stopped using it, but it was never meant insultingly.

I think I understand why my partner used it: his dad is super old and a widower, and used "ol' lady" a lot when it came to the woman after his wife. My partner grew up hearing it a lot.