You haven't wasted years of your life. That's a huge disparity in size and strength, and you're not very well versed in grappling. Striking is only one component of fighting, and it doesn't exactly translate to groundwork. If you were a wrestler, would you expect to win a boxing match?
Plus, like someone else mentioned, most guys would've lost this fight too. Except you've actually applied your training in real-life, which is all that really matters. This whole 'scenario' with 300 pound guy is one step removed from BUT WHAT IF YOU GOT JUMPED BY 2 GUYS AND ONE OF THEM HAD A SHOTGUN
Also, I find your brother and his friend super childish (at best), but there is one kernel of truth in the condescending 'lol karate dont work gurlz kant fite' shtick: I don't think you should be trying to "win" a self-defense altercation. More often than not, you should be trying to escape. Footwork is especially important here. Thai clinching someone that has 10 inches of height and over a hundred pounds on you might not be the best choice That doesn't invalidate all your training - it just means that there are certain circumstances where different tools are more important.
Your brother's 'argument' (if you can even call it that. it's mostly i-know-more-than-you, youre wasting ur time lol) is based on quite a few dangerous false premises. Granted, martial arts aren't going to give you superpowers. They aren't going to magically erase physical advantages. But these "useless sport gimmicks" are called...familiarizing yourself with aspects of actual fighting. Real-world altercations don't suddenly morph into bizarro realms where 'sport specific' training suddenly has no relevance. A left hook is...still a left hook. Sweeps and technical standups still... exist. The more skilled you are at these things, the closer you will come to bridging the strength/size gap.
Which brings me to another point: in the grand scheme of things, your brother doesn't really know anything about fighting. He isn't some all knowing authority. Or are we going to ignore the entire history of combat sports and competition (judo, wrestling, sambo, etc.) just because your brother says it's 'useless'? If that's the case, I'm pretty sure your brother (and his friend!) would have a couple of olympic medals, world titles, and the fame and money that comes along with these things. But, hey, maybe these martial arts are just too easy for them? I'm sure they could do it if they felt like it.
See, here's the thing: a fight has many variables. Your training is just a means to even out the odds - and sometimes, it won't even the odds out as much as you think it will. The heavier and stronger they are, the more skill you'll need to match them. You're meant to be discouraged, but that doesn't invalidate the fact that these techniques work. After all, you can slip dozens of punches, and one can still find its mark and knock you clean out. Does that mean it's useless?
You can have years of training and still lose. On any given day, there's always someone better than you, stronger than you. But that doesn't you're not evening out those odds. That doesn't mean a fight will go the same way every time.
Now, of course weapons can be helpful. Except your brother mentioning them is kind of besides the point. It's just a lazy strawman - because, really, is anyone actually arguing that weapons aren't hekpful? The thing is: they don't magically render unarmed combat irrelevant. Not every fight is going to be a knife fight to the death with lava encircling you and your opponent. And not every physical altercation will be against a football lineman.
Either way, learning grappling is essential to learning how to defend yourself. It's not something you can really get around. But you're doing fine, and you can only get better. You won't be invincible, but none of us are.