I suggest we re-think some basic elements.

Bran is a green-seer - his is a world based on earthly ground things and light (sun) - wierwood trees and he uses wierwood.net

The Others are in the form of a solid water-misty-ice particle being. Their's is a world based on water and shadows and darkness, cold - like you would find deep in the ocean. Just like Bran in his cave, the Others are in their own cave in the Frostfang Mountains projecting their images into past, present (and likely future). The Others are most likely therefore IronBorn, as everything they do is based on water, a basic element. So they would have been blue-seers (or ice-seers) using frostyice.net They need water in some form - ice, mist, Winter, Snow to exist - they move as Winter moves- Winter is Coming - the King of Winter in fact. When The Others die (in physical form) they end up a puddle in the dirt. Would it not be possible that their 'spiritual being' is just like Bran, elsewhere.

The only other possibility besides being IronBorn would be if they were from the Riverlands. Not a fish-fauna, but some sort of hybrid-water-flora. But not enough has been revealed about them for this to be truly known. I suspect The Others were not Starks, nor Boltons (Red Kings) but are Greyjoys - possibly a blue-seer Greyjoy.

They didn't high-jack Bran's weirwood.net - they are operating in the same dimension using frostyice.net. The wights are that which is beneath the sea -- do a search of the text of ASOIAF and you will find what I am talking about - I am not sure if it is Patchface or another jester who refers to the Others as being "below" - but that is the reference that answers their origination.