my (more or less) cookie cutter response for new bike buyers:

Don't let anyone else tell you what bike you should be riding, try some out for yourself to find out! you could easily get on a 29er, shorter people still get the benefits of bigger wheels, but you may find that you prefer the 27.5 (since your on a road bike now I'll point out that 27.5 wheels with big tires are about the same dia. as road wheels with skinny tires, so switching from one to the other doesn't feel as strange as switching from 700cc to 26" or 29", at least for me, ymmv). Don't rule anything out until you've tried it.

Take your time. Watch out for demo days on trails near you. You'll really get a feel for the types bikes available if you can go to an event where you'll actually be able to ride different bikes on trails. Go into a shop and test ride anything that you can. Ask questions. Remember or write down what you liked/disliked about each bike so you'll know what to look out for. The range of bikes that fits into your budget is huge, but anything in that range should be solid and purpose-built, i.e. a $1500 cross country bike will be good for cross country, a trail bike would be good for average trails, etc. Sticking to the big brands is a safe bet (Giant, Trek, Spec, Santa Cruz, Norco, Kona, etc. to name just a few) but of course there are tons of awesome bikes from boutique brands too.

Try different bikes until you find one that speaks to you and feels good on the terrain you like riding on, then research to find out if other people have had issues with it. if it gets decent reviews then go for it!

P.S. moving to the bay area with an SO that permits the N+1 rule? I'm so jealous...