I wish you'd stop talking about Los Olivos. It's terrible, and there's no reason to go there. So just don't, and don't even mention it to anyone.
SB is not the most expensive place to live in the US or even California (that would most likely still be on the SF peninsula / Palo Alto area), but it's a different demographic as well, with very many extremely wealthy retired people.
I think that whole Central Coast area is what most people really want when they think of a California dream. The weather and beaches are amazing, yet it's still remarkably easy and quick to get remote and rural quickly. You feel kind of isolated, but if you're really jonesing a big-city experience, LA begins not too far away.
I agree the wine region is probably tops - at the very least it's a totally different vibe to visit compared to Napa - but they also make some darn fine grape juice as well. There's a lot to love between SB and Paso Robles. That region is really underrepresented in any shops or restaurants in the Bay Area, unfortunately. It was only recently I figured out how much good stuff there is (beyond just Hitching Post, which is relatively easy to find).
The hardest thing moving here from Texas (and 'here' can be SD, LA or SF) is re-calibrating not only what you will pay in a mortgage, but also what home looks like. If a single/not-single guy with no kids feels he needs a 4BR house, then it's going to be a shock to the system. You quickly recalibrate what you really, really need. We underwent a serious bloodletting when we moved to the Bay Area - gutting our possessions, throwing out half of our clothes (no closets big enough for them anyway), and downsizing the amount and size of our furniture. Simplify, and decide what you really need. Those that figure it out tend to never leave, though I've known many people from Texas who came out this way and never adjusted, and didn't stay long. Remember that your home doesn't need to be a palace, because you're going to be outdoors most of the time anyway.