The main reason I'm not as worried as some about the defense is because recruiting talent is a much better predictor of success than returning starters.
Every single starting defensive back this coming season was ranked in the top 100. Jones (40), Sterns (19), Foster (24), Cook (64), Green (47). Even backups: Overshown (52).
The 2005 Texas team relied heavily on young players that hadn't contributed much or at all before that season: Charles, Ramonce, Sweed, Pittman, Kelson, Orakpo, Wright. More so on offense than defense but still. If you were going off returning production, Texas basically had nothing returning from the skills positions. It wasn't a weakness because the talent was elite. Same thing for Clemson last year. 5 of their 6 most productive offensive players were underclassmen.
People are hesitant to be confident because of how recruits turned out under Mack and Strong, but this level of recruiting is different. For every Hill or Elliott, Strong had a Jason Hall or Dylan Haines bringing the defense down. This defense does not have the same kind of holes in talent. Even the positions people are worried about (LB, DT, DE) are places where the talent level is elite.
I think it's the most talented defense in the conference and one of the five most talented in the nation. You can survive and even thrive with youth when that's the case.
Also, I think we're all pretty confident in the offense and the defense looked like the better side of the ball in the spring game. Not that it matters much, but they looked impressive.