Said very respectfully, this is pretty much all wrong :).
"Swinging hard" doesn't mean that you're at all putting energy from your body into the club in the most efficient manner. Doing pushups isn't the best way to increase golf speed, either.
Golf is a rotational game, played from the ground up. Watch the guys that hit it a long way, especially ones who aren't huge - Justin Thomas, Rory, Rickie. They all end up on their tippy-toes at impact, because they're starting their swings from the ground up. They also all finish with their chest AND belt buckle facing the target itself, or even LEFT of the target, because they've rotated so far and so hard. Think about a spiral staircase - you're doing mostly a rotational thing to go up, but you still gotta push up on each stair. It's a pretty good picture for how you unwind from the top of a backswing to get the most power - push DOWN into the ground, then unwind going UP. Bryson's swing has some unique technical bits, so don't start there, but go find a slow-mo of JT or Rory's golf swing. Consider Rory to be the most technically sound driver swing for the modern era - copy it, and you'll do fine (of course, your wedge game will suck, but that's not what you asked).
In terms of workouts, it's legs, legs, legs, core, more legs, back, legs, shoulders, core, and then more legs. Push ups, bench press, incline press, all gonna slow you down.