Aside from the long hours, hard work, and low pay being reasons why American citizens won't be rushing to fill those vacated construction jobs, there's also the fact that construction is skilled labor. I spent about 10 years working for a plumbing contractor as a supervisor. We did nothing but new apartments. We'd have subcontractor piece work crews of maybe 4 guys that would do all the top-out and water pipe on a job. They could do an entire 8 unit floor of top-out (the drainage system) with 2 or 3 bathrooms per unit in a day. That's all the time they got between the framers standing up walls and another crew of framers returning to lay out floor trusses for the floor above. This work included drilling all the holes for drains and vents, putting all the materials in the building, installation itself, plus things like nail plates and strapping. The same top-out crew would later fall back to the more finished units and do the water pipe as well. They were also required to test everything for us to get our city inspections. The crew you had made all the difference in the world. They were elite, highly skilled workers that would require years of training to even begin to replace. Almost none of them were here legally. We also had specialized tub setters, gas pipers, ground rough crews, fixture setters. My job was pretty much to make sure the sub crews that did the actual work had materials and everything else they needed and weren't burdened by distractions like repairs or dumbass jobsite politics. Other trades like framing, electrical, HVAC, roofing work the same way and all have their sub specialties. You will never replace people like this. To me, the idea that you can take someone with no previous training, bring them to a jobsite, point to a pipe-threader and a pile of materials and have them install a bunch of gas pipe is the most absurd, disconnected notion in all of this bullshit.
I'm sure it's much the same in all sorts of work that often gets labeled as unskilled. These work forces are built over time and new people are integrated into them as necessary. You can't have abrupt absolute turnover without a complete halt to production even if you had legions of willing workers.