TLDR: an archaic system with leadership hoping the solution would work itself out instead of requiring a decision. And the anti-vax folks.
There's a fairly big anti-vax population in Japan. On top of that, when the current vaccines started hitting the market, instead of buying them the Japan government sat on their hands hoping a local Japanese pharma company would develop, locally test, and have a vaccine ready soon. So when the local pharmas did not have anything ready and Japan decided to go with an overseas tested Pfizer, Moderna, etc., they were last in line to actually order the vaccines in a market where supplies had already thinned out.
Further....despite what many believe, Japan is not technologically advanced which ends up making it super inefficient. Tons of bureaucracy relying on paper only, rare integration with the internet to do anything. So, when the vaccines finally did start arriving, the only way you could get the jab was by waiting for a piece of paper in the mail called a vaccine voucher containing your special code. Then, you could call a scheduling center with your special voucher code to get your name on the list at a clinic. So if you wanted a vaccine you were at the mercy of your local ward office mailing you a voucher. On top of that, Japan was only allowing medical doctors to administer the shot. Nurses and medical assistants were not allowed to, and it got to a point where dentists were asked to volunteer to help jab people because they were technically a doctor. But a nurse couldn't. So there were some logistical issues as well simply to get a shot in the arm.
Add on top of that the ongoing Olympics controversy and half-assed emergency lockdowns without enforcement, so the population has lost all trust in the government to get anything right. Leading to overall a lot of hesitancy on who and want to believe. Sounds familiar huh.
Source: I lived in Tokyo through the COVID era and moved back to the US in April.