So as a follow up, I went out today to take some pictures in case I had to use them in the near future. As it turns out the fence is being held in place by a line of concrete for some reason, rather than individual concrete footings per post. Further inspection, the line of fence concrete is not tied to the patio slab and there's about 2' of either dirt, rocks or garden between it. The only issue, assuming I can legally move the fence back to the property line, is they installed their ac unit right next to the fence so it would have to be moved 2' closer towards their house.
I'll definitely be looking into whether they'd have an adverse possession claim for sure before taking any more steps. They actually have had the house in the market for a few months before now so I'm not sure how this effects them vs the new owners should it sell before the property line issue is resolved.