I liked Portland more in the winter than I did in the summer. In the winter it is chill and the bars feel welcoming. Portland in the summer (outside of some good restaurants) felt kind of Fisherman's wharf-y to me.
We took the ferry and stayed on a small island of Northhaven (off of Camden). Very low key, nice to bike ride around the island, one good restaurant, one good pizza place, no real bars. High point was a nice sail on a beautiful boat.
We spend 10 days every summer in West Marin at Point Reyes Nat. Seashore, and to be honest, the island kind of freaked me out wiht it's lack of wildlife. It's as if skunks and raccoons and rabbits hadn't made their way over there, and while there was commercial fishing, there weren't any sea lions and relatively few birds. It was just really very still, California was just more alive and vibrant. Maine was, I don't know, quiet and weird.
So as odd as it sounds, I'd stay on the mainland and tool around with a side trip to Acadia. (Foxes deer and beaver).
Maine can do "white trash" pretty damn hard, when it wants to. I was peripherally involved in a case involving a grisly murder in Lewiston. Lot of surly types.