Oli McBurnie and Yan Dhanda scored after George Baldock had put the home side in front.
Dhanda, who was making his debut after coming on as a substitute a minute earlier, scored with his first touch to snatch victory in the 86th minute.
Potter said: "I thought we did really well. It was tough but we limited them in the first half and in the second half we showed real quality and character.
"The goal knocked us and you've then got a choice, whether you cave in or respond. You don't know how it's going to go, but they kept going and I'm very pleased with them.
"All the guys who came on made a real impact. Yan can pop up and score a goal - he's got that quality.
"Oli worked really hard for the team and deserved his goal.
"I'm delighted, of course, with the result and very proud with a lot of aspects of the performance. Any win in football is hard and it's nice when the supporters go home happy."
When asked for an update on the future of Sam Clucas, who has been linked with a move, Potter said: "I've not heard anything."
Potter is keen to make further signings, adding: "We need to improve the group and the team."
United boss Chris Wilder said: "You can't defend like that against quality players. You've got to defend properly.
"It was a tight game and we started off okay. To go one-nil up on day one was a good position but the two goals were very disappointing from our point of view.
"We went to press in the corner and we didn't need to. The second goal was a really poor one from our point of view. All of a sudden, we're back-tracking.
"I think we had the best opportunities so they are fine dividing lines. The worst it should have been was a draw. A Swansea win was the least likely of the three results.
"We were up against a good side. They've got World Cup players and really good players. I don't think we should be expecting to steamroller a side that's just come out of the Premier League.
"We're still light and we're still looking. The search goes on."