Potter’s side were beaten by the Premier League’s bottom club, who took the points home from Stamford Bridge thanks to a trademark free-kick from James Ward-Prowse late in the first half.
Southampton won for just the second time in the league since October, leaving hapless Chelsea’s winless run in the league at four matches.
Chelsea struggled to create clear chances and were booed off by frustrated fans at the end of an eighth goalless effort in their last 13 games.
With the pressure mounting on Potter, who was only hired to replace the sacked Thomas Tuchel in September, the former Brighton boss launched a stout defence of his credentials.
“After a 1-0 defat at home, any criticism you get is understandable,” Potter said.
“We’ve had a tough period, we’ve had lots of challenges, integrating young players into the Premier League. Whilst results don’t go your way, it can be tough.
“I’m sure there will be people out there who will think I’m the problem. I don’t think they’re right.
“I’m not arrogant enough to say their opinion isn’t worth articulating. My job is to help the team keeping working through a tough period.”
Chelsea spent more than £300 million ($361 million) on new signings in the January transfer window, but Potter is yet to turn that huge investment into an improvement in results.
Aware of the talk that he could be facing the sack, Potter conceded Chelsea’s latest loss, coming days after a 1-0 defeat at Borussia Dortmund in the Champions League last 16 first leg, was another blow.
“The truth is we took a step back in our performance in the first half. The response in the second half was good, but not good enough,” he said.
“When we’ve lost 1-0 at home to Southampton, with the greatest respect to them, it isn’t good enough. The first half was as much my responsibility as anybody’s. I thought the performance wasn’t good.”
Potter’s problems were compounded by a serious injury to captain Cesar Azpilicueta in the second half, with the defender leaving the pitch on a stretcher with an oxygen mask after being kicked in the head by Sekou Mara.
“He’s in hospital. He’s conscious, he was speaking to his wife. We are obviously really concerned. He’s in the best place and we’ll keep monitoring him,” Potter said.
“I think he was unconscious. We have to take all the precautions we need to take now. It was worrying. It was a horrible incident.”
© Agence France-Presse