Symonds was involved in an altercation with India's Harbhajan Singh on day three of the Test, during which the bowler allegedly called Symonds a "monkey". Harbhajan was handed a three-match ban, which was later overturned on appeal. Symonds only played nine Tests after the scandal broke, ending his career after a tally of 26 matches.
"That [the scandal] was the start of the end for Andrew Symonds," Ponting said in a television interview. "His career spiralled downhill after that because I know for a fact that he didn't feel like he could trust the people he needed to trust."
Ponting said as a captain of the Australia team and as a member of the society, he had to take a stand against racism to make sure it didn't happen around him.
"I knew how powerful Indian cricket was, everyone did," he said. "But at the same time, I know how powerful racism is in society as well.
"I was told every day, every week by Cricket Australia that we had to stamp it out of the game. And [when] it did [happen], I had to make a stand against it. I followed the instructions to the letter. I did everything I was expected to do. I know there are a lot of administrators in Cricket Australia who can't say the same thing."
Speaking on Mickey Arthur, the former Australia coach under whom Ponting played as well, he said Arthur lacked the ingredients Australia needed from their coach at that time.
"Mickey might be one of the great cricket coaches in the world, but he wasn't the guy, the personality, the coach that Australian cricket needed at that time," he said.
"We needed to bring the old, hard-nosed Aussie way back, and that certainly wasn't the way Mickey was trying to coach our team."
Arthur was involved in a controversy during the 2013 tour of India for sending four players back to Australia in the middle of the tour. He was sacked 16 days prior to the summer Ashes and was replaced by Darren Lehmann.