Panneerselvam, 63, was elected leader of the AIADMK legislature party on Sunday evening and succeeds J Jayalalithaa, who was sentenced to four years in prison Saturday for indulging in corruption.
Several AIADMK legislators were also sworn in as ministers.
A farmer from Theni district, Panneerselvam, or OPS as he is known in the party, is a staunch loyalist of Jayalalithaa.
He entered politics in 1996 and became the head of a local body. In 2001, he was elected to the assembly from Periakulam and became the minister for public works department.
He become the chief minister of Tamil Nadu in 2001 in similar circumstances. He was also leader of the opposition when the AIADMK was voted out in 2006.
Panneerselvam and a few other AIADMK leaders met Jayalalithaa in the Bangalore Central Jail earlier Sunday.
A special court in Bangalore Saturday sentenced Jayalalithaa to four years in prison and fined her Rs.100 crore for amassing wealth illegally when she was the chief minister in 1991-96.