Gen Katumba Wamala said the decision to withdraw Ugandan troops by December 2017 was taken because of frustration with the Somali army and military advisers from US, UK and Turkey.
Uganda provides the biggest contingent of soldiers to the AU force, Amisom.
Amisom has helped push al-Shabab out of most of Somalia's major towns.
But the militants continue to mount deadly suicide bombings and guerrilla attacks.
Uganda joined Amisom in 2007 and has just over 6,000 troops in the 22,000-strong force.
Gen Wamala told the BBC's Catherine Byaruhanga that it was time for the Somali army to take over its own security, but that it was not happening.
Countries including the US, UK and Turkey have been training Somali soldiers and police officers.
Kenya, Ethiopia, Djibouti and Burundi also have troops in Amisom.
In May, Kenya also threatened to withdraw its troops from Somalia if the United Nations did not offer funding to support the mission.
Our correspondent says some have questioned Amisom's effectiveness because it has struggled to secure areas outside of the big towns and has been plagued with accusations of corruption.
Gen Wamala says Uganda could pull out before its deadline of December 2017 if another country is found to replace it.(BBC)