Citing survivors, the UN agency said on Saturday the vessel left the city of Zuwara with around 86 people on board.
It said high waves swamped the boat and that 61 migrants, including children, were missing and presumed dead.
Libya is among the main departure points for migrants trying to cross the Mediterranean and enter Europe.
The IOM estimates that more than 2,200 people have drowned while attempting the crossing this year alone, making it one of the most dangerous migration routes in the world.
The agency said most of the victims of the latest incident were from Nigeria, Gambia, and other African countries, the AFP news agency reports.
It also said 25 survivors had been transferred to a Libyan detention centre and were being given medical support.
Writing on X, formerly Twitter, an IOM spokesperson called the death toll for this year a "dramatic number which unfortunately demonstrates that not enough is being done to save lives at sea".
In June, at least 78 people died and another 100 were rescued after a fishing boat sank off southern Greece.
The Mediterranean crossings frequently see scores of migrants crowded onto boats too small to safely hold them.
Those making the trip are typically hoping to land in Italy before making their way to other countries in Europe, some escaping conflict or persecution, others in search of work.
More than 153,000 migrants have arrived in Italy this year from Tunisia and Libya, according to the UN's refugee agency.
On Saturday, UK Prime Minister Rishi Sunak had talks in Rome with his Italian counterpart Giorgia Meloni and Albania's PM Edi Rama on how to reduce illegal migration to Europe.
At the start of this year, Mr Sunak made stopping small boat crossings in the English Channel one of the five key priorities of his government.
BBC