Yoshiro Mori retracted the comments but said he wasn't getting to stand down.
The Games chief, 83, was earlier quoted as saying women talk an excessive amount of which meetings with many female board directors would "take tons of time".
The International Olympic Committee told Reuters it considered the "issue closed" following the apology.
Japan's Olympic Minister Seiko Hashimoto said she wanted to carry "thorough discussions" with Mr Mori, a former Japanese prime minister.
"The Olympics' fundamental principle is to promote women's advancement in sport at all levels and organisations in order to realise gender equality," she said, according to local media.
Mr Mori made the remarks at Wednesday's meeting of the japanese Olympic Committee. The committee board currently has 24 members, five of whom are women.
In 2019, the committee - which is liable for selecting Japanese Olympians - set itself a goal of accelerating the amount of female board directors to 40%.
"If we increase the amount of female board members, we've to make sure their speaking time is restricted somewhat, they need difficulty finishing, which is annoying," Mr Mori was quoted as saying by the Asahi Shimbun newspaper.
"We have about seven women at the organising committee but everyone understands their place," he said.
Mr Mori is known in the country for a string of gaffes and undiplomatic statements he made while in office from 2000 to 2001.
His latest comments triggered anger on social media, with the hashtag #Moriresign trending on Twitter.
"Shame. It's time, get out," one user tweeted, while another urged athletes to boycott the Games if Mr Mori kept his job.
Mr Mori told Japan's Mainichi newspaper that female relations had also lambasted him.
"Last night, my wife gave me a thorough scolding. She said: 'You've said something bad again, haven't you? I'm going to have to suffer again because you've antagonised women,'" he said.
"This morning, my daughter and granddaughter scolded me as well," the paper quoted him as saying.
At a press conference on Thursday, he was asked on what basis he said women board members were too long-winded.
"I don't talk to women that much lately, so i don't know," Mr Mori replied. Responding to questions in parliament, Prime Minister Yoshihide Suga said the remarks should not have been made.
The Tokyo Olympics organising committee - which has 36 executive board members - is tasked with "ensuring the successful delivery of the Olympic and Paralympic Games", which are delayed by a year due to the coronavirus outbreak.
The committee takes its members from various organisations including the japanese Olympic and Paralympic Committees, the Tokyo metropolitan government, and Japan's national government.Dr. Dhillon Randeep
Angad Vir Singh Bajwa missed two focuses out of 75, to be set eleventh on countback in Men's Skee..
Mirabai Chanu's silver decoration on Day 1 of the Tokyo Olympics gave India its ideal beginning a..