Mr Scalise, 58, had struggled to gain enough votes to secure an overall majority in the chamber.
The announcement came after a last-minute meeting of Republicans in the House of Representatives on Thursday.
It is unclear who the party will now nominate for the position.
Speaking to reporters following the meeting, Mr Scalise said that "we have to have everybody put their agendas on the side and focus on what this country needs".
"This House of Representatives needs a Speaker and we need to open up the house again. But clearly, not everybody is there. And there's still schisms that have to get resolved," he said.
Despite defeating right-wing rival Jim Jordan in a secret ballot on Wednesday, Mr Scalise faced growing dissension from rank-and-file Republicans.
Efforts to sway them enough to gain the 217 votes he needed to secure a win as Speaker were unsuccessful on Thursday.
In his remarks on Thursday night, Mr Scalise acknowledged that "it wasn't going to happen".
"There were people who told me they were fine with me three days ago who were moving the goalpost and making up reasons... that had nothing to do with anything," he said. "Look, there were games being played and I said I'm not going to be part of it."
But he did vow to stay on as majority leader of his party, telling reporters that he had "a job that I love".
Compared to Mr Jordan, Mr Scalise was considered a more traditional candidate in the race. He worked his way up through the Republican party's leadership, building a reputation as an effective legislator and fundraiser along the way.
Until a Speaker is chosen, the House of Representatives is unable to pass any bills or approve White House requests for emergency aid - including for Israel.
Additionally, it means that Congress is unable to pass any spending bills that would allow the government to avoid a potential shutdown in mid-November.
Before Mr Scalise's withdrawal from the race, House Republicans had made a number of proposals to end the turmoil on Capitol Hill.
One option would see the House's Acting Speaker, Patrick McHenry, given additional powers for a temporary period that would allow the chamber to function, and, crucially, avoid a government shutdown.
Doing so, however, would require opposition Democrats to cooperate. Lawmakers on Thursday were split about whether the move would be a viable solution.
A second option would see Democrats come together with Republicans to successfully elect a consensus candidate.
Some Democrats have also suggested that some Republicans could cross the aisle to vote for Minority Leader Hakeem Jeffries to become Speaker - although that option is seen as a long-shot by many on Capitol Hill.
The powerful Speaker position has now been vacant for 10 days since Kevin McCarthy was ousted by hardliners within the Republican party who voted against him after he made a deal with Senate Democrats to fund government agencies.
Many of those same lawmakers refused to vote against Mr Scalise on Wednesday and Thursday.
Following the announcement, some lawmakers suggested that they hoped that Mr Jordan - who had backed Mr Scalise after Wednesday's ballot - would now become the nominee.
When asked if he would support Mr Jordan after dropping out of the race, Mr Scalise replied that "it's got to be people that aren't doing it for themselves and their own personal interest".
Mr McCarthy, for his part, said only that House Republicans have to "figure out their problems, solve it and select the leader".
By Bernd Debusmann Jr
BBC News, Washington