Another 863 people were injured in the incident at Mina, which occurred as two million pilgrims were taking part in the Hajj’s last major rite.
They converge on Mina to throw stones at pillars representing the devil.
Preparations for the Hajj were marred when a crane collapsed at Mecca’s Grand Mosque on September 11, killing 109 people.
Six Nigerians died in that incident.
Mina, a large valley about 5km (3 miles) from Mecca, is the location of the three Jamarat pillars and also houses more than 160,000 tents where pilgrims spend the night during the pilgrimage.
The Saudi civil defence directorate said in a statement that the stampede occurred on Thursday morning at the junction of Street 204 and Street 223, as pilgrims walked towards the five-story structure which surrounds the pillars, known as the Jamarat Bridge.
The incident happened when there was a “sudden increase” in the number of pilgrims heading towards pillars, the statement said.
This “resulted in a stampede among the pilgrims and the collapse of a large number of them,” it added.
Security personnel and the Saudi Red Crescent were “immediately” deployed to prevent more people heading towards the area, the directorate said.
The hundreds of wounded have been taken to four hospitals in the area.
Amateur video and photographs posted on social media showed the bodies of dozens of pilgrims on the ground. They were all dressed in the simple white garments worn during the Hajj.
The civil defence directorate said the victims were of “different nationalities”, without providing details.
The head of the Central Hajj Committee, Prince Khaled al-Faisal, blamed the stampede on “some pilgrims from African nationalities,” Saudi-owned al-Arabiya TV channel reported.
Iran’s state news agency, Irna, said at least 43 Iranians were among the dead.
There is yet no confirmation of Nigerians involved in Thursday’s tragedy.
BBC Hausa Service correspondent Tchima Illa Issoufou, who was with some of those affected in Mina, said many pilgrims from Niger were also killed.
The UK Foreign Office said it was in contact with the local authorities and was urgently seeking more information about whether British nationals were involved.
Saudi-owned al-Arabiya TV reported that the head of the central Hajj committee, Prince Khaled al-Faisal, had blamed the stampede on “some pilgrims with African nationalities”.
But the head of Iran’s Hajj organisation, Said Ohadi, told Irna that two routes to the Jamarat pillars had been inexplicably closed off by the Saudi authorities, resulting in the build-up in pilgrims.
The stampede was the deadliest at the Hajj since 2006, when more than 360 pilgrims were killed in the same area.
The Saudi authorities have spent billions of dollars on improving transport and other infrastructure in the area in an attempt to try to prevent such incidents.
The Hajj is the fifth and final pillar of Islam, and one of the most celebrated events on the Islamic calendar. It is the journey that every able-bodied adult Muslim must undertake at least once in their lives if they can afford it.
Previous Hajj tragedies
2006: 364 pilgrims die in a crush at foot of Jamarat Bridge in Mina
1997: 340 pilgrims are killed when fire fuelled by high winds sweeps through Mina’s tent city
1994: 270 pilgrims die in a stampede during the stoning ritual
1990: 1,426 pilgrims, mainly Asian, die in a stampede in an overcrowded tunnel leading to holy sites
1987: 402 people die when security forces break up an anti-US demonstration by Iranian pilgrims