Many Greeks object to the country of the same name calling itself Macedonia, saying it implies a territorial claim on Greece's northern Macedonia region.
Protesters oppose Greek government proposals on resolving the issue.
Celebrated Zorba The Greek composer Mikis Theodorakis, 92, was among those addressing the crowds.
Anarchists daubed red paint over his home on Saturday but he was unfazed, declaring, "I am calm and ready."
The man who has focused on one word for 23 years
What's in a name?
Macedonia country profile
How high are passions in Athens?
Demonstrators carrying Greek flags chanted "hands off Macedonia" and "Macedonia is Greece", as they assembled in Syntagma Square outside parliament.
Greek composer Mikis Theodorakis gives a speech during a demonstration to urge the government not to compromise in the festering name row with neighbouring Macedonia, at the Syntagma Square in Athens, on 4 February 2018Image copyrightAFP
Image caption
Mr Theodorakis urged the crowds to hold out for an acceptable solution to the dispute with the "Skopjans", as he termed the northern neighbour
"Macedonia was, is and will forever be Greek," Mr Theodorakis told the huge crowds, adding that any suggestion of a name to resolve the dispute must be put to a referendum.
"If a government considers signing on behalf of our country... there is no doubt it must first ask the Greek people."
He said the neighbouring northern state was "illegitimate".
Greek Foreign Minister Nikos Kotzias recently received death threats when he said he expected the dispute to be resolved within months.
Greeks protesting in Athens, 4 February 2018Image copyrightEPA
Image caption
It is the second such protest inside two weeks
"Here are the borders," said protester Rania Mainou from Xanthi in northern Greece. "This is Macedonia. Here, these are Slavs, they are not Macedonians, we are Macedonians. Macedonia is Greek, no one can take this name, no one can use it."
It is the second such protest in a fortnight. Some 90,000 demonstrators rallied in Thessaloniki, the capital of the Macedonia region, on 21 January.
Organisers of Sunday's protest estimated that 1.5 million people had attended but police said turnout was less than one tenth of that.
How long has this row been rumbling?
The dispute has simmered since Macedonia gained independence from Yugoslavia in 1991 and it has held up its attempts to join Nato and the EU.
Greece's left-wing Syriza government says the issue is a diplomatic obstacle it wants resolved and has proposed agreeing to a composite name for the country which would include the word Macedonia but ensure a clear differentiation from the Greek region.
Macedonia argues that its people can be traced back to the ancient kingdom of Macedon, once ruled by Alexander the Great, and that the name "Macedonia" is therefore the logical option.
However, Prime Minister Zoran Zaev said last month that Macedonia would change the name of its airport from Skopje Alexander the Great airport, to show good will.
The Greek Orthodox Church backs the campaign to stop Macedonia using any variant of the name.
What names are being mooted?
In organisations such as the UN, where talks have been under way, the country is officially known as the Former Yugoslav Republic of Macedonia (FYROM)
At home, the Macedonian government calls the country it administers simply "Republic of Macedonia".
UN mediator Matthew Nimetz has suggested alternatives such as "Republic of New Macedonia".
A proposal to name it "Republic of Macedonia-Skopje" was reportedly accepted by Greece but rejected by Macedonia.