The Greco-Roman culture has had a tremendous influence on Western Civilization, an influence well recognized by those living in the 19th century. The reason why Poe came up with those two names.
As to the quote itself: "The glory that was Greece" refers to the many important contributions that the Greeks made to Western philosophy, science and art(among other things). They were 'the first'. Their achievements were new and original, different from all the nations living around the Mediterranean sea. "...the grandeur that was Rome" refers to the sheer size and scale of Rome, and the Roman world in general. Think of the aquaducts, the enormous bathhouses, the lavishly decorated Forum Romanum, the Colloseum. The mighty emperors dressed in purple and gold lording over most of the Mediterranean from their stunning palace on the Palatine. The Romans were heavily influenced by Greek ideas about art and science, so in that respect they were not as original as the Greeks had been. The Romans did however build on a massive scale, they meant to impress, and they ruled over a large empire for a long time, something which can not be said about the Greeks(the empire of Alexander fell apart after he died). Therefore Greece is glory and Rome is grandeur