Prior was probably born in Middlesex. He was the son of a Nonconformist joiner at Wimborne Minster, East Dorset. His father moved to London, and sent him to Westminster School, under Dr Busby. On his father's death, he left school, and was cared for by his uncle, a vintner in Channel Row. Here Lord Dorset found him reading Horace, and set him to translate an ode. He did so well that the Earl offered to contribute to the continuation of his education at Westminster. One of his schoolfellows and friends was Charles Montagu, 1st Earl of Halifax. It was to avoid being separated from Montagu and his brother James that Prior accepted, against his patron's wish, a scholarship recently founded at St John's College, Cambridge. He took his B.A. degree in 1686, and two years later became a fellow. In collaboration with Montagu he wrote in 1687 the City Mouse and Country Mouse, in ridicule of John Dryden's The Hind and the Panther.