Granby Manor was built in 1811 by Ambrose Granby, a captain based in the busy port of York. He designed the house with many windows and a back porch that provided him with views far across the river. The estate was also aligned so that the captain could easily send and receive lantern signals from the rear rooms of the house. 

Because of Granby's well-guarded relationships with both pirates and politicians, he designed his home with secret passages, which link not only portions of the house, but also the cemetery and the river. These passages also provided a safe means of concealment or escape if attacked by thieves or pirates. The passages also allowed Granby to secretly gain valuable information that he could later use to his own advantage.

In 1828, his wife bore them a son, George Granby, who spent countless hours with his father, learning everything he could. Throughout his early years, young George loved Granby Manor - especially the secret passageways.

After his father sent him to England for his university, George became obsessed with becoming richer, more successful, and more respected than any other family of Yorktown.

Before leaving England to return to America, George stayed at a summer estate, where he met a young butler named Ramsley. George asked him if he would like to come to America and serve as his butler at Granby Manor. Ramsley accepted, and in the following years proved an invaluable asset to George in achieving the social goals he set for himself and his family.

After his father died unexpectedly in a carriage accident, George bought steamboats and expanded the shipping company, making it one of the largest in the south.

While visiting Scotland with his wife, George purchased a unique cast iron fireplace and pipe organ salvaged from an estate. The locals believed that the landowner was evil and his house was cursed, so George was able to acquire the extraordinary furnishings at a very low price. The impressive Scottish organ was placed in the newly completed grand ballroom, which would soon be the setting for countless parties, masked balls, and annual Mardi Gras fêtes.

As the years progressed, the couple had a daughter and son, who filled the house with their laughter. A few years later, a third child, Edward Granby, was born. A parade of animals, pets, and childhood playthings filled the manor.

One particularly hot summer, the children were taken to visit relatives in Maryland. While there, they fell ill with scarlet fever. The two eldest children died, leaving Edward the sole survivor.

All social visits were canceled and the manor became filled with gloomy darkness. The upstairs sitting room was converted into a séance room and a gypsy medium, Madame Leota, was brought in.

George told Ramsley of his dreams for Edward's future as a governor, ambassador, or even president. To prepare for this, George wanted Edward to get the right education. Ramsley pledged to do everything in his power to make sure the boy had the proper training and education to perform his proper role in society. Ramsley saw to the boy's social training and saw that he was accomplished in a range of gentleman-like activities.

When George died the following year, Edward turned the day-to-day operation of the shipping company over to his father's trusted advisors. At Ramsley's insistence, Edward joined the social elite in Yorktown where he met the most well-bred young ladies in the region. The balls and annual celebrations began again at Granby Manor, now with Edward as the soft-spoken, but convivial host.

But suddenly, for no known reason, the parties ceased, the house fell dark, and Edward stopped his social whirl. Although no one knows exactly what happened, it was at this time that rumors about Granby Manor began to surface. Locals reported seeing the gypsy woman at the mansion from time to time, but for the most part, the house was silent. It soon became the subject of gossip and speculation.

Workers at the manor claimed that the house had a spirit of its own. Some said that daytime would turn to night and lightning would strike on a sunny day. Others claimed that brass knobs transformed into bats, wall sconces became gargoyles, cracks in the floorboards glowed, and family portraits changed into images of the dead. Some even reported seeing ghostly dancers in the ballroom, who appeared and disappeared at will.

Today, Granby Manor remains a commanding landmark, an embodiment of the grandeur, romance, legends and mysteries of another time.