In 1911, Swett-Ilsley House became the first property acquired by Historic New England, just a year after its founding. The original portion, built in 1670 by Stephen Swett, was one room deep, and later additions more than doubled the size of the house. Over the centuries, the building served as a tavern, chocolate shop, chandlery, and press room, in part due to its location on Newbury's most traveled road.
Swett-Ilsley House had been brought to the attention of Historic New England founder William Sumner Appleton by an acquaintance who believed it to be one of the oldest surviving houses in the region. After the house was acquired for $2,400, nineteenth-century building layers were removed to expose the earliest architectural features. It has one of the largest fireplaces in New England, more than ten feet wide, and containing three beehive ovens.