Built as the home of merchant Charles Stedman, its side garden attests to the wealth of the owner. Large windows light the front rooms that faced onto 3rd Street and generous detail across the facade mark it as one of the great city houses of its era. Before it was ever occupied it was sold to bon vivant Samuel Powel, who in 1770 commissioned Robert Smith to adapt it to meet the desires of his new wife. The era's best architect designed the splendid carved wooden interior highlighted in red and gilt. Powel lived the life of a gentleman, made the grand tour of Europe, invested in such projects as the City Tavern
By the middle of the nineteenth century, the neighborhood had changed to commerce and manufacturing, affecting first the house's exterior and by the early twentieth century, the interior. After serving as a law office, it was converted in the early twentieth century into a factory for the manufacture of bristle brushes. When it seemed likely that the house would be demolished, important interior elements were removed to the Philadelphia Museum of Art and New York's Metropolitan Museum. The Philadelphia Society for the Preservation of Landmarks, established to preserve the house, undertook the restoration and re-creation of the original interior in 1931–1932 under the direction of Herman L. Duhring. The society continues to manage it and other houses.