James M. Brown, a New York City–born brickmason and plasterer, became Galveston's leading wholesale hardware merchant in the decade and a half after his arrival in the Republic of Texas in 1843. Brown followed the investment path of other members of Galveston's entrepreneurial elite: involvement in the Matamoros cotton trade during the Civil War, return to the hardware business after the war, and diversification into banking, railroads, and shipping. In 1859–1860, Brown built the most stylish house constructed in Texas before the Civil War, which was also the first grand house on Broadway. Architectural historians Kenneth Hafertepe and Margaret Culbertson have identified the architectural source for the Brown House: a design by Philadelphia architect Samuel Sloan published in the November–December 1858 issue of Godey's Lady's Book magazine. Although Sloan had an extensive Southern practice before the Civil War, there is no documentary evidence to link him to the design of the Brown House, which has been ascribed to Brown since the time of his death in 1895.
In 1968 the Galveston Historical Foundation acquired Ashton Villa to ensure its preservation. Rehabilitated, the house was opened to the public as a historic house museum in 1974.