The core of the inn is a two-story, frame, Greek Revival, central-hall building with a Greek key frieze and elegant cornice. Numerous additions were made to accommodate its use as a public house. The entrance door is flanked by multipaned sidelights and topped with a multilight transom. Brick side chimneys served the four interior fireplaces, two on each floor. The inn was the residence of Jacob Haller and his wife, Mary Hargrove Haller, daughter of one of the county's first settlers. Haller was the town's first postmaster and operated a dry goods store and ferry business on the Brazos River. The house became known as an inn in 1852 and served as a center for mail delivery, message exchange, and a stop for post riders and stagecoaches during the Civil War. In the early 1960s, Faith Bybee of Houston, a noted early patron of historic preservation in Texas, purchased the inn, saving it from neglect.
Across the street, the W. Evan House (1853) and the Witteborg House (1853) at 4953 and 4955 Main Street, respectively, are two variations on Greek Revival. The Evan House is more formal in composition with a single-bay pedimented porch, while the full-width shed-roofed porch of the Witteborg House is a more casual, vernacular form.