Millbrook was settled by Quakers in the middle eighteenth century. What now comprises the village originally had two areas of settlement, Hart's Village and the hamlet of Mechanic. The area known as Hart's Village was centered around the grist mill along the East Branch of the Wappinger Creek off of Hart's Village Road. The building still stands today, and has been converted to an apartment building. The Hamlet of Mechanic is now known as South Millbrook, and is located near the junction of NY route 343 and Old Route 82 (Dutchess County route 111), where the turn monument is.

The village is from the Great Nine Partners Patent, and a street and burial ground at the south east edge of town now bear the name. Prior to the Revolutionary War, what is now known as the Town of Washington was called the Charlotte Precinct.

The Village of Millbrook, as it is known today, was developed around the Dutchess and Columbia railroad train station which was located on the village green. The train station was built in 1870 on the lands of Issac Merritt, who laid out the streets soon thereafter.