The land that Miller Place occupies was purchased from the native Setalcott tribe in 1664 by settlers of Setauket. The parcel also included what would become Mount Sinai, New York, an adjacent community of similar character with which Miller Place would share a variety of functions throughout its history.
While the original settler of Miller Place is unknown, the settling of the region is largely accredited to the original Miller family. In 1679, an East Hampton settler named Andrew Miller purchased a 30-acre (120,000 m2) plot. Miller was a cooper by profession, and records indicate that he had emigrated from either Maidstone, England or Craigmillar, Scotland.