Known as Milne House, the building has been abandoned since 1992. At its peak, Milneford Mills was composed of up to 16 buildings, including woollen mills, sawmills, workshops, barns, and homes. When it proved difficult to cultivate crops there, he shifted his focus to the Don River Valley and eventually founded Milneford Mills around 1846. Milne House Garden Club, which used to meet at the site of the Milne House building, lost to fire, now meets in a dedicated clubhouse space at the Toronto Botanical Gardens. Nearby, the older Milne House still stands, and whether it will ever be restored or reused remains a subject of ongoing debate.