Residents said they heard a ‘big crash’ at about 7am when the roof of the former Heeley National School building, which dates back more than 200 years, collapsed.
It comes just a year after conservationists issued a rallying cry calling for the old school to be saved.
The building will undergo a partial demolition on Saturday in order to ensure no other parts of it collapse.
Andy Jackson, manager of Heeley Trust, which works to restore disused buildings, fears the roof collapse could signal the death knell for the old school.
He said: “It is really sad but it is an example of what happens if you just leave an old building and don’t look after it.
“Sadly I think with the demolition we will see it being finished off.”
He added that the Trust looked into doing something with building about 15 years ago but it was in a state of disrepair even then.
It is understood the building has been derelict for some time and nobody is though to have been injured in the incident this morning.
A number of residents also took to Twitter to express their concern over the future of the building.
Annie Bethell tweeted: “Well I did wonder what the big crash was this morning but couldn't see anything.
“When will people learn that buildings need care, protection and mostly investment. I will be keeping a keen eye on what happens next.”
Beauchief and Geenhill councillor Richard Shaw tweeted: “Awful scene this morning - the roof of the historic Heeley National school on Gleadless Road has collapsed overnight.”
Officers at Sheffield City Council are now making efforts to contact the building owners.
A small part of the building will undergo demolition work tomorrow to ensure no further parts of it collapse.
Gleadless Road will be closed while the work is carried out in the morning.
The former school building dates back to 1801 and was built on the site of an earlier 18th-century establishment providing education long before it was available to the masses.
It is believed to be the second oldest school building in Sheffield, after a former school in Ecclesfield.
The Star understands there was an attempt a few years ago to secure listed status for the building but this failed because it has been altered internally.
Leaseholder CareTech Community Services said last year that the owner did not have any proposals in place for the building's future.
A spokesman for CareTech said at the time: "At the moment there are no plans for the site, other than to maintain the safety of the building as it is."
We have asked the company for comment today and are waiting for a reply.