Niche on Sidney Street opened in 1992 on Sidney Street and shut its doors in 2005 following a police raid.
Work has now taken place to enable the building to be pulled down.
The site is being cleared as part of the creation of a new development involving office space, housing and student flats.
Emotional tributes to the club have been paid on Facebook by DJs and clubbers.
DJ Jamie Duggan said he had met his wife at the club and they now have two children.
He added: “This place gave me a platform to progress in a career that’s not gone too bad since!
“Playing here meant everything to me and I always said if I could trade all my gigs, even the biggest ones, for one last set at Niche I would, instantly.
“I’ve tried opening the place back up a couple of times in the past 10 years, even emptied the full place of all its rubble and squatters’ waste that filled two giant skips.
“It got shut down both times by council and police.
“Wasn’t meant to be I suppose but would’ve been the most memorable party I would’ve ever thrown!”
A statement on the club’s official Facebook page on Saturday said: “Sheffield has no idea what they have demolished today but they can’t take our memories and music.
“RIP to the best club Sheffield ever had!
“Gone but never forgotten.”