Striker Kieffer Moore took his goal tally for the season to six and new boy Richie Towell scored a stunning late winner as Rotherham United made it back-to-back victories in League One.
Towell's superb 89th-minute shot sealed another three points for the Millers after last week's away triumph at Portsmouth.
But Moore is the big favourite at AESSEAL New York Stadium and the 6ft 5in centre-forward was on the mark twice against the Shakers.
The 6ft 5in centre-forward struck in each half for the Millers who twice let the visitors back into the match but fully deserved their success.
Paul Warne's men were leading through Moore's 20th-minute effort and in complete control when Jon Taylor missed a 34th-minute sitter and Bury equalised through a Jermaine Beckford header seconds later.
Moore headed Rotherham back in front in the 54th minute. However, Bury levelled proceedings again after 66 minutes when Nathan Cameron headed home.
It set the stage for loan signing Towell, on his home debut, to smash the ball low into the far corner with just a minute to go to send New York into raptures.
Another new boy, goalkeeper Marek Rodak, was handed an immediate debut as Warne dropped Richard O'Donnell to the bench.
He did nothing wrong, but the real star was at the other end as Moore added to his growing fan club.
Goals: Moore 20, 54, Towell 89 (Rotherham); Beckford 34, Cameron 66 (Bury).
Rotherham United (4-1-4-1): Rodak; Cummings (Towell 40), Ihiekwe, Wood, Purrington; Potter; Williams (Forde 75), Vaulks, Frecklington (Ball 85), Taylor; Moore. Subs not used: O'Donnell, Mattock, Yates, Newell.
Bury (4-4-2): Murphy; Williams (Dobre 55), Cameron, Whitmore, Leigh; Edwards, Ince, Laurent, Bunn; Beckford, Lowe (Smith 60). Subs not used: Fason, Maguire, Reilly, Aldred, Cooney.
Referee: Chris Sarginson (Staffordshire).
Attendance: 7,848 (574).