Two-goal Ben Whiteman scored a contentious 90th minute penalty to send Doncaster Rovers through to the FA Cup fifth round for the first time since 1956.
Whiteman put Rovers ahead midway through the second half with a good strike from the edge of the box.
League Two Oldham thought they had earned a replay when Peter Clarke pounced from close range after 84 minutes.
However, Rovers sealed their place in the last 16 when Clarke was penalised for handball and Whiteman slotted home from the spot.
Doncaster had the better of a tense first half which ended in stalemate.
The visitors offered more threat after the break but Whiteman’s opener was the cue for a thrilling climax to the game which saw Clarke equalise, concede a penalty and then get his marching orders.
Paul Downing and Kieran Sadlier both made their debuts as part of five changes from the side that drew with Burton Albion last time out.
Herbie Kane and Whiteman flashed shot wides, while James Coppinger’s volley forced Dan Iversen into a good save.
Oldham stayed compact and played on the break, with Chris O’Grady testing Marko Marosi’s handling following a counter attack.
The clearest opening came midway through the first period when stand-in captain John Marquis was neatly played in but, after rounding Iversen, his shot was cleared off the line by the back-tracking Clarke.
Sadlier went close to breaking the deadlock five minutes after the re-start when his curling effort from the angle of the box was brilliantly kept out by Iversen.
Jose Baxter headed just wide and Marosi had to deny Johan Branger and Chris O’Grady as the Latics started to cause more problems.
However, just when the visitors appeared to be gaining in confidence, Rovers took the lead on 68 minutes when Whiteman unleashed a shot from the edge of the box and it went in with the help of a deflection.
Marquis and Kane went close to doubling Doncaster’s lead before Chris Missilou struck the post at the other end and Callum Lang wastefully put the rebound wide.
The equaliser did arrive on 84 minutes when Branger’s freekick was blocked and Clarke reacted quickest to slam home.
But there was more drama to follow when, during the dying moments of normal time, Clarke was punished for handball during a goalmouth scramble and Whiteman held his nerve to convert from the spot.
Clarke’s eventful afternoon then saw him sent off in stoppage time after receiving a second yellow card for a challenge on Marosi.
Rovers: Marosi, Mason, Anderson, Downing, Andrew, Whiteman, Kane, Coppinger (Blair 63), Sadlier (Wilks 63), May (Wright 90), Marquis. Subs not used: Jones, Lewis, Rowe, Crawford.
Oldham: Iversen, Hamer, Edmundson, Clarke, Nepomuceno, Lang, Missilou, Sylla, Branger, Baxter (Vera 76), O’Grady. Subs not used: De La Paz, Taylor, Dummigan, Benteke, Robinson, Hunt.
Referee: Peter Bankes
Attendance: 11,260 (3,796)