The Spireites went down to a stoppage-time header from Peter Clarke, after having a man advantage for the entire second period.
Anthony Gerrard saw red during the half-time break, leaving his side to battle for 45 minutes a man down.
After a resolute defensive display, all of the hosts’ hard work was undone at the Proact thanks to captain’s Clarke’s stoppage-time header.
“I’m bitterly disappointed,” admitted Caldwell, who experienced defeat for the first time since taking over at the Proact.
“Sometimes in football playing against ten men is actually harder, believe it or not. I told the lads at half time that that could be the case and that they would pack their defence.
“They managed to make it hard for us to score and they were always a threat on the counter and from set pieces.”
Caldwell named a much-changed side from his first game in charge against AFC Wimbledon last time out and gave debuts to Sadiq El-Fitouri, Liam Grimshaw and David Faupala.
Everyone was understandably deflated in the dressing room after the defeat.
And Caldwell has urged the players to use this feeling as a catalyst to starve off relegation.
He added: “I’ve told them to use this feeling to drive us forward.
“There’s still a long way to go in this battle and we can take positives from the game.
“I feel like we did a lot right. We got the ball into good areas, into wide areas.”
The Spireites have scored just once in their last eight and that stat will have to change if they are to stay up.
Caldwell said: “We need to be better in the final third.
“We need to be more aggressive, you can’t score a goal unless you’ve got real hunger and desire to do so.”
The Spireites were caught napping early on as seconds after the whistle went Ryan McLaughlin hit an effort just wide.
New boy Faupala had an early shot deflected for a corner as the hosts began to feel their way into the game.
The hosts held a lot of the ball without bursting through the Latics’ backline.
On the stroke of the half hour the Spireites had their best chance.
Welsh striker Ched Evans played wing-back El-Fitouri away, with Jones unable to sort his feet out to bury from close range.
Jon Nolan was then unable to find the back of the net after picking up the loose ball as his effort was blocked and then cleared.
Carke went off at the break after an incident in the tunnel in which he picked up a second yellow card.
The already hard to break down visitors kept even more men behind the ball after the restart.
Jones whipped a ball across the face of goal on the hour, but no Spireites stud could get on the end of it to tap it home.
The hosts were almost caught on the counter by Aaron Amadi Holloway, with Dan Gardner covering well to avert the danger.
However, right at the death a well-earned point was snatched away from Caldwell’s men.
Clarke rose highest to head home past Ryan Fulton, with Oldham fans running onto the pitch after the ball went in.
Chesterfield: Fulton, El-Fitouri, Evatt, Anderson, Jones (Martinez, 65), Gardner, Grimshaw (Simons, 81), Nolan (Brown, 65), Donohue, Evans, Faupala.
Unused subs: Allinson, Maguire, Dennis, Mitchell.
Oldham: Ripley, Wilson, Gerrard, Clarke, Hunt, Flynn (Green, 45), O’Neill, Fane, Taylor (Obadeyi, 78), Holloway (Osei, 90+3), McLaughlin.
Unused subs: Ngoo, Law, Reckford, Banks.
Attendance: 6375 (883)
Referee: Ross Joyce (Cleveland)