Around 25 supporters made their way onto the pitch after Solihull's third goal went in and although Allen said he understood their unhappiness, he couldn't condone their actions.
With Chesterfield sitting third from bottom in the National League, Allen accepted there would be questions asked of his future.
He was bitterly disappointed with the way his side started.
"The first 10 minutes our defending set-plays, long balls, direct balls was appalling," he said.
"It picked up a little bit after that, we had some chances, some good patterns of play.
"But every time they went forward I thought they were going to score.
"I thought it was a poor performance."
Allen made three changes for the game, two of them enforced by injuries to Tom Denton and Alex Kiwomya.
He picked Michael Nelson at centre-half, for the first time since September, but replaced him after less than 10 minutes, admitting the selection was a mistake for which he took full responsibility.
The manager said the standards set in the 8th December win over Salford were not replicated, by the players who came in or some of those who played in the only league victory since August.
Allen, the board and his players were all the subject of angry chants from supporters, some of whom entered the playing surface in protest.
"The crowd reaction, I fully understand that. I can't condone them coming on the pitch, but I can understand why they're so unhappy," he said.
"Everyone that pays their money is more than entitled to their opinion.
"The toxic attitude of the supporters, I don't blame them.
"They're fed up, they're upset, they've seen the demise of their fantastic football club.
"I think most of them realise we're in the middle of what you'd call a transitional period, they've seen an improvement over the last few weeks with the Salford game and the Orient game.
"The players have got to stand up and be counted, you've got to man up and deal with it, face it.
"If you don't play very well and don't head the ball out of your box, I think every person in this ground is entitled to shout whatever they like, be it against me or the players.
"When we played against Salford, every supporter to a person stood and applauded every tackle, every pass.
"The supporters want us to play well and if we do I know every supporter will be behind those players.
"No criticism of the supporters at all."
The man who did all the damage for Solihull in the first half, Nathan Blissett, is a striker Allen has twice tried to sign.
Hes been told there's no funding for new players until some of the transfer listed players depart.
"There's no money for any new faces.
"That's been made well clear to me.
"I tried to sign that centre forward today who battered us.
"I have to respect the owners' decisions, I have to respect what they want and what we can pay out.
"It's not my money, so who am I and who are any of us to criticise how much they pay.
"They've lost an awful lot of money on this football club.
"I can put the players forward and if they're there, they're there and if they're not there, I will always give it everything I've got with what I've got to get the best out of them.
"Today clearly we haven't got the best out of them because we've fallen well short.
"When you see the centre-forward down there, who I've tried to sign twice, I thought he was the best player on the pitch by a country mile.
"We could do with him in our team."
Allen, who hopes both Denton and Kiwomya will return for the clash with Hartlepool at the weekend, said today's performance was painful to watch.
And he knows his position will come under scrutiny.
"Of course. You lose at home like that there's bound to be questions asked. It's part and parcel of the job I do.
"You stand in the technical area watching that, you can't imagine how painful it is to see your team, some of those individuals I've brought here, play like that.
"It hurts me deeply, it does upset me.
"I want this place to be like it was against Salford and Basford, I love seeing the players happy.
"I was okay with them after the Orient game because it was a real proper good committed performance.
"We need some of those injured players back in, but in the meantime the others should be good enough to put in a good performance and a far, far better performance than that."