The Blues are seventh in the Premier League table with 13 points from seven matches and Cahill has been targeted by supporters on social media as one of the main culprits for their struggles following the defeats by Liverpool and Arsenal.
The 30-year-old England international made a crucial error as Chelsea succumbed to a dispiriting 3-0 reverse at the Emirates as Conte’s men fell five points behind early pacesetters Manchester City with one point from three matches.
Speaking to BBC 5 Live, he said: "I don't care what anybody says, I don't listen to anybody. The only people I have to prove myself to is the manager of the football club that I'm playing at and my team-mates that I'm playing with.
"For me, my form up to Swansea [2-2 draw with Chelsea on September 11] was fine. I was playing well, I was happy.
"But then I made a few big errors and we all know that when you make a few errors in football, basic ones as well, it gets noticed. I'm not stupid.
"But if you make a couple of mistakes in a couple of big games, then all of a sudden the whole season is tarnished like you're having a bit of a nightmare.
Chelsea vs Arsenal player ratings
Chelsea vs Arsenal player ratings
1/22 Petr Cech - 6 out of 10
Enjoyed a surprisingly quiet evening against his former club, but stayed alert to save well late on from Batshuayi.
2/22 Hector Bellerin - 7 out of 10
Won the battle on the wing against his compatriot Azpilicueta, and added a fine assist to his clean sheet.
3/22 Shkodran Mustafi - 8 out of 10
Impressed as a physical force in the Arsenal defence, giving as good as he got from Costa - and better.
4/22 Laurent Koscielny - 7 out of 10
His partnership with Mustafi continued to blossom as he led the backline with authority.
5/22 Nacho Monreal - 7 out of 10
Quietly efficient as he kept Willian under wraps for long periods.
6/22 Francis Coquelin - 5 out of 10
Battled well early in the game, but had to be replaced after coming off worst in a fierce 50-50 challenge with Kante.
7/22 Santi Cazorla - 7 out of 10
Although he tired towards the end, the Spaniard dictated the midfield with his creative play.
8/22 Theo Walcott - 7 out of 10
Once again worked tirelessly for the team. The Walcott of this season is like a new signing for Arsenal.
9/22 Mesut Ozil - 8 out of 10
Given time and space to play by Chelsea, the German bossed events at the Emirates. His turn and run in the build-up to his goal was sublime.
10/22 Alex Iwobi - 8 out of 10
An excellent display that showcased the youngster as a real attacking threat. Demonstrated awareness of his teammates around him.
11/22 Alexis Sanchez - 8 out of 10
Scorer of a well-taken first, his assist for Ozil was just as good. Leading the line much better than last month.
12/22 Thibaut Courtois - 5 out of 10
Offered little protection by his defence, he still suffered a number of breakdowns in communication with the outfield players.
13/22 Branislav Ivanovic - 5 out of 10
Constantly troubled by Iwobi's energetics, time continues to catch up with Chelsea's experienced defender.
14/22 Gary Cahill - 4 out of 10
Directly responsible for some awful defending, most notably when he lost possession in the lead up to Arsenal's first.
15/22 David Luiz - 5 out of 10
His partnership with Cahill needs work - and a lot of it. However, he did show an ability to pick a pass when going forward.
16/22 Cesar Azpilicueta - 6 out of 10
Often outnumbered as Walcott and Bellerin doubled up, he had more joy when moved to right wing-back in the second half.
17/22 N'Golo Kante - 4 out of 10
An unrecognisable performance given last season's heroics. Second to many loose balls and showed an unwillingness to track back when out of possession.
18/22 Willian - 5 out of 10
Went close to netting with a shot wide in the first half, he was otherwise kept quiet by Monreal and was eventually subbed.
19/22 Cesc Fabregas - 6 out of 10
Booed constantly on his return to the Emirates, he remained calm and played some fine passes. Surprisingly subbed when clearly performing better than others.
20/22 Nemanja Matic - 5 out of 10
Struggled to deal with the attacking runs from deep of Ozil, Iwobi and Walcott.
21/22 Eden Hazard - 5 out of 10
Showed flashes of skill and creatively, but more often it was his inability to help out at the back that was highlighted.
22/22 Diego Costa - 6 out of 10
Offered no service by his teammates who simply didn't play to his strengths. Never stop battling for possession.
"Yet it's not like you can turn around and say 'he's lost his legs or is being beaten all ends up'.
"I basically made my own mistakes out of nothing, so as ridiculously as bad as they were, they are also easy to correct."Reuse content