Fastly says just one customer caused huge internet outage by accident

‘This outage was broad and severe, and we’re truly sorry for the impact to our customers’

Andrew Griffin
Thursday 10 June 2021 05:58
Comments
<p>Fastly said the bug was in a software update shipped to customers on 12 May but was not triggered until one customer carried out settings changes</p>

Fastly said the bug was in a software update shipped to customers on 12 May but was not triggered until one customer carried out settings changes

The global internet outage that took down much of the internet for more than an hour was caused by a software bug, the company behind it said.

Fastly said that an issue in its software had been triggered when one of its customers changed their settings.

When that happened, it began a series of issues that knocked many of the world’s biggest websites offline. Reddit, Amazon, the UK government and many of the world’s biggest news organisations were unavailable, with users instead seeing an array of error messages.

“This outage was broad and severe, and we’re truly sorry for the impact to our customers and everyone who relies on them,” the company said in a blog post authored by Nick Rockwell, its senior engineering and infrastructure executive.

He said the problem should have been anticipated.

Fastly operates a group of servers strategically placed around the world to help customers move and store content close to their end users quickly and safely.

The company post gave a timeline of events and promised to examine and explain why Fastly had failed to detect the software bug during its own testing process.

Fastly said the bug was in a software update shipped to customers on 12 May but was not triggered until one unidentified customer carried out settings changes that triggered the problem “which caused 85 per cent of our network to return errors.”

Fastly noticed the outage within a minute it occurring at 9.47am, and engineers worked out the cause at 10.27am. Once they disabled the settings that triggered the problem, most of the company’s network quickly recovered.

“Within 49 minutes, 95 per cent of our network was operating as normal,” the company said.

Its networks were fully recovered at 12.35pm and it began rolling out a permanent software fix at 5.25pm, Fastly said.

Additional reporting by Reuters

Register for free to continue reading

Registration is a free and easy way to support our truly independent journalism

By registering, you will also enjoy limited access to Premium articles, exclusive newsletters, commenting, and virtual events with our leading journalists

Please enter a valid email
Please enter a valid email
Must be at least 6 characters, include an upper and lower case character and a number
Must be at least 6 characters, include an upper and lower case character and a number
Must be at least 6 characters, include an upper and lower case character and a number
Please enter your first name
Special characters aren’t allowed
Please enter a name between 1 and 40 characters
Please enter your last name
Special characters aren’t allowed
Please enter a name between 1 and 40 characters
You must be over 18 years old to register
You must be over 18 years old to register
Opt-out-policy
You can opt-out at any time by signing in to your account to manage your preferences. Each email has a link to unsubscribe.

By clicking ‘Create my account’ you confirm that your data has been entered correctly and you have read and agree to our Terms of use, Cookie policy and Privacy notice.

This site is protected by reCAPTCHA and the Google Privacy policy and Terms of service apply.

Already have an account? sign in

By clicking ‘Register’ you confirm that your data has been entered correctly and you have read and agree to our Terms of use, Cookie policy and Privacy notice.

This site is protected by reCAPTCHA and the Google Privacy policy and Terms of service apply.

Register for free to continue reading

Registration is a free and easy way to support our truly independent journalism

By registering, you will also enjoy limited access to Premium articles, exclusive newsletters, commenting, and virtual events with our leading journalists

Already have an account? sign in

By clicking ‘Register’ you confirm that your data has been entered correctly and you have read and agree to our Terms of use, Cookie policy and Privacy notice.

This site is protected by reCAPTCHA and the Google Privacy policy and Terms of service apply.

Join our new commenting forum

Join thought-provoking conversations, follow other Independent readers and see their replies

Comments

Thank you for registering

Please refresh the page or navigate to another page on the site to be automatically logged inPlease refresh your browser to be logged in