Millions of websites offline after fire at French cloud services firm

[clapclapclap 9000]

 

https://www.reuters.com/article/us-france-ovh-fire-maginot-firewall-fail

Spoiler

PARIS (Reuters) - A fire at a French cloud services firm has disrupted millions of websites, knocking out government agencies’ portals, banks, shops, news websites and taking out a chunk of the .FR web space, according to internet monitors.

Slideshow ( 4 images )

The fire, which broke out on Wednesday shortly after midnight at OVHcloud, destroyed one of four data centres in Strasbourg, in eastern France, and damaged another, the company said.

There was no immediate explanation provided for the blaze, which erupted just two days after the French cloud computing firm kicked off plans for an initial public offering.

Europe’s largest cloud services provider told clients including the French government, the Centre Pompidou and cryptocurrency exchange Deribit to activate their disaster recovery plans following the blaze.

“Firefighters were immediately on the scene but could not control the fire” in the affected data centre, founder and chairman Octave Klaba said on Twitter. He said the plan for the next couple of weeks would include rebuilding the centres’ equipment and checking their fiber optic connections.

Some 100 firemen fought the blaze which sent a thick plume of black smoke into the night sky. Video images showed firefighters dousing one smouldering, multi-storey building in the early morning as they cooled down the site.

Founded by Klaba in 1999, OVHcloud competes against U.S. giants Amazon Web Services, Microsoft Corp’s Azure and Alphabet Inc’s Google Cloud, which dominate the market.

“OVH is a pretty important hosting company on the internet,” said Mike Prettejohn, who directs UK-based network security company Netcraft. He said the affected servers hosted 3.6 million websites, including niche government platforms in France, Britain, Poland and the Ivory Coast.

Prettejohn estimated that just under 2% of the sites with the French .FR domain extension had been affected.

French politicians have championed OVHcloud as a possible alternative to U.S. cloud services providers, but it has so far lacked the scale and financial clout to dent their market share.

The company said on Monday it had started the process for a potential IPO, without giving details.

The Strasbourg centres were among OVHcloud’s 17 data centres in France, and 32 globally.

Several clients said their websites had gone offline or emails could not be accessed. There was no immediate news of any major data losses.

The Centre Pompidou, one of France’s best known art complexes, said its website was down. Cryptocurrency exchange Deribit said its blog was down but that trading had not been impacted, free chess server Lichess.org said it had lost 24 hours’ worth of games history and news outlet eeNews Europe said its websites were offline.

PR agency Caroline Charles Communication said its data was held at another site but that it had lost access to emails in the middle of Paris Fashion Week.

OVHcloud declined to comment on its fire safety protocols. The site did not pose a toxic risk, it added.

Reporting by Matthieu Protard and Mathieu Rosemain in Paris and Raphael Satter in Washington; Writing by Richard Lough; Editing by Mark Potter and Matthew Lewis

Update:  Interpol has named Frank Gehry as an Architect of Interest.

[Hate 20157]

There is no way that wasn't arson.  With all of the protections modern data centers have, that was done purposefully. 

[AUS-97HORN 11874]

heck of a disaster backup plan they have there.

they should have been able to fail over somewhere within 10 mins. 

[crash_davis 23154]

3 minutes ago, AUS-97HORN said:

heck of a disaster backup plan they have there.

they should have been able to fail over somewhere within 10 mins. 

were the Austin water treatment plant execs also in charge of the data center back up plans and implementation?

[Hate 20157]

5 minutes ago, AUS-97HORN said:

heck of a disaster backup plan they have there.

they should have been able to fail over somewhere within 10 mins. 

The dirty little secret in IT is that backup plans suck for the most part.  Most companies put the disaster in disaster planning. 

[Deej 77160]

So, no firewall?

[Blotto 23678]

14 minutes ago, Hate said:

There is no way that wasn't arson.  With all of the protections modern data centers have, that was done purposefully. 

 

Quote

The company said on Monday it had started the process for a potential IPO, without giving details.

[Updawg 2301]

Yeah, seems like an inside job

Is that place made out of containers? Pretty big though, they are pretty big.

Should be Less scanning from them against my place hopefully.

[Guest Lobo]

Whose idea was it to name this firewall "Maginot"?  

Has that thing ever held?  

[Hate 20157]

The Maginot Line did it's job, they just went around it.

[Guest Lobo]

So basically French Security, then?  

[washparkhorn 9939]

 

[washparkhorn 9939]

11 minutes ago, Hate said:

The Maginot Line did it's job, they just went around it.

 

[Surly Bevo 25501]

[Wiler77 1584]

OVH is huge. They are the largest EMEA-based cloud hoster. They are in IDC's top 20 list of server consumers.  In the grand scale of their business, though, I think this is fairly small. I think they have like 30+ data centers. But I assume some big French customers were serviced from that data center, being as it's in France. Likely government. Not good.

To someone's point above, doesn't sound like disaster recovery went all that swimmingly.

Here's a look into the board room as they learned about the fire.

[thunderlounge 8029]

Has anyone came across video of the scene with yakkity sax as the soundtrack?

[bernorange 8373]

I have banned a lot of ip blocks from ovh over the years because of bots/spam traffic coming from their servers.  Fuck 'em.

[Enchubben 5555]

I saw reports elsewhere that the fire department showed up to go to war with the fire, but saw how dangerous it was and just surrendered the facility. 

[Onboard 2.0 9083]

2 minutes ago, Enchubben said:

I saw reports elsewhere that the fire department showed up to go to war with the fire, but saw how dangerous it was and just surrendered the facility. 

Fucking cheap, lame, cracks about the French, and their cowardly, surrendering ways.  I like it.

[Shoxthemonkey 1983]

As long as the French Girls at Work channel is still up at Pornhub, I will be too.

[clapclapclap 9000]

9 hours ago, Lobo said:

Whose idea was it to name this firewall "Maginot"?  

Has that thing ever held?  

John Lennon + Apple (aka immamac)

 

iMaginot Firewall.  It's easy if you try.

[Pescado_Rojo 19453]

11 hours ago, Hate said:

The dirty little secret in IT is that backup plans suck for the most part.  Most companies put the disaster in disaster planning. 

Several years ago, one of my customers did a complete remodel of the cool old building that hosted one of their DCs. We sold them a big server refresh to go into the newly expanded DC, and were on site the day they were going to light everything back up. We had a professional photographer there and everything because my company was going to do a big press release on how we helped company XYZ modernize their DC. Had a big glass wall, cool lights, all that shit. Anyway, they throw the switch, and the entire building goes dark. CIO is like "huh...well, the battery backups should come on any second now........huh....well, the generator should come on any second now........" The batteries in the battery backup were dead as doornails, and someone had stolen all the fuel (like, 1,000 gallons of diesel) from the generator tank. It was like a scene from a comedy watching these poor fuckers run around trying to figure out why the whole building was pitch black. 

[Orange&White 5554]

My organization was building a new facility for a different business unit and they incorporated a bunch of money into the build to house our backup/DR datacenter. We were included in all the design and drawings discussions to make sure we had everything we needed included on the design.

Construction took about two years, and about a month prior to opening the facility for use, the contractors allowed us in to start putting in switching and networking equipment.

We walked into the proposed datacenter and someone had done a change order mid stream that no one bothered to tell IT about. The change order ran about six exposed 2" water lines directly over the top of, and down the full length of, the eight server racks.

Management couldn't understand why we refused to put several million dollars worth of equipment into the room.

 

 

Edited March 11, 2021 by Orange&White

[AUS-97HORN 11874]

15 minutes ago, Orange&White said:

We walked into the proposed datacenter and someone had done a change order mid stream that no one bothered to tell IT about. The change order ran about six exposed 2" water lines directly over the top of, and down the full length of, the eight server racks.

Management couldn't understand why we refused to put several million dollars worth of equipment into the room.

sounds to me like you wasted a perfectly good opportunity to turn those racks into liquid cooled servers.

[Enchubben 5555]

19 minutes ago, Orange&White said:

My organization was building a new facility for a different business unit and they incorporated a bunch of money into the build to house our backup/DR datacenter. We were included in all the design and drawings discussions to make sure we had everything we needed included on the design.

Construction took about two years, and about a month prior to opening the facility for use, the contractors allowed us in to start putting in switching and networking equipment.

We walked into the proposed datacenter and someone had done a change order mid stream that no one bothered to tell IT about. The change order ran about six exposed 2" water lines directly over the top of, and down the full length of, the eight server racks.

Management couldn't understand why we refused to put several million dollars worth of equipment into the room.

 

 

no problem. you just gotta drip the faucets.

[ABSR 5327]

And just like that they took the top position in the Black Cloud market.