Jump to content

Pipeline Problem Potentially Problematic, production possibly paralyzed

Wally Fairway

By Wally Fairway
in Daily Texan

 Share

Recommended Posts

C-Man

C-Man 27230

Posted
Posted
this.  american senior management, especially long-tenured lifers, have almost zero understanding of how the IT industry works today.  they think software is magic, you buy it, it runs itself, and the people that run it are an expense for the CEO to shoot when the board is up his ass about $.

Colonial CEO after the company apparently paid the $5M ransom the day after the attack?

giphy.gif

Should’ve paid more attention to these guys

giphy.gif
  • Haha 1
Link to comment
Share on other sites
EE2B

EE2B 1565

Posted
Posted
It's incredibly common for big/old IT shops to have a backup solution in place, but to have never actually tested it or restored a backup to exercise their ops runbook. Turns out, just having copies of the data doesn't do you any good if nobody knows how to restore it from archive

Our emissions data is calculated partially in a VM with a SQL database. The ESXi host that it was running on crashed causing the VM to shutdown, migrate, and restart. When it did this the SQL database was corrupted. I was on the phone with the 3rd party support folks for the software when they asked if we could restore from a snapshot prior to the crash. Sure, should not a problem. Let me walk across the hall to the IT folks and ask them to take care of it. The answer I got was - ugh yea it can be done but we’ve never done it before. The 3rd party folks got the database repaired and started rebuilding before it could be figured out. Some days I wonder....
Link to comment
Share on other sites
mycox

mycox 582

Posted
Posted (edited)

Backups may have or may not have helped in this case. The hackers had been snooping around the network for a year before they took the system down. I'm sure they could restore, but it would take weeks to get everything back to where it was before the network was infiltrated.

It's not just a simple restore.

Edited by mycox
  • Hook 'Em 1
  • Like 1
Link to comment
Share on other sites
NeverMarryAStripper

NeverMarryAStripper 21907

Posted
Posted

We got hit with ransom attack a while back  We're E&P so it didn't have a huge effect on operations.  We had just acquired another company and hadn't yet integrated systems.  The acquired company's network got hit and our IT department managed to sever the connection before our network got infected, but we shut it down too.  They asked for $2 million. Our IT guys said they thought they could get everything restored.  We hired a specialist to negotiate with them.  After a week they were down to about $200K and our IT guys said we were clean and fully restored so we told them to fuck off.  It ended up costing us about $80K and a week of downtime on our network, but production never stopped.

  • Hook 'Em 2
Link to comment
Share on other sites
Hagbard Celine

Hagbard Celine 10871

Posted
Posted

msm embargo continues on what actually happened.   msm does not understand the event at all.  or is deliberately (by instruction?) not talking about what really happened.

the executives shut down the pipeline.  the hack did not touch the pipeline itself.  the legacy oracle erp as not just hacked, it was encrypted, locked, frozen.  prem iron can't be "restored" if it's encrypted.    the hardware was bricked.   execs didn't call the white house or the nsa.   they took down national security infrastructure because they were afraid they would not get paid.

  • Hook 'Em 2
Link to comment
Share on other sites
High Plains Drifter

High Plains Drifter 11050

Posted
Posted
1 hour ago, NeverMarryAStripper said:

We got hit with ransom attack a while back  We're E&P so it didn't have a huge effect on operations.  We had just acquired another company and hadn't yet integrated systems.  The acquired company's network got hit and our IT department managed to sever the connection before our network got infected, but we shut it down too.  They asked for $2 million. Our IT guys said they thought they could get everything restored.  We hired a specialist to negotiate with them.  After a week they were down to about $200K and our IT guys said we were clean and fully restored so we told them to fuck off.  It ended up costing us about $80K and a week of downtime on our network, but production never stopped.

 

just curious, how much did you pay the specialist negotiator?

 

 

Link to comment
Share on other sites
atomheartbevo

atomheartbevo 89727

Posted
Posted
54 minutes ago, Hagbard Celine said:

msm embargo continues on what actually happened.   msm does not understand the event at all.  or is deliberately (by instruction?) not talking about what really happened.

the executives shut down the pipeline.  the hack did not touch the pipeline itself.  the legacy oracle erp as not just hacked, it was encrypted, locked, frozen.  prem iron can't be "restored" if it's encrypted.    the hardware was bricked.   execs didn't call the white house or the nsa.   they took down national security infrastructure because they were afraid they would not get paid.

MSM didn't understand it from what I saw.

Plus, it was a lot sexier/click-baitier to show videos/photos of people filling up all kinds of different containers with gasoline than talk about networks being hacked.

There better be a lot more Darwin awards out of this, based on some of the stupid shit that's been posted.

Link to comment
Share on other sites
Wally Fairway

Wally Fairway 5559

Posted
  • Author
Posted

WSJ article about paying the ransom

https://www.wsj.com/articles/colonial-pipeline-ceo-tells-why-he-paid-hackers-a-4-4-million-ransom-11621435636?mod=hp_lead_pos4

Spoiler
Quote

Because paywalls suck

 

The operator of the Colonial Pipeline learned it was in trouble at daybreak on May 7, when an employee found a ransom note from hackers on a control-room computer. By that night, the company’s chief executive came to a difficult conclusion: He had to pay.

Joseph Blount, CEO of Colonial Pipeline Co., told The Wall Street Journal that he authorized the ransom payment of $4.4 million because executives were unsure how badly the cyberattack had breached its systems or how long it would take to bring the pipeline back.

Mr. Blount acknowledged publicly for the first time that the company had paid the ransom, saying it was an option he felt he had to exercise, given the stakes involved in a shutdown of such critical energy infrastructure. The Colonial Pipeline provides roughly 45% of the fuel for the East Coast, according to the company.

“I know that’s a highly controversial decision,” Mr. Blount said in his first public remarks since the crippling hack. “I didn’t make it lightly. I will admit that I wasn’t comfortable seeing money go out the door to people like this.”

 

“But it was the right thing to do for the country,” he added.

For years, the Federal Bureau of Investigation has advised companies not to pay when hit with ransomware, a type of code that takes computer systems hostage and demands payment to have files unlocked. Doing so, officials have said, would support a booming criminal marketplace.

But many companies, municipalities and others debilitated by attacks do pay, concluding it is the only way to avoid costly disruptions to their operations.

U.S. officials have linked the ransomware attack on Colonial to a criminal gang known as DarkSide, believed to be based in Eastern Europe, which specializes in crafting the malware used to breach systems, and shares it with affiliates—for a cut of the ransoms they obtain.

Mr. Blount said Colonial paid the ransom in consultation with experts who had previously dealt with the criminal organization behind the attacks. He and others involved declined to detail who assisted in those negotiations. Colonial said it has cyber insurance, but declined to provide details on ransomware-related coverage.

In return for the payment, made in the form of bitcoin, about 75 in all, according to a person familiar with the matter, the company received a decryption tool to unlock the systems hackers penetrated. While it proved to be of some use, it was ultimately not enough to immediately restore the pipeline’s systems, the person said. 

The pipeline, which transports gasoline, diesel, jet fuel and other refined products from the Gulf Coast to Linden, N.J., wound up being shut down for six days. The stoppage spurred a run on gasoline along parts of the East Coast that pushed prices to the highest levels in more than 6 ½ years and left thousands of gas stations without fuel.

East Coast stockpiles of gasoline dropped by about 4.6 million barrels last week, the steepest weekly drop since late February, Energy Department data showed.

The pipeline company, which is based in Alpharetta, Ga. and owned by units of IFM Investors, Koch Industries Inc., KKR & Co. and Royal Dutch Shell PLC, restored service on the pipeline last week. It said Monday that it was transporting fuel at normal levels, though it warned that it would take time for the supply chain to recover.

The crisis was a test of leadership for Mr. Blount, 60 years old, who has led the company since 2017. He had co-founded private equity-backed pipeline company Century Midstream LLC in 2013, after working as an executive and in other roles at energy companies over an almost 40-year career.

Over the past five years, Mr. Blount said, Colonial has invested about $1.5 billion in maintaining the integrity of its 5,500-mile pipeline system, and has spent $200 million on IT.

For Mr. Blount, the cyberattack was akin to the Gulf Coast hurricanes that often force segments of pipelines and refineries to shut down for days or weeks. However, it was in some ways more devastating. The Colonial Pipeline had never before been shut down all at once, he said.

The attack was discovered around 5:30 a.m. on May 7 and quickly set off alarms through the company’s chain of command, reaching Mr. Blount less than a half-hour later as he was getting ready for the workday. The company has stressed that operational systems weren’t directly impacted, and that it shut down pipeline flows while it investigated how deeply the hackers had gotten inside.

It took Colonial about an hour to shut the conduit, which has about 260 delivery points across 13 states and Washington, D.C. The move was also meant to prevent the infection from potentially migrating to the pipeline’s operational controls.

As Colonial shut the pipeline, employees were instructed not to log in to its corporate network, and executives made a volley of phone calls to federal authorities, starting with the FBI’s offices in Atlanta and San Francisco, as well as a representative from the Cybersecurity and Infrastructure Security Agency, or CISA, Mr. Blount said.

CISA officials confirmed Colonial representatives informed them of the hack shortly after the incident occurred. FBI representatives didn’t respond to requests for comment.

Over the next several days, the Energy Department acted as a conduit through which Colonial could provide updates to multiple federal agencies involved in the response, Mr. Blount said. Energy Secretary Jennifer Granholm and Deputy Secretary David Turk stayed in regular contact with the company, in part to “gain information to guide the federal response,” Energy Department spokesman Kevin Liao said.

As Colonial prepared to restore service, its personnel patrolled the pipeline searching for any signs of physical damage, driving some 29,000 miles. The company dispatched nearly 300 workers to keep their eyes on the pipeline, supplementing its usual electronic monitoring, Mr. Blount said.

Though the pipeline’s flow of fuel has returned to normal, the impact of the hack hardly ended with the ransom payment. It will take months of restoration work to recover some business systems, and will ultimately cost Colonial tens of millions of dollars, Mr. Blount said, noting that it is still unable to bill customers following an outage of that system.

Another costly loss, Mr. Blount noted, was the company’s preferred level of anonymity.

“We were perfectly happy having no one know who Colonial Pipeline was, and unfortunately that’s not the case anymore,” he said. “Everybody in the world knows.”

 

 

Link to comment
Share on other sites
pantone159

pantone159 719

Posted
Posted
On 5/16/2021 at 7:35 AM, Armybrat said:

Americans have traditionally been lax with security.... 12/7, 9/11, 1/6, 2/10.

What happened on Feb 10? I was not sure what that one was.

I suppose you might be referring to when the Wampanoag Indians under King Philip killed all the men in Lancaster Mass in 1676, maybe you barely escaped from that in time or something.

Link to comment
Share on other sites
Armybrat

Armybrat 40850

Posted
Posted
42 minutes ago, pantone159 said:

What happened on Feb 10? I was not sure what that one was.

I suppose you might be referring to when the Wampanoag Indians under King Philip killed all the men in Lancaster Mass in 1676, maybe you barely escaped from that in time or something.

Texas power grid failure in the Big Freeze this year.

  • Hook 'Em 1
Link to comment
Share on other sites
Buzzrock

Buzzrock 14922

Posted
Posted
On 5/15/2021 at 11:23 PM, Hagbard Celine said:

msm embargo continues on what actually happened.   msm does not understand the event at all.  or is deliberately (by instruction?) not talking about what really happened.

the executives shut down the pipeline.  the hack did not touch the pipeline itself.  the legacy oracle erp as not just hacked, it was encrypted, locked, frozen.  prem iron can't be "restored" if it's encrypted.    the hardware was bricked.   execs didn't call the white house or the nsa.   they took down national security infrastructure because they were afraid they would not get paid.

How do you know the hardware was bricked? Encrypted data on drives can be deleted/formatted and restored from backup. Bricking would require something to be wrong with the firmware. Even then you could restore to other hardware, which most suppliers can provide in hours. 
 

More likely they had no or poor backups. It’s amazing how few companies have a real BCDR plan. 

Link to comment
Share on other sites
  • 2 weeks later...
Brisketexan

Brisketexan 105154

Posted
Posted

Russia is happy to harbor cyber-terrorists and thieves who fuck with our infrastructure and economy.

We need to fund/support a similar effort to hit them where it hurts: the pocketbooks of the oligarchs who keep Putin in power.  Empty a few of their bank accounts.  Hack into the systems of their villas, and cause a fire to burn them to the ground.

They want to fight this low-intensity war?  Then let's fucking fight.

  • Hook 'Em 1
  • Like 1
Link to comment
Share on other sites
Nice Guy Eddie

Nice Guy Eddie 13515

Posted
Posted
On 5/15/2021 at 10:54 PM, High Plains Drifter said:

 

just curious, how much did you pay the specialist negotiator?

 

 

 

On 5/15/2021 at 11:02 PM, NeverMarryAStripper said:

That was most of the $80k.  There was some contract IT help for our guys.

What if I told you that the specialist negotiator was part of the hack, and he legitimately earned high 5 figures for less than a week's effort.

Matrix-Oracle-Quotes-Vase.jpg?q=50&fit=c

  • Hook 'Em 1
Link to comment
Share on other sites
Wally Fairway

Wally Fairway 5559

Posted
  • Author
Posted
23 minutes ago, Brisketexan said:

Russia is happy to harbor cyber-terrorists and thieves who fuck with our infrastructure and economy.

We need to fund/support a similar effort to hit them where it hurts: the pocketbooks of the oligarchs who keep Putin in power.  Empty a few of their bank accounts.  Hack into the systems of their villas, and cause a fire to burn them to the ground.

They want to fight this low-intensity war?  Then let's fucking fight.

You want to get the Russians full attention, cyber-attack their vodka production

  • Haha 1
Link to comment
Share on other sites
GRHorn

GRHorn 105

Posted
Posted
On 6/2/2021 at 12:27 PM, Brisketexan said:

Russia is happy to harbor cyber-terrorists and thieves who fuck with our infrastructure and economy.

We need to fund/support a similar effort to hit them where it hurts: the pocketbooks of the oligarchs who keep Putin in power.  Empty a few of their bank accounts.  Hack into the systems of their villas, and cause a fire to burn them to the ground.

They want to fight this low-intensity war?  Then let's fucking fight.

I’m extremely skeptical on the Russian angle here. 

You mean to tell me that a sophisticated group of Russian hackers shut down a pipeline and collected Bitcoin for it, then left it in an unsecured or third party hosted wallet? In the US? The Surly crypto thread could tell you that’s regard level personal security.
 

Something is extremely fishy here. I’m close to going full Alex Jones false flag here. 

  • Fuck You 1
Link to comment
Share on other sites
Captainant

Captainant 27680

Posted
Posted (edited)
15 hours ago, GRHorn said:

You mean to tell me that a sophisticated group of Russian hackers shut down a pipeline and collected Bitcoin for it, then left it in an unsecured or third party hosted wallet? In the US? The Surly crypto thread could tell you that’s regard level personal security.
 

Something is extremely fishy here. I’m close to going full Alex Jones false flag here. 

I disagree, from the reporting there was the most evidence that this was the actions of a group who bought the exploit (written by the russians) and started using it.

A group of gangsters "buying a hack" off the shelf (trying to frame it as they'd see it) and cutting some corners on the tech-side of things? I totally see that happening - it happens every day in enterprise space and off the shelf consumer software.

This wasn't some earth-shatteringly sophisticated hack, it was just a digital smash and grab on a big target.

EDIT: just to clarify - the pipeline infrastructure wasn't what was impacted in this attack. It was the billing system for the gas pipeline. The operators shut the flow down because their information technology was impacted and they wouldn't be able to bill accurately for it - not because they lost control of their operational technology.

Edited by Captainant
Link to comment
Share on other sites
NeverMarryAStripper

NeverMarryAStripper 21907

Posted
Posted
On 6/2/2021 at 12:27 PM, Brisketexan said:

Russia is happy to harbor cyber-terrorists and thieves who fuck with our infrastructure and economy.

We need to fund/support a similar effort to hit them where it hurts: the pocketbooks of the oligarchs who keep Putin in power.  Empty a few of their bank accounts.  Hack into the systems of their villas, and cause a fire to burn them to the ground.

They want to fight this low-intensity war?  Then let's fucking fight.

I say we do it the old fashioned way

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing


Football ... Basketball ... Baseball ... Other Sports ... Futbol ... 🤫995🤫 ... Gambling ... Movies & TV ... Music ... Hobbies ... Lulz ... Food & Travel ... Daily Texan ... Business and Markets ... Cloak Room ... Help ... For Sale ... Board Discussion ... Subscribe!... Donate!... Advertise... COOKIE MONSTER!

×
×
  • Create New...