Solarwinds Sunburst attack discussion thread

[Guest Lobo]

Here's a fun fact:  Mike Widmann from Silver Lake was apparently the ring-leader on this stock dump (he's also a board member with Solar Winds).  He was also on the desk that worked with the Trump family while at Credit Suisse and handles some of the institutional capital onboarding from Eastern Europe for Silver Lake.  

Better not sit for a deposition Mr. Widmann.  And if you do, don't start your car for a couple years.  

[TexasGrunt 72]

I worked at Equifax when they had their breach, and I was also there for the clean up. (Stayed away from the threads on that one)  I remember a C level type got in trouble for doing same and he is now in jail.  

[drt 639]

His lawyers must love how dumb he is. 

[Dnaguy 12001]

7 hours ago, Buzzrock said:

Some people are going to jail. Two big PE firms that own 70% of the company and control the board dumped $280M in stock days before the breach was made public. 
 

https://www.washingtonpost.com/technology/2020/12/15/solarwinds-russia-breach-stock-trades/

 

How quaint.

Go to jail?

These fuckers will probably get elected to the Senate.

 

[Blotto 23559]

10 hours ago, Buzzrock said:

Some people are going to jail. Two big PE firms that own 70% of the company and control the board dumped $280M in stock days before the breach was made public. 
 

https://www.washingtonpost.com/technology/2020/12/15/solarwinds-russia-breach-stock-trades/

 

2 hours ago, Dnaguy said:

 

How quaint.

Go to jail?

These fuckers will probably get elected to the Senate.

 

No, you see the stock sale was a “private placement” with a single institutional investor, and added that the investment firms “were not aware of this potential cyberattack at SolarWinds prior to entering into” the deal." How convenient. That single institutional investor will probably be very motivated to learn exactly what Silver Lake and Thoma Bravo did know. 

[pacman 7359]

On 12/14/2020 at 3:47 PM, burntorangebongos said:

As a precaution, Financial institutions are being urged to review their vendors using SolarWinds Orion platform and disconnect if so to keep in line with their security compliance requirements. That is my understanding anyway. Sucks for SolarWinds big time.

Solarwinds got hacked last Spring. The hackers unserted a backdoor dll into the next Orion update that a ton of people installed around March 2020.

Ince Solarwinds pushed that update, and the companies installed it, Russian hackers had undetectable access to everything. Solarwind installs operate with the highest privileges and the hackers made all of the malicious activity appear identical to the products activity.

 

We still wouldn't know of this if the hackers hadn't gotten greedy and went after a somewhat competent company in FireEye. FireEye's great IR is what led to this discovery.

 

The current claim by Solarwinds is only 18,000 of their 330,000 customers downloaded the malicious update.

Of course, that damn update was still up, and accessible for download from Solarwinds, as late as yesterday. So, their competence to provide accurate info is very much in question.

 

2020 will 2020

 

Edited December 17, 2020 by pacman

[burntorangebongos 10747]

Wow, thanks for the backstory, Pacman. This is likely very massive and disruptive. They are so emboldened right now. They really need a slap down. 

[SubliminalHorn 6806]

48 minutes ago, burntorangebongos said:

Wow, thanks for the backstory, Pacman. This is likely very massive and disruptive. They are so emboldened right now. They really need a slap down. 

lol that won’t come until 1/20 at the earliest 

[Guest Lobo]

Imagine if the Executive Branch focused 1/1000th of the energy into Solarwinds software vulnerability as they did into the software vulnerability of Dominion.  Both are important.  But good news outta D.C., there is substantial interest and tweeting about a California High School changing its name from Lincoln High.  Which is so critical to cover from D.C. right now given our Treasury Dept. having been compromised.  I mean, no offense to folks at HUD or Education, but the U.S. Department of the Treasury is kind of a big deal.  It's kind of an important federal Department/Cabinet Post.  But I'm comforted by the fact that every single message coming out of D.C. right now is about Rand Paul and Michigan.  Really good for the markets.  

[Buzzrock 14922]

6 hours ago, pacman said:

Solarwinds got hacked last Spring. The hackers unserted a backdoor dll into the next Orion update that a ton of people installed around March 2020.

Ince Solarwinds pushed that update, and the companies installed it, Russian hackers had undetectable access to everything. Solarwind installs operate with the highest privileges and the hackers made all of the malicious activity appear identical to the products activity.

 

We still wouldn't know of this if the hackers hadn't gotten greedy and went after a somewhat competent company in FireEye. FireEye's great IR is what led to this discovery.

 

The current claim by Solarwinds is only 18,000 of their 330,000 customers downloaded the malicious update.

Of course, that damn update was still up, and accessible for download from Solarwinds, as late as yesterday. So, their competence to provide accurate info is very much in question.

 

2020 will 2020

 

They also figured out how to compromise multi-factor authentication by hacking an Outlook Web server and then stealing the akey via its integration with the MFA system. So when it came time to enter the one time key the system thought it had already been satisfied and just a username and password was all that was needed. Clever. 

[Shaggy3.0 1695]

On 12/14/2020 at 10:31 AM, TXSG8R said:

 

Solarwinds will be fine long term.  They are a behemoth in the network admin world, but they could have fucked themselves out of gov't contracts for the foreseeable future.  

This will not age well.

PW: "solarwinds123"  // <= LOLOL.

https://www.newsweek.com/solarwinds-update-server-could-accessed-2019-using-password-solarwinds123-report-1554986

Solar Winds is TOAST.

Edited December 18, 2020 by Shaggy3.0

[Guest Lobo]

I, for one, am glad to see the Federal Government not rushing to judgement and not condemning the foreign actors behind this.  Sure hundreds, maybe thousands, of innocent hard working Texans at Solarwinds will lose their jobs over this...but the important thing is that our national leadership not take this seriously and publicly remark on this disaster.  

[SubliminalHorn 6806]

14 minutes ago, Lobo said:

I, for one, am glad to see the Federal Government not rushing to judgement and not condemning the foreign actors behind this.  Sure hundreds, maybe thousands, of innocent hard working Texans at Solarwinds will lose their jobs over this...but the important thing is that our national leadership not take this seriously and publicly remark on this disaster.  

Excuse me sir, but haven’t you heard the government is focusing on getting rid of the dreaded low pressure shower heads. Yeah, didn’t think so. Priorities. 

[BrazilHorn 6613]

26 minutes ago, Lobo said:

I, for one, am glad to see the Federal Government not rushing to judgement and not condemning the foreign actors behind this.  Sure hundreds, maybe thousands, of innocent hard working Texans at Solarwinds will lose their jobs over this...but the important thing is that our national leadership not take this seriously and publicly remark on this disaster.  

I’ve a friend over there who is super stressed. Deals falling out, everyone in total cya mode etc. Sounds like total shitshow 

[Captainant 27680]

On 12/17/2020 at 2:09 AM, pacman said:

Solarwinds got hacked last Spring. The hackers unserted a backdoor dll into the next Orion update that a ton of people installed around March 2020.

Ince Solarwinds pushed that update, and the companies installed it, Russian hackers had undetectable access to everything. Solarwind installs operate with the highest privileges and the hackers made all of the malicious activity appear identical to the products activity.

 

We still wouldn't know of this if the hackers hadn't gotten greedy and went after a somewhat competent company in FireEye. FireEye's great IR is what led to this discovery.

 

The current claim by Solarwinds is only 18,000 of their 330,000 customers downloaded the malicious update.

Of course, that damn update was still up, and accessible for download from Solarwinds, as late as yesterday. So, their competence to provide accurate info is very much in question.

 

2020 will 2020

 

I'll do you one better, the "hack" was just correctly guessing a password on one of their development boxes: "solarwinds123" 

[TXSG8R 3362]

58 minutes ago, Shaggy3.0 said:

This will not age well.

PW: "solarwinds123"  // <= LOLOL.

https://www.newsweek.com/solarwinds-update-server-could-accessed-2019-using-password-solarwinds123-report-1554986

Solar Winds is TOAST.

Meh, we will see.  Companies hate having to change streams on IT investments due to the cost and infrastructure challenges.  I think you'll see plenty hang on just because its the cheaper/easier option, especially if they didnt get hacked.  Solarwinds will take a beating, but their market share will probably see them through.  

 

[RPM 26322]

On 12/17/2020 at 7:51 AM, SubliminalHorn said:

lol that won’t come until 1/20 at the earliest 

They have 30+ days to mine away. What will be left secure?

[Captainant 27680]

22 minutes ago, TXSG8R said:

Meh, we will see.  Companies hate having to change streams on IT investments due to the cost and infrastructure challenges.  I think you'll see plenty hang on just because its the cheaper/easier option, especially if they didnt get hacked.  Solarwinds will take a beating, but their market share will probably see them through.  

 

Of the four businesses I work with that use SolarWinds, three of them are in the planning stages of ripping out SolarWinds by Q2'21 in favor of Prometheus+Grafana or other open source tools for monitoring.

SolarWinds is gonna be toast, risk averse enterprise IT orgs are running from it like the plague

[Captainant 27680]

As an aside - it's going to be interesting to see how we as a nation handle this after Jan 20. US cybersecurity has lost a large chunk of its funding to pay for the WALL, and the leadership of those agencies have been hollowed out. There's not much apparatus in place to even investigate the extent of the breach (according to the ousted cybersecurity officials)

Sure is a motherfucker of a clapback for letting the genie out of the bottle with Stuxnet though

[Blotto 23559]

Quote

When reached for comment by Newsweek, Kumar forwarded his email correspondence with SolarWinds. He first notified the company of the issue on November 19, 2019. SolarWinds' information security team responded a few days later on November 22, 2019.

"Thank you again for reporting the misconfiguration in a responsible manner. The GitHub repository misconfiguration has been addressed and it's no longer publicly accessible, also treatment has been applied to the exposed credentials. We'd like to ask that you verify this on your end," the team wrote in response to Kumar.

Can someone explain the significance of this. I guess I think of Github as a cloud hosted tool for software development, version control, source code management etc... I cant believe a company like Solarwinds would keep their source code on external infrastructure, so  presumably this means that you can run also run github on your own server infrastructure? Also what is the meaning of "update server" in this context? A separate article on Reuters on the topic makes this seem like evidence of poor security practices in general, but not necessarily relevant for the current shitstorm. 

https://www.reuters.com/article/global-cyber-solarwinds/hackers-at-center-of-sprawling-spy-campaign-turned-solarwinds-dominance-against-it-idUSKBN28P2N8

Quote

Security researcher Vinoth Kumar told Reuters that, last year, he alerted the company that anyone could access SolarWinds’ update server by using the password “solarwinds123”

“This could have been done by any attacker, easily,” Kumar said.

Neither the password nor the stolen access is considered the most likely source of the current intrusion, researchers said.

 

[TXSG8R 3362]

1 hour ago, Captainant said:

Of the four businesses I work with that use SolarWinds, three of them are in the planning stages of ripping out SolarWinds by Q2'21 in favor of Prometheus+Grafana or other open source tools for monitoring.

SolarWinds is gonna be toast, risk averse enterprise IT orgs are running from it like the plague

Ironically open source is very hard to get approved on DoD networks (I assume for gov’t networks as a whole, but I don’t have direct experience outside of DoD). As much as Solarwinds is taking a bath on this, there’s a handful of gov’t people that are also going to get ass blasted over this because all of this is pretty black and white in RMF. They either approved mitigations that were faulty, or they fudged risk assessments and vulnerability reporting.  Proper defense in depth / layering would also have reduced the impacts of the breaches, which is also a very big deal within gov’t RMF. As much as we laugh about solarwinds123, the bigger issue will be the lax IT dep’t at treasury that gave their “good” admin the keys to everything and his credentials gave the hackers access to the good stuff. 
 

Vulnerabilities and exploits are a when, not if. Your cyber security posture should expect them and be designed to minimize the impact and catch as quickly as possible. That’s why I don’t expect the impact to Solarwinds to be as massive as some people are predicting. Microsoft had their 2FA bypassed as well during this breach, and there have been plenty of other IT/tech heavyweights that have had exploits/breaches (how many of us are still using intel chips?). Market share and response will drive how Solarwinds weathers this. 

[Captainant 27680]

1 minute ago, TXSG8R said:

Vulnerabilities and exploits are a when, not if. Your cyber security posture should expect them and be designed to minimize the impact and catch as quickly as possible. That’s why I don’t expect the impact to Solarwinds to be as massive as some people are predicting. Microsoft had their 2FA bypassed as well during this breach, and there have been plenty of other IT/tech heavyweights that have had exploits/breaches (how many of us are still using intel chips?). Market share and response will drive how Solarwinds weathers this. 

It's overstating it somewhat to say that MS's MFA was breached - the attackers were able to steal a private key from MS using their SolarWinds exploit, and then craft a valid (but forged) session cookie using that private key to show that they had MFA'd recently in order to bypass the MFA altogether (primary source). So while the tech wasn't breached and is still sound, the security process surrounding it failed in that they did not rotate keys after the breach became known, enabling the OWA MFA bypass.

If you're into that kind of thing, read through the link for a more thorough technical analysis from actual security researchers and not an architecture person like me lol

[TXSG8R 3362]

23 minutes ago, Blotto said:

Can someone explain the significance of this. I guess I think of Github as a cloud hosted tool for software development, version control, source code management etc... I cant believe a company like Solarwinds would keep their source code on external infrastructure, so  presumably this means that you can run also run github on your own server infrastructure? Also what is the meaning of "update server" in this context? A separate article on Reuters on the topic makes this seem like evidence of poor security practices in general, but not necessarily relevant for the current shitstorm. 

https://www.reuters.com/article/global-cyber-solarwinds/hackers-at-center-of-sprawling-spy-campaign-turned-solarwinds-dominance-against-it-idUSKBN28P2N8

 

Those are great questions. Access to that server is still a long leap from them pushing an exploited update out to the world. To tack on, what is their process for creating updates, and what credentials are required for submitting/approving them? How did an update get through their QA process with foreign code?  How does external traffic to/from their software not get sniffed and audited to match up with expected traffic and destinations during testing, both internally, and externally during security scans (gov’t has quarterly or monthly scans depending on the org)?  Ports, protocols, and services are usually white listed on gov’t networks, so how did this traffic get back to an external control server?  

[TXSG8R 3362]

2 minutes ago, Captainant said:

It's overstating it somewhat to say that MS's MFA was breached - the attackers were able to steal a private key from MS using their SolarWinds exploit, and then craft a valid (but forged) session cookie using that private key to show that they had MFA'd recently in order to bypass the MFA altogether (primary source). So while the tech wasn't breached and is still sound, the security process surrounding it failed in that they did not rotate keys after the breach became known, enabling the OWA MFA bypass.

If you're into that kind of thing, read through the link for a more thorough technical analysis from actual security researchers and not an architecture person like me lol

Yea, I'm a network guy primarily, just got firehosed with RMF as the CyberSec lead for my program so I know more than Id ever care to about all the other controls.  

I read this like these were sequential events  If so, the 2FA exploit occurred prior to the Orion breach, right?  I know later on in the article they show how they hit it again during the Solarwinds breach, but it occurred earlier as well?

Quote

At one particular think tank, Volexity worked three separate incidents involving Dark Halo. In the initial incident, Volexity found multiple tools, backdoors, and malware implants that had allowed the attacker to remain undetected for several years. After being extricated from the network, Dark Halo then returned a second time, exploiting a vulnerability in the organization’s Microsoft Exchange Control Panel. Near the end of this incident, Volexity observed the threat actor using a novel technique to bypass Duo multi-factor authentication (MFA) to access the mailbox of a user via the organization’s Outlook Web App (OWA) service. Finally, in a third incident, Dark Halo breached the organization by way of its SolarWinds Orion software in June and July 2020.

 

[TXSG8R 3362]

2 hours ago, Captainant said:

Of the four businesses I work with that use SolarWinds, three of them are in the planning stages of ripping out SolarWinds by Q2'21 in favor of Prometheus+Grafana or other open source tools for monitoring.

SolarWinds is gonna be toast, risk averse enterprise IT orgs are running from it like the plague

I forgot to ask earlier, are those companies transitioning their Solarwinds budgets to code analysis / monitoring for their new open source toys?  While taking more of their IT security in house is good on paper, I wonder if they fully understand what they’re are signing up for. Open source isn’t any more secure than an IP product, you’re just signing up for the code security in house, which can be daunting depending on the size of your org/needs. Hence why the gov’t won’t go that route, and Solarwinds will likely keep on rolling with gov’t contracts. As much as they distrust companies, they distrust their IT staff’s competence even more. 

[Blotto 23559]

5 hours ago, TXSG8R said:

I forgot to ask earlier, are those companies transitioning their Solarwinds budgets to code analysis / monitoring for their new open source toys?  While taking more of their IT security in house is good on paper, I wonder if they fully understand what they’re are signing up for. Open source isn’t any more secure than an IP product, you’re just signing up for the code security in house, which can be daunting depending on the size of your org/needs. Hence why the gov’t won’t go that route, and Solarwinds will likely keep on rolling with gov’t contracts. As much as they distrust companies, they distrust their IT staff’s competence even more. 

Based upon my somewhat limited interaction with government IT staff, that distrust is probably well earned. 

[TXSG8R 3362]

1 hour ago, Blotto said:

Based upon my somewhat limited interaction with government IT staff, that distrust is probably well earned. 

Absolutely, the way gov’t codes for jobs, especially for civil service positions, leads to a lot of very unqualified people in important positions. And with contractors, you’re only as good as the staff that wrote the requirements.  That’s why relying on skilled people managing open-source software is a non-starter for the bulk of gov’t IT jobs.  There aren’t enough skilled people to run it, and even fewer skilled GS managers to run those teams.  They will pay Solarwinds type outfits all the money in the world for turn-key solutions, they don’t have any choice. 

[TexasEd 8067]

@immamac Can we change thread title to Solarwinds supply chain attack or something like that?

 

Also,

Watch this fascinating video on the topic.  This really sheds some light on the complexity and scale of the problem and the response dilemma.

 

 

 

 

[immamac 42865]

Done.

[StruggleBus 3922]

On 12/14/2020 at 3:45 PM, JimmyJames said:

I’ve sued them before. They are a shitty, disorganized and poorly run company. This doesn’t surprise me at all. 

At least they aren't lawyers. Advantage solarwinds

[UT_OB1 2224]

Csb: my neighbor works for a contractor for NASA. They routinely work with Russia on the space station (guidance I think?...). Anyway, they had along week and pretty much shut everything down.  Russian counterparts were frustrated because they couldn’t get some information.  Gust of the ensuing conversation was along the lines of “well quit hacking our shit guys”

[Dbeasy 15068]

I’ve run a few monitoring companies over the years and have a free schedule now to pursue a new startup. Is there a startup opportunity here? Anyone want to evaluate a newco opportunity? I have access to VC funding.

[immamac 42865]

I have thought a lot about it Dbeasy - there is going to be a big push to replatform in the near future with the level of exposure that companies had.
 

They can use the underlying hardware, but everything from the hypervisor up is going to have to get blown away and started fresh and migrated through some logical capacity. I can't think of anything quick enough from a software perspective to facilitate this better and in the time needed to go to market, but a services play is definitely there. 

[smokebomb 357]

19 hours ago, immamac said:

They can use the underlying hardware, but everything from the hypervisor up is going to have to get blown away and started fresh and migrated through some logical capacity. 

Actually- there is serious talk that if you got breached, you may also have to replace all your networking hardware. I believe the Solarwinds install required admin privileges to the networking gear (and if not required, a lot of admins gave it to them). With that access, you could install your own customized firmware. At this point, the hardware can’t be trusted & you have to pull it out & replace. I believe the majority of government agencies will go this route, if not also large enterprises that were known victims.

As for Solarwinds, there was already talk pre-attack they were going to split up into two different companies. My guess is that this split is expedited, the spin-off gets all the good people and money while “Solarwinds” eventually gets shuttered. 

[Guest Lobo]

the most important thing to remember is that if there is a spin-off, it's that Russia did nothing wrong and we should not publicly condemn them under any circumstances and that thousands of innocent, hard-working Texans should be laid off and made to suffer for something they had nothing to do with.  That's the real lesson here.  

[Blotto 23559]

18 minutes ago, smokebomb said:

Actually- there is serious talk that if you got breached, you may also have to replace all your networking hardware. I believe the Solarwinds install required admin privileges to the networking gear (and if not required, a lot of admins gave it to them). With that access, you could install your own customized firmware. At this point, the hardware can’t be trusted & you have to pull it out & replace. I believe the majority of government agencies will go this route, if not also large enterprises that were known victims.

As for Solarwinds, there was already talk pre-attack they were going to split up into two different companies. My guess is that this split is expedited, the spin-off gets all the good people and money while “Solarwinds” eventually gets shuttered. 

Probably a naive question, but is it impossible to wipe the hardware and reinstall firmware you know isn't malicious? Is it just not worth the time and money?

Edited December 29, 2020 by Blotto

[kevwun 15798]

You can never know for sure where they hid something on a device if they had admin rights on it.  This was an all timer as far as hacking schemes go.

[smokebomb 357]

2 minutes ago, Blotto said:

Probably a naive question, but is it impossible to wipe the hardware and reinstall firmware you know isn't malicious? Is it able to just not worth the time and money?

Not a naive question - you can reinstalled a known, good firmware but you have no idea what other areas on the hardware they may have left a backdoor. They could have installed a firmware that reflashed a chip that normally doesn’t get modified during a normal update. Reflashing the network gear to a known good will fix part of the problem but not the secondary chip they also reflashed/hacked. The reality is they probably didn’t do that but you can’t be certain & because these are classified networks, you have to assume they left a backdoor. So out comes all the network hardware. And they may even have to replace server hardware since they could have also gotten the credentials to manage the server firmware. 

[StruggleBus 3922]

22 hours ago, Dbeasy said:

I have access to VC funding.

That's pretty much meaningless these days. Any fuckstick with a shitty idea that can put together a slide deck and say SaaS a lot has access to VC funding. 

[Guest Lobo]

You don’t even need that.  An exposed labia and a webcam can get you funding these days. 

[TexasEd 8067]

I've been reading about how MS Office 365 could have also been compromised or used as a vector for spread.  Microsoft is now admitting that their source code was compromised.

 

https://gizmodo.com/microsoft-says-solarwinds-hackers-also-broke-into-its-s-1845974783?traffic_source=Connatix

 

 

[TXSG8R 3362]

I received an IAVM for Solarwinds from the DoD today.  Directed to use a DoD specific patch and validate the hash, and specifically NOT use any patches from Solarwinds.  3 week turnaround with a trusted patch, not completely horrible I guess?

[Captainant 27680]

3 minutes ago, TXSG8R said:

I received an IAVM for Solarwinds from the DoD today.  Directed to use a DoD specific patch and validate the hash, and specifically NOT use any patches from Solarwinds.  3 week turnaround with a trusted patch, not completely horrible I guess?

[F250 28781]

On 12/28/2020 at 12:32 PM, UT_OB1 said:

Csb: my neighbor works for a contractor for NASA. They routinely work with Russia on the space station (guidance I think?...). Anyway, they had along week and pretty much shut everything down.  Russian counterparts were frustrated because they couldn’t get some information.  Gust of the ensuing conversation was along the lines of “well quit hacking our shit guys”

A number of years ago I was taking a SANS course, Threat Hunting/DFIR course as I recall. There is a lot of lab work in their courses so at the beginning of the week a young Information Security analyst from NASA asked to partner with me.

She was 23 and fresh out of the Air Force where she did Cybersec for several years. Pretty smart but still very inexperienced. The interesting thing about her was she had the Scooby Doo Velma (Cosplay version) thing going on. So hot but didn't know it. She would wear snug Linux themed tee shirts with snug shorts that were higher than mid thigh, very smooth and toned legs. Cute face with eye glasses and she would often do arm hugs everytime we figured out a technical problem. Completely oblivious to how hot she was.

I forgot where I was going with this... I'll be in my bunk.

 

[Trey3216 15347]

I've seen this thread on here for a few weeks and figured we were talking about getting ready for a solar EMP.  Carry on  

[TexasEd 8067]

On 1/7/2021 at 7:44 AM, TXSG8R said:

I received an IAVM for Solarwinds from the DoD today.  Directed to use a DoD specific patch and validate the hash, and specifically NOT use any patches from Solarwinds.  3 week turnaround with a trusted patch, not completely horrible I guess?

Good,  I started the program to implement them at a previous job back in the day.