Ransomware attack today for me

[Brew 3049]

6 hours ago, BearSchlong said:

@Bevo RAID, in various arrays, helps prevent against data loss in the event of one/multiple hard drive failures, it's an entirely separate thing.  With 2 drives you could set up RAID 1, or Mirrored drives.  One could die and you could replace it and the RAID controller would rebuild the failed drive.

Why in the hell do any companies have on-premise servers anymore for ANYTHING?

disclaimer: I sell bandwidth/networks for a living, I get it, not everyone has budget for wire speed fiber connectivity, but it's a hell of a lot simpler to secure data centers/cloud platforms against the kind of bandits that hit Stanco today.

I've seen probably a dozen SMBs get hit by ransomware assholes.

@markstanco did you call your insurance carrier yet?

 

We got hit last summer. They came in through an app that our IT company used for remote login to computers. It ended up being a huge issue for us with a multimillion dollar ask that I’m not going to get into the details of. I will say offsite backup worked great until we needed our data back quickly. We were bandwidth limited and moving terrabytes worth of data from an offsite data center became a significant issue. We run a dual setup now where a mirror image is created all day and if one view down, we can just roll over the mirrored system.

[Captainant 27679]

16 hours ago, NoRagrets said:

I feel like there’s probably some phishing links on that site. Not clicking.

Check it out on iTunes or Spotify then lol. It's a great listen and is descriptive of the actual problem and why it happens

[markstanco 12577]

15 hours ago, Brew said:

We run a dual setup now where a mirror image is created all day and if one view down, we can just roll over the mirrored system.

Implementing a similar option on this currently.  Looks like out DB file with SAP was corrupt, and we have to go back to October 3 2019.  That's re-entering orders, receiving goods, credit memos, etc.  Going to be a total nightmare.  No IDEA what is in stock right now. 

[JesusSweatDuck 17781]

1 minute ago, markstanco said:

Implementing a similar option on this currently.  Looks like out DB file with SAP was corrupt, and we have to go back to October 3 2019.  That's re-entering orders, receiving goods, credit memos, etc.  Going to be a total nightmare.  No IDEA what is in stock right now. 

shiiiiiit.  Sorry man, that really sucks

[Bevo 13577]

3 minutes ago, JesusSweatDuck said:

shiiiiiit.  Sorry man, that really sucks

Yeah, but it sound like it could be part of a sequel to Office Space.

[JesusSweatDuck 17781]

1 minute ago, Bevo said:

Yeah, but it sound like it could be part of a sequel to Office Space.

Only if @markstanco burns the place down

[Bevo 13577]

29 minutes ago, JesusSweatDuck said:

Only if @markstanco burns the place down

That's the beauty of it. You don't need to burn the place down if you corrupt all the data.

[Irwin F Fletcher 3550]

Only if [mention=1619]markstanco[/mention] burns the place down

That won't work, his business is flame retardant.

[Updawg 2266]

Gotta love america, where even a flaming retard can own and operate a successful business.

[CooterBrown 18014]

How do ransomware attacks originate? Is it really as simple as a phishing email? 

Yes.

My wife had to fire someone last year that continually clicked on fake phishing emails sent by their IT dept. all external emails have “EXTERNAL” appended to the front of subject line. She was instructed to just never click on external emails since her job didn’t have external contact. She could not stop. It was stupid fake phishing emails too like “your XBox has shipped! Click here to view your order.”

I guess some people are too dumb to be employed.

[markstanco 12577]

.

[markstanco 12577]

Found out more today about a customer who got it in june. Their ransom was 1.2mm. Negotiated down to 318k between the insurance company and FBI.

[Updawg 2266]

Found out more today about a customer who got it in june. Their ransom was 1.2mm. Negotiated down to 318k between the insurance company and FBI.

And they paid and got a key to decrypt?

[markstanco 12577]

And they paid and got a key to decrypt?

The VP i was talking to was vaguely familiar with the process, but there was an intermediary between the crooks and insurance company offshore.

 

On edit: paid in bitcoin.

 

[George 493]

If you’re strapped for cash or a really small shop, Go get Veeam community edition and back up to an external hard drive. Disconnect it from the computer and boom! Air gapped backup. Veeam is pretty damn simple to use too. 

Edited February 21, 2020 by George

[woohorn 1131]

Isn't the problem that the ransomware is present in the background for weeks/months?

Wouldn't recent backups have it also?

[Hate 20108]

6 minutes ago, woohorn said:

Isn't the problem that the ransomware is present in the background for weeks/months?

Wouldn't recent backups have it also?

That's why I've mentioned backing up the data as opposed to making "images" of the servers.  You most likely have no idea how long the ransomware has been on your network.  That's also why I am a big proponent of cloud computing now.  Azure or AWS is where most people need to be.

[Hagbard Celine 10867]

I have to be very reserved in what i say due to my job.  I'm sure many other surlers in the industry for decades are going to remain reserved in what they say.

People who have money or have grown businesses often believe it is due to some magical powers bestowed on them by God.

This creates a situation where the "decision makers" decide how things work and what they cost, regardless of any ability or experience on their part to support their "decisions".

Then they hire people who provide confirmation bias ("yes men/women").

This paradigm drove IT when it became "democratized" in the 1980s.  A lot of stupid people came out of the oldline mainframe, minicomputer and networking companies.   They thrived in the 2-tier client/server perpetual-license industry.  People retiring in the last 5 years from executive IT careers lived their entire lives avoiding ownership of outcomes in order to protect their kingdom of blinking lights and the drone armies which tended it.

I have to stop now.

Anyone compromised, twitter is your friend.  Your vendors don't want publicity.  I trust the surl can figure it out from there.

[After irth 1243]

30 minutes ago, Hagbard Celine said:

I have to be very reserved in what i say due to my job.  I'm sure many other surlers in the industry for decades are going to remain reserved in what they say.

People who have money or have grown businesses often believe it is due to some magical powers bestowed on them by God.

This creates a situation where the "decision makers" decide how things work and what they cost, regardless of any ability or experience on their part to support their "decisions".

Then they hire people who provide confirmation bias ("yes men/women").

This paradigm drove IT when it became "democratized" in the 1980s.  A lot of stupid people came out of the oldline mainframe, minicomputer and networking companies.   They thrived in the 2-tier client/server perpetual-license industry.  People retiring in the last 5 years from executive IT careers lived their entire lives avoiding ownership of outcomes in order to protect their kingdom of blinking lights and the drone armies which tended it.

I have to stop now.

Anyone compromised, twitter is your friend.  Your vendors don't want publicity.  I trust the surl can figure it out from there.

Huh?

[NoRagrets 1125]

On 2/19/2020 at 11:42 PM, mulletpelini said:

Oh, I read about them in Time Magazine.

Holly!!!

[KuЯdt 248]

13 hours ago, Hate said:

That's why I've mentioned backing up the data as opposed to making "images" of the servers.  You most likely have no idea how long the ransomware has been on your network.  That's also why I am a big proponent of cloud computing now.  Azure or AWS is where most people need to be.

While moving to the cloud comes with many benefits, it certainly isn't a panacea.  It requires an entirely different skillset to secure cloud than it does your on-premises infrastructure.  And it's not like AWS or Azure do it for you by default, in fact it is the opposite.
How often do you hear about companies being compromised because someone left a production mongodb open to the world, or their AWS keys publicly accessible on GitHub?  All the damn time.
So just saying "move it to the cloud" is dangerous.  By all means make the move to the cloud, but hire some people that know what the fuck they are doing to implement and run it.

 

[Bevo 13577]

16 minutes ago, KuЯdt said:

While moving to the cloud comes with many benefits, it certainly isn't a panacea.  It requires an entirely different skillset to secure cloud than it does your on-premises infrastructure.  And it's not like AWS or Azure do it for you by default, in fact it is the opposite.
How often do you hear about companies being compromised because someone left a production mongodb open to the world, or their AWS keys publicly accessible on GitHub?  All the damn time.
So just saying "move it to the cloud" is dangerous.  By all means make the move to the cloud, but hire some people that know what the fuck they are doing to implement and run it.

 

So, I have a few 8 person offices that run independently. The main programs are an imaging program and patient management system. I backup just the data but have not tried to use the backup data. I don't have an IT guy unless I run into a problem. I have 1 VPN and use a fortinet UTM device but it is only setup to the extent that Fortinet has helped me. Each office has a server, a Dell R430. What would you recommend?

[markstanco 12577]

14 hours ago, George said:

If you’re strapped for cash or a really small shop, Go get Veeam community edition and back up to an external hard drive. Disconnect it from the computer and boom! Air gapped backup. Veeam is pretty damn simple to use too. 

Going with Datto.

14 hours ago, woohorn said:

Isn't the problem that the ransomware is present in the background for weeks/months?

Wouldn't recent backups have it also?

Numerous images, and any changes (as I am told) will be logged and an email sent. With the Datto, if we go down we can work off the image(s) on their servers until we are live again. 

[Hate 20108]

While moving to the cloud comes with many benefits, it certainly isn't a panacea.  It requires an entirely different skillset to secure cloud than it does your on-premises infrastructure.  And it's not like AWS or Azure do it for you by default, in fact it is the opposite.
How often do you hear about companies being compromised because someone left a production mongodb open to the world, or their AWS keys publicly accessible on GitHub?  All the damn time.
So just saying "move it to the cloud" is dangerous.  By all means make the move to the cloud, but hire some people that know what the fuck they are doing to implement and run it.
 

I don’t disagree at all. I was making the assumption that most people wouldn’t know how to move to the cloud in their own and thus would engage someone with the required skill set.

[F250 28777]

58 minutes ago, KuЯdt said:

While moving to the cloud comes with many benefits, it certainly isn't a panacea.  It requires an entirely different skillset to secure cloud than it does your on-premises infrastructure.  And it's not like AWS or Azure do it for you by default, in fact it is the opposite.
How often do you hear about companies being compromised because someone left a production mongodb open to the world, or their AWS keys publicly accessible on GitHub?  All the damn time.
So just saying "move it to the cloud" is dangerous.  By all means make the move to the cloud, but hire some people that know what the fuck they are doing to implement and run it.

 

This man speaks the truth. Moving to the cloud will likely make you even more vulnerable than an on premise setup if your organization doesn't have a complete understanding of how to  manage a cloud environment. Many traditional IT people with only physical data center experience struggle to comprehend all of the ins and outs of the cloud, especially when it comes to networking. Anyone who migrates, needs to be ready to invest in new support staff that costs more than your average IT staff or just cough up the money to a managed cloud service company. Either way, prepare your anus.

 

[Paco 1934]

I have to be very reserved in what i say due to my job.  I'm sure many other surlers in the industry for decades are going to remain reserved in what they say.
People who have money or have grown businesses often believe it is due to some magical powers bestowed on them by God.
This creates a situation where the "decision makers" decide how things work and what they cost, regardless of any ability or experience on their part to support their "decisions".
Then they hire people who provide confirmation bias ("yes men/women").
This paradigm drove IT when it became "democratized" in the 1980s.  A lot of stupid people came out of the oldline mainframe, minicomputer and networking companies.   They thrived in the 2-tier client/server perpetual-license industry.  People retiring in the last 5 years from executive IT careers lived their entire lives avoiding ownership of outcomes in order to protect their kingdom of blinking lights and the drone armies which tended it.
I have to stop now.
Anyone compromised, twitter is your friend.  Your vendors don't want publicity.  I trust the surl can figure it out from there.

Bingo

[RollLeft 5045]

On 2/19/2020 at 12:56 PM, DFWTexEx said:

Client of ours got hit on Monday.  USD$1.5m ask.  They are in financial services and have a pretty sophisticated IT group (or so we think).   We train and train and train, but still have employees fail our internal tests. 

Damn you porn!

[RollLeft 5045]

15 hours ago, Hagbard Celine said:

I have to be very reserved in what i say due to my job.  I'm sure many other surlers in the industry for decades are going to remain reserved in what they say.

People who have money or have grown businesses often believe it is due to some magical powers bestowed on them by God.

This creates a situation where the "decision makers" decide how things work and what they cost, regardless of any ability or experience on their part to support their "decisions".

Then they hire people who provide confirmation bias ("yes men/women").

This paradigm drove IT when it became "democratized" in the 1980s.  A lot of stupid people came out of the oldline mainframe, minicomputer and networking companies.   They thrived in the 2-tier client/server perpetual-license industry.  People retiring in the last 5 years from executive IT careers lived their entire lives avoiding ownership of outcomes in order to protect their kingdom of blinking lights and the drone armies which tended it.

I have to stop now.

Anyone compromised, twitter is your friend.  Your vendors don't want publicity.  I trust the surl can figure it out from there.

Shall we include that they are Pedo's "allegedly" or just that they suck at IT.

[RollLeft 5045]

16 hours ago, markstanco said:

Found out more today about a customer who got it in june. Their ransom was 1.2mm. Negotiated down to 318k between the insurance company and FBI.

So this info eliminates American hackers imo.  

[BearSchlong 8098]

This man speaks the truth. Moving to the cloud will likely make you even more vulnerable than an on premise setup if your organization doesn't have a complete understanding of how to  manage a cloud environment. Many traditional IT people with only physical data center experience struggle to comprehend all of the ins and outs of the cloud, especially when it comes to networking. Anyone who migrates, needs to be ready to invest in new support staff that costs more than your average IT staff or just cough up the money to a managed cloud service company. Either way, prepare your anus.
 

My assumption would be a fully managed cloud solution, either SAAS or a company to manage virtualization.

[Brew 3049]

On 2/21/2020 at 12:29 PM, RollLeft said:

So this info eliminates American hackers imo.  

Ours was out of one of the former Soviet bloc countries. They had offices leased in the US with equipment in the offices that could not be back traced past that location. It was a very sophisticated system, but luckily we have a solid list of contacts.

[F250 28777]

22 hours ago, BearSchlong said:

On 2/21/2020 at 10:41 AM, F250 said:

This man speaks the truth. Moving to the cloud will likely make you even more vulnerable than an on premise setup if your organization doesn't have a complete understanding of how to  manage a cloud environment. Many traditional IT people with only physical data center experience struggle to comprehend all of the ins and outs of the cloud, especially when it comes to networking. Anyone who migrates, needs to be ready to invest in new support staff that costs more than your average IT staff or just cough up the money to a managed cloud service company. Either way, prepare your anus.
 

My assumption would be a fully managed cloud solution, either SAAS or a company to manage virtualization.

Yeah, that's the prepare your anus part.

[The University 851]

[RamjetFDO 10577]

Oh hai.

[EE2B 1565]

That actually looks a lot like Stanco.

[Parliament 14260]

 

[markstanco 12577]

That actually looks a lot like Stanco.

I do have a black shirt.

[markstanco 12577]

Back online with SAP today, and damn, it was a long ass day. Just hit the couch and opened a shiner. Tomorrow will be equal.

I dont know how long it takes a company to get ransomware, with a shitty backup like us (october 3rd) but I would think being down for under a week is pretty decent from what I have read. We still have a lot of work to do.

Going forward, I have learned a lot. For those small business owners, take heed on what I can tell you about ransomware.

Close all RDP ports. I used to log in multiple times a day with my phone to see what was shipping. Not worth it.

Backups: physical backups daily. Swap a drive every day and throw it in your fireproof filing cabinet. I was lazy and did this every few months. Hence only having data from october 3rd and now we have to enter every single order from then, and receivables, and credit memos, etc. Nightmare stuff. About 50 orders a day so, lots of fun.

Going forward..as mentioned drive swaps. Also image cloud backups. And my now backups are with 3 services if you include daily drive swaps.

Oh, and the MOST IMPORTANT is get rid on your server the "administrator " account. I was retarded keeping that one open. That's half of the key to hackers getting into your server. They have the username, just pound away at passwords.

[George 493]

I don’t want to pile on but anyone with rdp open to the internet is just asking to get pwned. If you are doing this, you will be compromised at some point, assuming you aren’t already. This is security 101 and why things like VPN exist. 
 

sorry for your hell week. It’s a bitch, and much more sophisticated setups have been hacked. 

[Captainant 27679]

15 hours ago, markstanco said:

Backups: physical backups daily. Swap a drive every day and throw it in your fireproof filing cabinet. I was lazy and did this every few months. Hence only having data from october 3rd and now we have to enter every single order from then, and receivables, and credit memos, etc. Nightmare stuff. About 50 orders a day so, lots of fun.

What's your current backup solution(s)? Depending on the size of your target data, it may be worth it to build a vault to do daily/weekly snapshot backups for the trailing month so you'll have that separate airgapped appliance holding copies of your data at various points in time so you can do point-in-time restores for forensic work in the case of breaches like this one, or just to roll back someone deploying completely fucked business logic that mangled your records.

And I know it's easy to say, but really, running on a cloud platform addresses many of these risks and mitigations directly by decoupling the infrastructure from the runtime environment of your workload. They may pwn that box or fleet of boxes, but you can just roll back in time and restore to wipe away the impact (assuming you have good defense in depth).

[pacman 7357]

16 hours ago, markstanco said:

Back online with SAP today, and damn, it was a long ass day. Just hit the couch and opened a shiner. Tomorrow will be equal.

I dont know how long it takes a company to get ransomware, with a shitty backup like us (october 3rd) but I would think being down for under a week is pretty decent from what I have read. We still have a lot of work to do.

Going forward, I have learned a lot. For those small business owners, take heed on what I can tell you about ransomware.

Close all RDP ports. I used to log in multiple times a day with my phone to see what was shipping. Not worth it.

Backups: physical backups daily. Swap a drive every day and throw it in your fireproof filing cabinet. I was lazy and did this every few months. Hence only having data from october 3rd and now we have to enter every single order from then, and receivables, and credit memos, etc. Nightmare stuff. About 50 orders a day so, lots of fun.

Going forward..as mentioned drive swaps. Also image cloud backups. And my now backups are with 3 services if you include daily drive swaps.

Oh, and the MOST IMPORTANT is get rid on your server the "administrator " account. I was retarded keeping that one open. That's half of the key to hackers getting into your server. They have the username, just pound away at passwords.

at least you have learned, i have seen entities who have the same exact incident within a year of each other, with the exact same issues that should have been remedied

convenience and speed of deployment always overtakes security

[markstanco 12577]

What's your current backup solution(s)? Depending on the size of your target data, it may be worth it to build a vault to do daily/weekly snapshot backups for the trailing month so you'll have that separate airgapped appliance holding copies of your data at various points in time so you can do point-in-time restores for forensic work in the case of breaches like this one, or just to roll back someone deploying completely fucked business logic that mangled your records.
And I know it's easy to say, but really, running on a cloud platform addresses many of these risks and mitigations directly by decoupling the infrastructure from the runtime environment of your workload. They may pwn that box or fleet of boxes, but you can just roll back in time and restore to wipe away the impact (assuming you have good defense in depth).

I posted a few posts back our solution, I feel good about it. Triple redundancy. One local and two offsite. The local is a daily image and the offsite is real time.

[Updawg 2266]

Change passwords often, keep your shit patched/updated and tell your users not to open or click on shit from email

[Updawg 2266]

Your ap,accounting, hr people will be the next to get spear phished.

[NotActuallyALonghorn 4976]

Our company recently switched from forcing us to change our passwords every quarter or so to requiring us to have nearly a full sentence password. The theory is that with a large number of employees having to change passwords often they will likely be something like orangejuice001, orangejuice002, etc. So having a longer more secure password is better in the long run. I think we have to have something like 15 characters total now with a mixture of capital letters and punctuation thrown in there.

[markstanco 12577]

Change passwords often, keep your shit patched/updated and tell your users not to open or click on shit from email

Good point here. They are all required to ask on clicking on "blue texts" within email to their super. That wasnt the issue here. Brute force was.

[Captainant 27679]

37 minutes ago, markstanco said:

1 hour ago, Updawg said:

Change passwords often, keep your shit patched/updated and tell your users not to open or click on shit from email

Good point here. They are all required to ask on clicking on "blue texts" within email to their super. That wasnt the issue here. Brute force was.

Another low hanging fruit is to enable/use 2 factor authentication, ideally via a gemalto or RSA token or a yubikey. It completely mitigates brute force password attacks by requiring physical possession of that token to authenticate as well as the password. 

It's standard practice for secure environments, and most ADs and identity services support it natively nowadays

[markstanco 12577]

Another low hanging fruit is to enable/use 2 factor authentication, ideally via a gemalto or RSA token or a yubikey. It completely mitigates brute force password attacks by requiring physical possession of that token to authenticate as well as the password. 
It's standard practice for secure environments, and most ADs and identity services support it natively nowadays

Doing similar. If I need to log in, I will have an app that the numeric password changes every 30 seconds.

[F250 28777]

30 minutes ago, markstanco said:

That wasnt the issue here. Brute force was.

 

You might want to give this a read.

https://docs.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/account-lockout-threshold

It is a simple control and will disrupt a brute force attack. You should also log failed login attempts and send them somewhere when an account is locked so you know this shit is occurring.

The open source version of AlienVault is free. There are others you could get at no charge but you should probably get something.

https://cybersecurity.att.com/products/ossim

There are probably a lot of other things you can do but these are basic controls that would have prevented this occurring. Well blocking TCP 3389 (RDP) would have made a huge difference but these other controls would help out with a lateral attack which could always happen again if they had full access to your environment. Those dudes are known to return via a backdoor after getting paid.

[markstanco 12577]

 
You might want to give this a read.
https://docs.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/account-lockout-threshold
It is a simple control and will disrupt a brute force attack. You should also log failed login attempts and send them somewhere when an account is locked so you know this shit is occurring.
The open source version of AlienVault is free. There are others you could get at no charge but you should probably get something.
https://cybersecurity.att.com/products/ossim
There are probably a lot of other things you can do but these are basic controls that would have prevented this occurring. Well blocking TCP 3389 (RDP) would have made a huge difference but these other controls would help out with a lateral attack which could always happen again if they had full access to your environment. Those dudes are known to return via a backdoor after getting paid.

Yea I have learned a lot over the last week.

I'm good now. RDP is closed, only open port is one for the cameras which is outside the network. Cant close 80 and 443 for obvious reasons.