What are the signs of a breach? Are you catching them all? Or do you have a false sense of security (yes, pun intended) when it comes to all things cyber within your organization? How about your systems, applications, data, end users and business partners? And how about your accounts and the business services you use to run the company?
“One mustn’t forget: being compliant and being secure are two different things,” says Demetrios Lazarikos (Laz), founder of Blue Lava.
Research from Accenture might help you answer these questions. According to the global management consulting and professional services firm, organizations are preventing most (87 percent) but not all of the targeted attacks that come their way.
While 87 percent might seem like a decent number, it points to the fact that some attacks are still making it through (or around) the defenses that companies are deploying.
Or perhaps, more likely, the attackers are walking right in, since no protections are in place in some instances. Further to this point, the same Accenture report states that companies are still facing two to three security breaches per month. Again, to emphasize the obvious, just because a company has achieved a point-in-time state of compliance for PCI-DSS or HIPAA, for example, doesn’t mean they are immune to attacks and the potential for a breach.
“As companies are building out their security program, it’s imperative that they recognize that the program is a constantly evolving solution that must not only address the compliance requirements but also align to the business requirements,” adds Laz.
As Thomas Richards, the senior security consultant at Synopsys Software Integrity Group (SIG), says, “Every company is a target now, not just for their data but also for their computing resources.”
Laz adds that “cybercriminals often use small- to mid-sized companies as their initial vector, pivoting from the smaller company systems to larger organizations while covering their tracks.”
What’s more troublesome, perhaps, is that 71 percent of those included in this same Accenture survey say that cyber attacks are still a ‘bit of a black box,’ and they do not know when or how attacks will impact their organization.
Personally, I read this statistic as, ‘Even if I am attacked, I won’t know how to spot an attack unless the security applications and log monitoring systems tell me they stopped something.’
Of course, this generally refers to the detection of the attack payload itself; the detection may be possible if the systems or data are damaged, destroyed, or taken offline—not so much if the data is slowly extracted from the organization under the radar.
What this scenario highlights is that the data is lacking and/or being misread. Therefore, it’s really about analyzing data. One could argue that it always has been, but statistics like these make it easier to justify.
“Data analytics is key!” says Laz. “Data is becoming critical for InfoSec teams to analyze. Companies must consider subscribing to threat intelligence services now with advanced analytics if they are to understand when they will be targeted and how to respond to a potential attack.”
The bottom line: if your company’s systems, accounts or data have been compromised, you probably don’t know it yet. And, therefore, the answer to the questions from this article is that you’ve been coasting on a false assumption of security and protection.
Still don’t believe the hype? Let’s go back to Accenture’s study for one more statistic. It seems that security teams are still only finding 64 percent of the attacks they face, and only 24 percent are able to detect less than half of all breach attempts.
More proof behind this failure to protect can be found in another study, this time by Verizon. Verizon found that only 14 percent of the responding organizations had implemented even the most basic cybersecurity practices. That means 86 percent have not!
And when the breach is finally detected, there’s often a significant time gap between the initial breach and its detection. While 10 percent may seem like a small percentage in the grand scheme of things, this number of companies that this figure represents is actually very troubling. Adding salt to the wound, these companies are also taking longer than one month to detect a breach.
The fact that companies don’t know when attacks occur is not all that surprising. “Unless we are talking about intentionally visible attacks like ransomware, wiper malware or hacktivism, in a majority of the cases there is no easy way to tell,” says Rene Kolga, Senior Director of Product Management with Nyotron.
Assuming you have been breached and don’t know it yet because, as Kolga said, “there is no easy way to tell,” how can you spot it?
Learning from the consumer world that has been trained by anti-virus marketing tactics, most people would know to look for ‘slow’ or ‘sluggish’ machines. “In some cases, like with crypto jacking, organizations can detect unusually high and prolonged CPU utilization,” says Kolga.
This definitely holds true in many cases in the corporate world, but not always. “These days, malware or threatening methods are highly sophisticated, so don’t expect the system to be merely running sluggish all the time,” says Shauntinez Jakab, director, product marketing at Virsec. “Performance may only be affected when a certain function or app component is in use or at certain times of the day. So, understand how the system is running when certain tasks or operations are being performed and if they spawn additional processes. One banking exploit only surfaced when the system was being used and a specific service was activated.”
Looking beyond a slowed down system, you actually have to prepare yourself for the act of monitoring and looking for activities, behaviors, and remnants of these things within the organization. Without this preparation, you are missing out on a lot of opportunities to help yourself.
“Proper logging and monitoring are necessary for an organization to identify signs of compromise,” says Richards. “These signs can include technical artifacts as well as human artifacts.”
Granted, monitoring in real-time can be overwhelming.
“What happens when you have so many alerts that you can’t look at them yourself?” asks Laz. “If you don’t know the answer or if the answer is not a positive one for your security program, it’s time to start thinking about how to outsource analytics to a third party.”
Adds Bill Dixon, associate managing director – cybersecurity & investigations for Kroll: “From a technical point of view, logs are your best friend. They are able to tell who did what and when in many incidents. It can be very obvious from the logs that something is not right. In many cases, there is activity from uncommon areas such as other countries that can immediately point to our anomalies that indicate something is going on that shouldn’t be.”
Beyond logs, it’s important to remember where the attacks are coming from. Here are a few additional recommendations from the security industry experts:
- "At least 96 percent of all cyberattacks begin with an email phishing attack despite all of the money and training resources that companies continue to put towards defending employee mailboxes from advanced threats.” - Eyal Benishti, CEO at IRONSCALES
- “Cybercriminals are people too. They make mistakes and sometimes forget about basic security hygiene. There will always be artifacts left internally within the organizations by even the most sophisticated attackers.” - Rene Kolga
- “There is no such thing as a single sign of compromise. If such a simple indicator existed, smart attackers would find a way to disguise it.” - Greg Scott, senior technical account manager at Red Hat and author at Infrasupport.
So, what are the other signs and places to look for them?
That’s exactly what I wanted to find out, so I reached out to the community of experts to help me identify some ways to spot what wasn’t immediately evident. Here’s the list of security experts that contributed to this article:
- Venky Balasubramanian, CEO of Plivo
- Eyal Benishti, CEO at IRONSCALES
- Paul Bischoff, technology and privacy expert with Comparitech.com
- Kevin Bocek, vice president of security strategy and threat intelligence at Venafi
- Dan Desko, senior manager, IT audit and risk advisory services at Schneider Downs
- Bill Dixon, associate managing director - cyber security & investigations, at Kroll
- Sam Elliott, director of security product management at Bomgar
- Shauntinez Jakab, director, product marketing at Virsec
- Rene Kolga, senior director of product management with Nyotron
- Matan Kubovsky, VP research and development at Illusive Networks
- Marc Laliberte, senior security analyst at WatchGuard Technologies
- Demetrios Lazarikos (Laz), founder of Blue Lava
- Dror Liwer, CISO and founder at Coronet
- Thomas Richards, senior security consultant at Synopsys Software Integrity Group (SIG)
- Paul San Soucie, President, Carefree Solutions
- Greg Scott, senior technical account manager at Red Hat and author at Infrasupport
- Elad Shapira, head of research at Panorays
- Ken Spinner, vice president of global field engineering at Varonis
Part 1 of this series will attempt to do the impossible by focusing on how to spot the breach by using some signs that likely exist in a breached environment.
Spotted: Unexpected System Reboots
Aside from machines running sluggishly, one of the closest consumer-like signs you can spot is that of unexpected system reboots. As a one-off event, these may not seem like much. But repeated activity in this area—or a series of these events across many systems in the organization—can be a sign that something is off on one or more computers in the organization.
“Unusual reboots can be an indication of compromise,” says Jakab. Also, be sure to look for servers or desktops running unknown processes.
“If a service begins to run without the user actively requesting it, it may indicate an infected program,” says Shapira. “Another indicator would be some strange process name that is not part of the operating system and is not identified as belonging to legitimate software running on the machine.”
However, aside from the signal, there is also risk of data loss associated with these reboots. Jakab followed up her comment above by saying, “Cold and warm boots could allow sensitive data to be read after supposedly having been deleted. Side-channel attacks are commonly used with cryptography to gather encryption keys on a cold boot, for example. The attack relies on the data remanence property of DRAM and SRAM to retrieve memory content that remains readable in the seconds to minutes after power has been removed.”
Spotted: Strange Application Behavior
Another sign that’s close to the consumer-like signs that we look for as home users of computers is that of unusual application behavior. Sure, sluggish applications could be a sign that an application is doing something it shouldn’t be doing, but the signs can go way beyond this.
“Some less obvious signs that you’ve been compromised include being redirected to websites other than the ones you intended to navigate to,” says Shapira. “Another sign is software programs that you’ve never heard of installing themselves and appearing on your computer without your consent.”
Matan Kubovsky, VP R&D at Illusive Networks, adds “fake and suspicious websites could appear in the browser.”
But strange or unexpected behavior and sluggish activity isn’t limited to the system and applications. Balasubramanian adds, “A spike in activity beyond the normal threshold for an account could also be a sign of a compromise.”
Spotted: Unauthorized Use of Native Tools
Another indicator of compromise could come in the form of good applications being used to perform “non-good” activities. Here’s what experts recommend you look out for:
“Sophisticated attackers use living-off-the-land techniques such as abusing legitimate administrative tools like PowerShell, WMI, etc.,” says Kolga.
Desko warns, “Cybercriminals often make use of native tools within the operating systems they compromise. For example, PowerShell is a tool that is built into every Windows operating system these days and is something that 99% of users should never use. Looking for the use of administrative applications on systems and by users that have no need for them is one way to spot a breach. You can review and lock down who is using PowerShell in your environment to look for potential compromise.”
Spotted: Suspicious IP Addresses and Locales
IP addresses are used to identify “things” on a network that are attempting or succeeding in their access to and communication with other resources on the network. Each organization will have a collection of IP addresses associated with the things they’ve installed on their network. Beyond that, everything connecting with an IP address that doesn’t fit should be looked at suspiciously.
“A key area to watch for is geo/location oddness,” says Elliott. “Watching where connection sources and destinations come from is important. If a printer suddenly starts talking to an IP address based in China, that could be an indicator of compromise. Don't limit your geo/location awareness to just infrastructure though. Watching user login locations is critical. If one of your users based in Virginia logs in from Virginia, then also logs in from Thailand five minutes later, you have another indicator of compromise.”
Laliberte emphasizes, “The sign of compromise could be an administrator login attempt from a foreign IP address. Most network-based threat detection tools these days can geolocate IP addresses, and some can even help point out anomalies automatically. Some SIEM (Security Information and Event Management) systems can help geolocate IP addresses in your logs after the fact too. If you see login attempts from a country that you know you don’t have any employees stationed in, that should be a massive red flag.”
Desko provides an example: “An account is logged in from New York at 10am EST and then Russia at 10:30am EST the same day. That is just an impossible scenario and should never happen.”
Elliot underscores that “the signs would be very visible in logs. Looking for source IP information can be very telling. Even if the source is coming from a VPN, that can be telling. If it isn't a VPN sanctioned by your organization, that is a flag that should be investigated.”
“Seeing your customer log in from a distant country versus the location they originally signed up from is another example,” adds Balasubramanian.
Bischoff points out, “Keep an eye on DNS and IP records in your DHCP system to spot intruders. Remote access from locations where no employees are can be a sign of malware.”
Spotted: Anomalous Internal Network Traffic/Communications
When cybercriminals compromise a system, they generally try to find ways to move laterally across the network within the organization. This lateral movement is often going unnoticed.
“Network admins are always paying attention to the firewall, the so-called north/south traffic, but not so much the east/west internal traffic,” Desko points out. “For example, Suzie and John work in the same department. While they work together, there is no reason for their computers to speak to each other. This could be a tell-tale sign of compromise.”
Also keep an eye on suspicious inbound and outbound network connections warns Shapira. “For instance, sudden peaks in traffic may indicate exfiltration attempts. At other times, an outbound connection might erratically send out signals to an unknown site. The latter was the case in the OPM breach, where the malware was pinging a similarly-named domain as the OPM but was not a domain belonging to the OPM.”
“Look out for peaks in processing power or network traffic to the device(s) in question, possibly at times that don't make sense,” adds Bischoff. “Sustained, heavy use of system and network resources is another sign.”
Spotted: Impossible or Failed Logins (and Account Lockouts)
Laliberte advises that a “Sign of compromise can be an employee getting an error message saying they have made too many login attempts.”
Adds Desko, “We often see password guessing against externally-facing login pages as a prime avenue for cybercriminals. We call this password ‘spraying.’ Some of these systems come inherent with logs and reports that can be viewed to spot these hacking trends, and you will see many invalid attempts in the logs over a period of time. These attacks typically guess one common password against multiple user names (spraying) rather than lots of different passwords against one user (brute force). It requires some technical expertise to get under the hood and review this. Compromise of email systems can be very damaging; we often see that people are pack rats and store sensitive in their email that has no business being there. There are reports in certain systems (e.g. Microsoft Office 365) where organizations can view ‘impossible’ logins."
“A spike in activity beyond the normal threshold on an account could be a sign of a compromise,” adds Balasubramanian.
Adds San Soucie, “Ensuring that all systems have account lock-out after failed login attempts along with complex passwords can help spot attacks or compromised user names. Often times bots are used to attempt to log in to systems using default user names and passwords. There is a growing trend where attackers load the bots with a known list of valid email addresses for that entity and try common passwords. If you find multiple accounts that have been locked out in a short period of time, this can be an indication that your organization is under attack.”
For login analysis, again, logs are your best friend here; in this case, these logs can be located anywhere access is managed and logins take place.
Spotted: Anomalies in Privileged User Activity
Looking for changes in the behavior of privileged users, such as login time and systems accessed, can indicate that systems are compromised. Cybercriminals attempt to capture these accounts and use them to their advantage all the time.
“Some [cybercriminals] are better at covering their tracks than others,” Desko says. “Sometimes they will also try to add disabled or stale user accounts to domain administrator groups. There are rules that can be built with Active Directory to spot these moves.”
Oddly enough, signs don’t always have to point to something that legitimately exists.
Spinner adds, “Admin changes—when a ghost user or employee changes or escalates access privileges—can be an indicator of compromise.”
And, when speaking of admin rights, Shapira recommends that inconsistent admin-level tasks (e.g., user account creation) should also be watched. “For example, in order for malware to install itself and propagate itself within the network, it will typically create a new account. Furthermore, to go under the detection radar of security products, the malware will work to gain higher privileges under that account. A company should monitor activities that enable the creation of privileged accounts as well as unfamiliar or unrecognized account names. These may indicate that there's unusual behavior or that inner systems were breached.”
Spotted: Suspicious or Fake Email Exchanges
Fake email exchanges are particularly frustrating for InfoSec teams. The Verizon 2018 Data Breach Investigations Report reveals that 96% of attacks come through user inboxes.
Email remains the favorite entry point of many attackers, according to Kolga, and Benishti points out, "If a company traditionally receives an invoice on the second week of the month from one of their supply chain partners, and one month they receive the invoice during the fourth week of the month, then they should view that message with extreme suspicion. The same suspicion should be applied to random internal emails. For example, if employees do not receive regular emails from HR about payroll, but several employees receive a message one morning asking them to take an action, then those employees should immediately report that email. In summation, any message that defies what is classified as ‘normal’ should be immediately recognized as a potential attack underway."
Even if the exchange appears to be real, it’s important to be suspicious of exchanges that could have an immediate impact on the business.
“An unusual email from the CEO to the CFO instructing her to wire money urgently to an account is also a sign,” adds Liwer.
Spotted: Outgoing Spam
Blocking incoming spam has always been a priority in order to avoid internal users from wasting their time deleting emails – or worse, clicking on something malicious. But outgoing spam, where a cybercriminal attempts to spoof emails as if they were being sent by someone inside the company, can be damaging too. Receivers of such spam may form a negative opinion of your company or even use the spoofing to conduct the fraudulent activity under your company’s name (domain).
“I've seen companies breached just so their mail servers can be used to propagate spam,” Desko says. “Often times, cybercriminals are smart enough to set a rule to auto delete the spam from the sender’s outbox. You can set up alerts to check for these rules. Alternatively, at the mail server, an administrator can analyze the outgoing email content periodically to review for mass spam.”
Adds San Soucie, “Monitoring outgoing email is key. I’ve had customers that find out about the breach when their email provider blocks an account, their users get hundreds of undeliverable email messages or they are notified that their email domain is on an email blacklist.”
Our desire to access email anywhere, anytime, makes this a bigger challenge for organizations.
“Companies need to be careful when allowing mobile access to their email system,” adds San Soucie. Many of the phones and tablets do not have the same security software that laptops and desktops have. A growing trend is to only allow authorized devices to access email from outside the network. While this adds work to email administrators, it is becoming increasingly necessary to prevent breaches to email.”
Sometimes, you have to ask your staff to understand what’s happening in your environment.
Adds San Soucie, “An issue that can complicate monitoring is that spammers can put valid email addresses in the email that makes it appear that it came from a legitimate address when in fact it came from a different email server. Oftentimes users will get bombarded with undeliverable email error messages and assume their account has been compromised. When this occurs, administrators need to act fast to check the email blacklists and ensure that their domain is not being blocked.”
Spotted: Suspicious or Unexpected Email Settings/Configurations
Many times, email settings are never touched; companies use the defaults that come with the software if on-premises or the service if running in the cloud. Even if the company changes the way their email processes are defined and how email is handled, it may not be reviewed for months or even years. This could lead to trouble.
Email settings and configurations can sometimes be tampered with by cybercriminals so that harmful emails can find their way into and out of a company. There are also cases where settings were inadvertently configured incorrectly, leaving holes open for data to leak or bad actors to enter. Asking end users to be on the lookout for improper settings can serve as a good back up to the technical controls.
“Companies can conduct a user self-audit which consists of reviewing sent emails, email rules that may have been set up, and the creation of folders within their email account,” says Dixon. “This step can eliminate the need for technical methods for identification and let the user be a line of defense on an ongoing basis.”
Spotted: Stolen Machine Identity
When we think about identities, we tend to think about people. We’re somewhat inclined to forget that our computer systems—even some of the services running on them—have identities, too.
“One of the best signs of a cyber attack is identifying a stolen identity,” points out Bocek. “Although most people don’t think about it this way, identities can be stolen from people and from machines. Machines use keys and certificates to identify and authenticate themselves in all kinds of machine-to-machine communication. And since we have lots to tools to help us identify human identity theft, I’m going to focus on machine identity theft because most companies don’t track this nearly as carefully as they track human identities.”
Bocek also recommends, “One way to identify a stolen machine identity is to discover a domain you don’t control that is very similar to yours. These domains are often used for a variety of malicious purposes. Cybercriminals use stolen machine identities to transfer or exfiltrate data through encrypted tunnels since most security tools can’t see inside encrypted traffic. Any breach that involves a large volume of stolen data probably involves a machine identity breach.”
Spinner expanded on this thought by adding, “When users behave like computers by, for example, opening folders at an extremely rapid pace, this can point to a machine being compromised.”
Spotted: Uncommon Fault Triggering
As with most of the other items above, security teams are advised to constantly monitor event logs so that suspicious activities are identified and responded to in a timely fashion. In this instance, it’s suggested that security teams look for server-side input validation failures which can be logged with sufficient user context to identify suspicious activity.
Jakab recommends that companies “review all warnings and errors generated to determine any unusual fault patterns or attempts. Faults often provide a wealth of information to an attacker about the security in place and the application infrastructure. Properly configured alerts can actually indicate that systems have been compromised and data is being exfiltrated by bad actors. Review logs and analyze the events across the system.”
Spotted: Signs of Human Compromise
Signs of human compromise—employees, contractors, and business partners—can be more difficult to spot. Companies need to have an inherent trust in the people they bring on board to help them run the business, so it may be difficult for people within the company to observe and understand that a fellow employee or a third-party vendor is acting suspiciously.
“Some of these activities can include suddenly staying late for projects, working on unassigned projects, and noticing USB devices being plugged into workstations,” says Richards. “You may also notice end users logging into another employee’s workstation and suddenly disconnecting from fellow employees.”
Spinner says to watch out for “unusual data access activity during ‘off’ hours, in the middle of the night, weekends and holidays—where in most cases the user is searching or viewing data irrelevant to their role. Also watch for ghost-user activity, where accounts belonging to former employees can still access your network.”
Spotted: Data Has Lost Its Integrity
Strange characters that appear in a database might be attempts for an SQL injection attack. They may also be a sign that the data has lost its integrity and can’t be trusted for accuracy.
“For instance, the format of an SQL Injection attack has unique characters within it,” says Shapira. “So a database record containing free text such as a <Name> or <Address > should not contain these characters.”
Spotted: Data Is Leaving the Organization
Oftentimes, people are so concerned about what is coming into the network that they don't think about what is leaving the network. Once an organization is compromised, if stealing data is the goal of the cyber attacker, they will try to exfiltrate the data in some way.
“Using native tools within the existing infrastructure (e.g. Netflow in routers/switches/firewalls) is very helpful in spotting trends and anomalies of the data leaving the network,” says Desko.
Liwer adds, “Say an attacker does get into the corporate Dropbox account and is now trying to exfiltrate files. Many companies will have a tool that identifies the anomalous user behavior and block such behavior in real time. So, for example, a user that normally logs in from NY and has an average data throughput of 1GB per month—if they suddenly log in from Russia and are trying to download 2TB of data, that should ring alarms. And that user should be automatically blocked with no questions asked.”
Spotted: Your Data Has Left the Building (and Was Found Online)
Last, but not least (and hopefully not the way you spot that you have been breached!), is finding your company’s information online.
“Worst case scenario is that you or an independent investigator finds your confidential data online.” says Kolga. “It is sometimes easier to spot the breach by monitoring chatter or the sale of an organization’s accounts or data on the Dark Web. A variety of security companies offer this type of monitoring service.”
This may sound easy but may not be the case.
According to San Soucie, “Monitoring the visible and hidden web requires special skills that many companies do not have in-house or cannot afford.”
San Soucie continues by providing some tips when looking for these signs.
“There are some free and low-cost tools that companies can use to self-manage. The key to success is determining what all the keywords and key phrases are that you should monitor for your company. Typically, the company name, company abbreviation, company nicknames are the minimums. This can be challenging if the company name contains generic words such as ‘Motor Works Inc.’ as an example. Google Alerts is a good starting point to find mentions of your company. This will give you an idea if anything is being mentioned and can be helpful to start tweaking the phrases you need to look for.”
If you truly care about your data or are in a business where there is tremendous liability if data is exposed, you almost have to use a professional monitoring company, suggests San Soucie.
“While there are common marketplaces and forums on the hidden (dark) web, they change frequently, and monitoring can be time-consuming. There is also a significant learning curve to review all of the data.”
It’s one thing to know where to look. It’s another to know what to look for.
“Some of the, hopefully obvious, items to look for are anyone asking for confidential information about your company or selling what appears to be compromised data,” says San Soucie. “We find that cybercriminals will compromise a system, not be able to figure out how to use the system and then sell access to that system to make a quick dollar.”
Cybercriminals who have a bit of patience, however, can make things a little more difficult to spot.
Add San Soucie, “The not-so-obvious items have to do with company reputation or anger towards your company. Many of the forums are used to connect these users with cybercriminals that are willing to carry out an attack for a much larger reward or payout.”
If we find good response to this Part 1 of this topic, I may work on a Part 2 where we, the experts and I, will look at the potential impact of a breach and what some of the next steps might be once you spot it.
For now, we hope you enjoyed and will benefit from reading the “Spotted” tips that our experts provided. As we mentioned at the beginning, checking the signs for potential breaches is critical. But remember, even if you don’t spot these signs, it doesn’t mean that you haven’t been breached. We recommend that you rely on multiple security monitoring tools—and maybe even some partners—to keep your infrastructure safe.