Register for our next deep dive! Who is Behind Portal Kombat? Exposing the Pravda Disinformation Machine with OSINT on Thursday, June 27, 2024, at 16:00 CET. Grab your spot now! close
05 Jul 2023

Advanced IOCs Collection with OSINT and Threat Intelligence Feeds

Mathieu Gaucheler

Moving from Vulnerability to Vigilance πŸ”—︎

Organizations constantly wage battles against malicious actors who seek to exploit vulnerabilities and infiltrate their digital defenses at any given time. As attacks grow in complexity, it’s imperative to keep your guard up against potential threats.

Fortunately, there are more and more methods for gathering information about attacks and attackers. By utilizing open source information (OSIF) and threat intelligence feeds, you can incorporate cyber threat intelligence into your defense strategies. This allows you to identify specific indicators of compromise (IOCs) within systems during incident response and comprehension of the targets.

In this article, we will cover the following topics:

  1. The use of IOCs in open source intelligence (OSINT) investigations
  2. Reliable sources for tracking IOCs
  3. Tips on using regular expression (regex) to identify patterns indicating a threat actor
  4. The role of threat intelligence feeds in keeping your teams informed about new threats


Understand the Scope and Impact of an Attack with IOCs πŸ”—︎

IOCs are artifacts or clues left behind after a cyberattack that you can observe on your system to determine if your system has been compromised. IOCs come in different forms, such as IP addresses, file hashes, domain names, URLs, and more.

Examples of IOCs

A typical application of Indicators of Compromise (IOCs) during incident response arises following a cyberattack. By parsing through logs and extracting IOCs, you can reconstruct the incident’s sequence of events, solving the puzzle of what happened.

IOCs help track threat actors because some can be linked to previous attacks or campaigns. Suppose a particular attack or campaign has already been attributed to a specific threat actor. In that case, it is possible that the same threat actor might be responsible for the IOCs you’re investigating.

IOCs are Valuable for OSINT Investigations πŸ”—︎

IOCs can be valuable for OSINT investigations as they provide specific clues related to malicious activities. They can lead us to more contextual information which helps us gain a deeper understanding of potential threats.

Incorporating IOCs into OSINT analysis can therefore enrich your data, improve threat detection, contextualize threats, and enable proactive defense.

How IOCs fit into OSINT investigations

If you have observed specific IOCs on your system and initiated the incident response process, you might want to delve deeper into a particular IOC to understand who is targeting you. This is essentially a part of Cyber Threat Intelligence (CTI).

Identify Traces of Threat Actors from Attack Associations and Shareable IOCs πŸ”—︎

When you’re trying to identify the culprits behind cyberattacks, one of the methods you can apply is searching for IOCs that have been connected to previous incidents. However, you also want to keep in mind how closely the IOCs are linked to past incidents as not all connections are equally strong or reliable.

Example:

IP addresses used during the same timeframe are a strong indicator that two attacks were carried out by the same threat actorβ€”unless the IP address belongs to a Tor exit node or to a content delivery network (CDN). In those cases, it could be merely coincidental.

Likewise, if two different instances of malware download a payload from a widely accessible website like Dropbox or Discord, it’s highly unlikely that these attacks are associated.


IOCs are primarily reactive in nature: You first notice something suspicious, and then search for IOCs to understand what has happened to your system. However, they can also be employed proactively to generate cyber threat intelligence.

Example:

As we read through this report by Talos, we can take note of various IOCs to study them later to get actionable intelligence. One simple yet effective step to take is to block the IP addresses mentioned in this article.


Another important point to consider is that threat actors often deploy infrastructure that can be shared across various devices to carry out an attack. The identifiers of this infrastructure later become IOCs. In order to track threat actors, you need to look into these types of IOCs.

Example:

A less effective indicator to track threat actors is a hash, as it is used to identify one file. Any threat actor can easily generate multiple variations of the same malware with slight changes, resulting in entirely different hashes (especially when using hashing algorithms like MD5, SHA256, or SHA1).

On the other hand, indicators like a domain, a domain pattern, or an SSL configuration are much more valuable. They require more effort for threat actors to create, so they are more likely to be reused. By identifying and monitoring these indicators, you have a better chance of discovering new components, such as related malware or infrastructure, that are associated with the threat actors.


Attribute IOCs to Threat Actors through Malware and Infrastructure πŸ”—︎

It’s critical to keep in mind that attributing cyberattacks to specific threat actors is a difficult task and rarely achieves 100% accuracy.

There are various reasons for this, including the reuse of infrastructure or software by different actors, the potential for shared members among threat groups, and the deliberate use of false flags to mislead investigators. Altogether, they render the attribution process a very imprecise and flawed science.

However, looking beyond attribution, the study of IOCs left behind can still provide us with valuable insights into similar malware and the infrastructure employed by threat actors:

  • If we come across malware on our infrastructure, we can examine its communication patterns by analyzing data from well-known sandbox websites like JoeSandbox or VirusTotal.
  • If the malware is communicating with a specific IP address, we can gather information about the behavior of that IP address by consulting providers like GreyNoise, or we can refer back to VirusTotal or OTX AlienVault.
  • If the identified IP address has been used to download other malware, we can further investigate its presence and activity within our infrastructure.

Use Public Sources to Keep Track of IOCs πŸ”—︎

In addition to keeping a close eye on your system for IOCs, you can tap into open-source data sources to enhance your IOCs collection. These sources include public repositories, online forums, and social media platforms, among others.

Public Sources for Keeping Track of IOCs

Here are some of our recommendations:

Another idea is checking specialized websites, such as Talos Intelligence. While these websites are not threat intelligence feeds, you can still obtain the analysis of a particular threat actor or malware from their research.

There are also communities where IOCs are shared, such as MISP or OTX AlienVault. While these communities may not strictly fall under the umbrella of open source intelligence (OSINT), they can still provide valuable IOCs for your collection.

Online forums can also be a potential source of IOCs, although to a lesser extent. It is rare for a threat actor to post a VirusTotal link to their malware, which would include the hash of the malware in the link. This is considered a bad practice among threat actors and is therefore not commonly seen.

Use Regex to Analyze Unstructured Data and Spot Patterns πŸ”—︎

Regular expressions (regex) may require more hands-on work, but it proves to be particularly useful in the context of IOCs and OSINT.

A regex is a string of characters that enables you to define a specific pattern. This allows you to search for and/or replace occurrences of this pattern within a given text. For example, with a simple regex, you can extract a list of IP addresses mentioned in a random log file, regardless of its length.

Regexes have the following advantages:

  • Efficiency: If you were to run a simple regex on the entire Bible, which comprises approximately 31,000 lines (considered relatively small compared to some log files), the results would be instantaneous.
  • Wide adoption: Regex has been around since the 1950s. There is extensive documentation, a variety of development tools, and a supportive community to help answer any questions. Also, many tools and platforms support the use of regex, enabling you to apply your knowledge in various contexts.
  • Easy to learn, hard to master: Basic regex searches, such as looking for any IP address within a given netblock, can be learned in about an hour. You can go much further and do some very complex and specific searches depending on your needs. This, however, requires more time and patience.
  • IOCs validation: Regex searches are like detectives who check whether the right pieces fit together. IOCs have specific formats and patterns, just like fingerprints. You can use regex searches to validate IOCs and see if they match the expected patterns. This can help maintain the accuracy of collected indicators, reduce errors in threat detection, and ensure you don’t miss potential threats or raise false alarms.

Key Considerations for Using Regex πŸ”—︎

Regexes can be quite complex, so it’s wise to test them before running them on large files. Websites like regex101 can help you test your regex with sample text to ensure it works as expected.

When conducting a regex search, it’s best to start with a broader search pattern and then refine it gradually to not miss any important information.

Example:

Before May 3rd, 2023, the string “malware.zip” was most likely considered a filename because, at that time, “.zip” was not a valid top-level domain (TLD). However, the issue arises when “malware.zip” can technically be a correct domain if your regex does not account for valid TLDs. This means that a generic regex for domains would mistakenly identify “malware.zip” as a domain, even though it was a filename, resulting in a false positive.

To avoid this, a skilled analyst would design a regex that considers only valid TLDs when detecting domains. This is because, as of May 3rd, 2023, Google started selling domain names using the “.zip” TLD, which means that any domains using this TLD would be missing from our results if our regex was not updated accordingly.


This example illustrates just one of the many situations where you have to choose between using:

  • A more generic regex, which identifies more results but may include false positives.
  • A more restrictive regex that minimizes the number of false positives. However, in doing so, it may miss some valid results.

It is generally easier to identify something in your results that shouldn’t be there, rather than noticing something that should be there but is missing.

There are two things that could be particularly interesting to look for when searching through log files:

  • Login attempts into databases or other critical tools on the machine (e.g., “mysql -u”)
  • Tools that could potentially be used for file download or upload purposes (e.g., wget or curl).

If you would like to experiment with some regex patterns, take a look at our regex cheat sheet for automatically extracting data from reports:

CVEs

CVE-\d{4}-\d*

Common hashes (MD5, SHA1 and SHA256)

[A-Fa-f0-9]{64}|[a-fA-F0-9]{40}|[a-fA-F0-9]{32}

URL even when defanged (ex: hxxps://google[.]com)

(?:h..ps?|f.p)://(?:www(?:.|[.])|(?!www))[a-zA-Z0-9]+a-zA-Z0-9[^\s]{2,}|www(?:.|[.])[a-zA-Z0-9]+a-zA-Z0-9[^\s]{2,}|https?://(?:www(?:.|[.])|(?!www))[a-zA-Z0-9]+(?:.|[.])[^\s]{2,}|www(?:.|[.])[a-zA-Z0-9]+(?:.|[.])[^\s]{2,}

IPv4 even when defanged (ex: 8[.]8[.]8[.]8)

\b[\d]{1,3}[?.]?[\d]{1,3}[?.]?[\d]{1,3}[?.]?[\d]{1,3}(:\d+)?\b

IPv6

[0-9a-fA-F]{1,4}(?::[0-9a-fA-F]{1,4}){7}|(?:(?:[0-9A-Fa-f]{1,4}(?::[0-9A-Fa-f]{1,4}){0,5})?)::(?:(?:[0-9A-Fa-f]{1,4}(?::[0-9A-Fa-f]{1,4}){0,5})?)

BTC address

(?:[13]|bc1)[a-zA-HJ-NP-Z0-9]{26,62}


Threat Intelligence Feeds Put IOCs into Context πŸ”—︎

Regex searches have a limitation in that they do not consider the surrounding context and focus solely on predefined patterns and structures. However, this limitation does not apply to threat intelligence feeds.

Intelligence does not equal to a collection of unorganized IOCs. These IOCs should have associated context and answer questions like:

  • When were they observed?
  • Who is the threat actor they are linked to?
  • What malware are they associated with?

This doesn’t mean that all the information is always available, but threat intelligence feeds provide as much contextual information about IOCs as possibly available.

We can think of it as a large encyclopedia that analysts can turn to when they’re trying to make sense of unusual discoveries in their systems.

Therefore, threat intelligence feeds are invaluable for threat analysts.

Here are a few examples of threat intelligence feeds with diverse sets of data sources commonly used in the industry:

One thing worth mentioning is that although they are useful for detecting and responding to security incidents, threat intelligence feeds are a tool that makes the analyst’s work easier, but they do not aim to serve as an automated solution such as Endpoint Detection and Response (EDR) platforms.

In any case, the data provided by threat intelligence feeds are structured and designed to be easily queried, meaning that the information obtained is already organized and tailored for ingestion.

Advancing Your Security Measures to a Proactive Level πŸ”—︎

To sum up, there are reliable methods for collecting and comprehending IOCs, such as:

  1. Open source intelligence (OSINT)
  2. Threat intelligence feeds
  3. Regular expressions (regex)

When used consistently and properly, these resources will help you stay ahead of adversaries and maintain a proactive security stance.

For a comprehensive guide on collecting threat intelligence and utilizing Maltego to streamline the lifecycle of threat intelligence, along with other common use cases, check out our handbook for Cyber Threat Intelligence below.

Download the resource

DE +49
Albania +355
Algeria +213
Andorra +376
Angola +244
Anguilla +1264
Antigua And Barbuda +1268
Argentina +54
Armenia +374
Aruba +297
Australia +61
Austria +43
Azerbaijan +994
Bahamas +1242
Bahrain +973
Bangladesh +880
Barbados +1246
Belarus +375
Belgium +32
Belize +501
Benin +229
Bermuda +1441
Bhutan +975
Bolivia +591
Bosnia and Herzegovina +387
Botswana +267
Brazil +55
Brunei Darussalam +673
Bulgaria +359
Burkina Faso +226
Burundi +257
Cambodia +855
Cameroon +237
Canada +1
Cape Verde +238
Cayman Islands +1345
Central African Republic +236
Chile +56
China +86
Cote d'Ivoire +225
Colombia +57
Comoros +269
Congo +242
Cook Islands +682
Costa Rica +506
Croatia +385
Cuba +53
Cyprus +90392
Czech Republic +42
Denmark +45
Djibouti +253
Dominica +1809
Dominican Republic +1809
Ecuador +593
Egypt +20
El Salvador +503
Equatorial Guinea +240
Eritrea +291
Estonia +372
Ethiopia +251
Falkland Islands (Malvinas) +500
Faroe Islands +298
Fiji +679
Finland +358
France +33
French Guiana +594
French Polynesia +689
Gabon +241
Gambia +220
Georgia +7880
Germany +49
Ghana +233
Gibraltar +350
Greece +30
Greenland +299
Grenada +1473
Guadeloupe +590
Guam +671
Guatemala +502
Guinea +224
Guinea-Bissau +245
Guyana +592
Haiti +509
Honduras +504
Hong Kong +852
Hungary +36
Iceland +354
India +91
Indonesia +62
Iran, Islamic Republic of +98
Iraq +964
Ireland +353
Israel +972
Italy +39
Jamaica +1876
Japan +81
Jordan +962
Kazakhstan +7
Kenya +254
Kiribati +686
Korea, Democratic People's Republic of +850
Korea, Republic of +82
Kuwait +965
Kyrgyzstan +996
Lao People's Democratic Republic +856
Latvia +371
Lebanon +961
Lesotho +266
Liberia +231
Libyan Arab Jamahiriya +218
Liechtenstein +417
Lithuania +370
Luxembourg +352
Macao +853
Macedonia, the former Yugoslav Republic of +389
Madagascar +261
Malawi +265
Malaysia +60
Maldives +960
Mali +223
Malta +356
Marshall Islands +692
Martinique +596
Mauritania +222
Mauritius +230
Mayotte +269
Mexico +52
Micronesia, Federated States of +691
Moldova, Republic of +373
Monaco +377
Mongolia +976
Montserrat +1664
Morocco +212
Mozambique +258
Myanmar +95
Namibia +264
Nauru +674
Nepal +977
Netherlands +31
New Caledonia +687
New Zealand +64
Nicaragua +505
Niger +227
Nigeria +234
Niue +683
Norfolk Island +672
Northern Mariana Islands +670
Norway +47
Oman +968
Pakistan +92
Palau +680
Panama +507
Papua New Guinea +675
Paraguay +595
Peru +51
Philippines +63
Poland +48
Portugal +351
Puerto Rico +1787
Qatar +974
Reunion +262
Romania +40
Russian Federation +7
Rwanda +250
San Marino +378
Sao Tome and Principe +239
Saudi Arabia +966
Senegal +221
Serbia +381
Seychelles +248
Sierra Leone +232
Singapore +65
Slovakia +421
Slovenia +386
Solomon Islands +677
Somalia +252
South Africa +27
Spain +34
Sri Lanka +94
Saint Helena +290
Saint Kitts and Nevis +1869
Saint Lucia +1758
Sudan +249
Suriname +597
Swaziland +268
Sweden +46
Switzerland +41
Syrian Arab Republic +963
Taiwan +886
Tajikistan +7
Thailand +66
Togo +228
Tonga +676
Trinidad and Tobago +1868
Tunisia +216
Turkey +90
Turkmenistan +993
Turks and Caicos Islands +1649
Tuvalu +688
Uganda +256
United Kingdom +44
Ukraine +380
United Arab Emirates +971
Uruguay +598
United States +1
Uzbekistan +7
Vanuatu +678
Holy See (Vatican City State) +379
Venezuela +58
Viet Nam +84
Virgin Islands, British +84
Virgin Islands, U.S. +84
Wallis and Futuna +681
Yemen +967
Zambia +260
Zimbabwe +263

By clicking on "Access", you agree to the processing of the data you entered and you allow us to contact you for the purpose selected in the form. For further information, see our Data Privacy Policy.

Stay connected with us on Twitter and LinkedIn, and don’t forget to subscribe to our newsletter for regular updates.

Happy investigating!

About the Author πŸ”—︎

Mathieu Gaucheler

Mathieu Gaucheler Mathieu Gaucheler is a subject matter expert at Maltego. His responsibilities include research-driven content development for blog posts, webinars, and talks. He started working in cybersecurity in Barcelona, focusing on malware analysis and sandbox development. He has previously presented his research at BotConf and RSA APJ.

By clicking on "Subscribe", you agree to the processing of the data you entered and you allow us to contact you for the purpose selected in the form. For further information, see our Data Privacy Policy.