Search Engine Reconnaissance (T1593.002): Detect and Reduce

Adversaries may use search engines to collect information about victims. MITRE lists this under Reconnaissance with platform PRE. Search engines crawl sites to index content, and may offer special syntax to search for keywords or content types such as file types.

How attackers use it

MITRE says adversaries may craft queries based on what they want to find. They may collect general information about the victim, or use specialized queries to look for spillages and leaks of sensitive information such as network details or credentials.

Results can lead to phishing for information, searches of open technical databases, creating or compromising accounts, and initial access through valid accounts or phishing.

Anything that is publicly reachable and crawled can be found. A forgotten file on a public server is as visible as the home page.

Search engines are effective because they are thorough. They index files nobody linked to, and they remember pages that have since changed. A leak that lasted an afternoon can live in a cache for much longer.

Specialized query syntax lets an attacker narrow results to a domain, a file type, or specific words. You can use the same syntax on yourself.

How to detect it

MITRE notes that this activity may have a very high occurrence and false positive rate, and may take place outside your visibility, which makes detection difficult. Its advice is to focus on related stages of the adversary lifecycle, such as initial access.

You cannot see a search query. You can see its effect: your own server logs show crawlers fetching documents, so compare what is fetched with what you meant to publish.

Review your robots.txt and server logs together. If a crawler fetches a file you did not expect, find out why it was reachable.

Keep a record of each finding and its removal date, so you can show that exposure is shrinking over time.

How to reduce the risk

• MITRE says this technique cannot be easily mitigated with preventive controls, because it is based on behaviors performed outside the scope of enterprise defenses and controls. Its guidance is to minimize the amount and sensitivity of data available to external parties.
• Search your own domain for file types you would not want public, such as spreadsheets, configuration files, and backups.
• Remove or restrict files that should not be public, and ask search engines to drop cached copies where that is possible.
• Keep sensitive documents behind authentication, so crawlers cannot reach them.
• Review what a robots.txt file lists, since it can point to places you would rather not advertise.

Treat every public file as permanent. If a file should never have been public, rotate any secret inside it first and remove it second.

Check your own environment

A read-only look at what your site tells crawlers, using curl on Linux or macOS against a domain you own:

curl -s https://yourdomain.com/robots.txt
curl -s https://yourdomain.com/sitemap.xml | head -50

Then ask:

• Do any listed paths reveal internal tools or backups?
• Which file types are indexed for our domain?
• Who removes a leaked file, and how fast?

Quick check

Sources

• MITRE ATT&CK, Search Engines (T1593.002): https://attack.mitre.org/techniques/T1593/002