Adversaries may search websites owned by the victim for information to use in targeting. MITRE lists this under Reconnaissance with platform PRE. Such sites can hold names of departments and divisions, physical locations, and details about key employees such as names, roles, and contact information, as well as business operations and relationships.
No special access is needed. The information is on the front door.
How attackers use it
MITRE says adversaries may search victim websites to collect actionable information. The results can lead to further reconnaissance, such as phishing for information and searching open technical databases. They can also support creating or compromising accounts and initial access through a trusted relationship or phishing.
MITRE adds that adversaries may do more than browse by hand. They may try to find hidden directories or files that hold sensitive information or vulnerable functionality, through automated activity such as wordlist scanning, and by using files such as sitemap.xml and robots.txt.
A robots.txt file asks well behaved crawlers to stay away from certain paths. Anyone can read it, so it also works as a list of places you would rather not draw attention to.
It helps to see this from the attacker's side. A company website is written for customers, investors, and candidates. It is not written with an intruder in mind, so it tends to be generous: a leadership page with titles, a contact page with direct lines, a careers page that lists the tools the team uses, a news page that names partners.
Each of those is harmless alone. Together they give a name, a role, an email format, a vendor, and a reason to be believed. MITRE's list of what comes next shows how that gets used: phishing for information, searching technical databases, building accounts, and then phishing or a trusted relationship for initial access.
The second half of the technique, the hunt for hidden directories and files, is a different problem. Here the risk is not what you chose to publish but what you forgot. Old backups, test pages, and exposed admin paths are the usual finds. Wordlist scanning is cheap and fast, so assume that anything reachable by URL will be tried.
How to detect it
This is one of the techniques where MITRE gives defenders a concrete signal. It suggests monitoring for suspicious network traffic that could indicate adversary reconnaissance, such as rapid successions of requests indicative of web crawling, and large quantities of requests originating from a single source, especially a source associated with an adversary.
It also suggests analyzing web metadata, including referer and user-agent fields, for artifacts attributable to malicious activity.
In practice, web server access logs and a web application firewall are your sources. Useful patterns are many requests for paths that do not exist, which suggests a wordlist, a burst of requests across the whole site from one address, and unusual user agent strings.
Counting by client address is the simplest start. The result is not proof of anything, since search engines also crawl, but an unknown address with thousands of requests and mostly 404 responses deserves a look.
Web logs also give you a chance to learn what normal looks like. Search engine crawlers, uptime monitors, and your own scanners all make bursts of requests. Build an allow list of the sources you know, so the unknown ones stand out.
If you run a web application firewall, check whether it has rules or rate limits for rapid enumeration, and look at what it reports on a normal week. A rule that fires every day and is ignored is worse than none. Tune it until a firing means something.
One more habit: when you find a scan, look at what it asked for. Requests for common admin paths, backup file names, and version control directories tell you what the visitor hoped to find. If any of those paths exist on your server, treat that as the urgent finding, not the scan.
How to reduce the risk
- MITRE says this technique cannot be easily mitigated with preventive controls, because it is based on behaviors performed outside the scope of enterprise defenses and controls. Its guidance is to minimize the amount and sensitivity of data available to external parties.
- Review the site as an outsider. Remove staff detail, such as direct contact data and org charts, that has no business purpose. Use role based addresses.
- Check every path in robots.txt and sitemap.xml. If a path should not be public, protect it with authentication instead of relying on a request to crawlers.
- Remove old files, backups, and test pages from the web root.
- Rate limit and alert on high volume crawling from single sources, using your web application firewall or server configuration.
- Keep the web application patched, since hidden functionality is only a problem when it is vulnerable.
A useful way to run this is a short quarterly review. One person reads the public site as an outsider, one person reads a week of logs, and the two compare notes. The first finds what you publish; the second finds who is looking. Keep a log of changes so the same detail is not put back by the next redesign.
None of this makes the site less useful to real visitors. The goal is the same one MITRE states: reduce the amount and sensitivity of data available to outside parties, and spend the rest of your effort on what happens after someone uses it.
Check your own environment
Read-only checks for a site you own. The first reads the files MITRE mentions, using curl on Linux or macOS.
curl -s https://yourdomain.com/robots.txt
curl -s https://yourdomain.com/sitemap.xml | head -50
The second ranks client addresses in an nginx access log, using the default log path on Linux. Adjust it for your setup.
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
The third counts 404 responses by address, which can show wordlist style scanning.
awk '$9 == 404 {print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
Then ask:
- Does any path in robots.txt point at something sensitive?
- What can a visitor learn about our staff and locations?
- Do we alert on a single address making thousands of requests?
- Are old backups and test pages gone from the web root?
Quick check
Which files does MITRE say adversaries may use to find hidden content?
sitemap.xml and robots.txt, along with automated wordlist scanning.
What traffic pattern does MITRE suggest watching?
Rapid successions of requests indicative of web crawling, and large quantities of requests from a single source.
Is listing a path in robots.txt a protection?
No. It is a request to crawlers that anyone can read, so sensitive paths need real access control.
Sources
- MITRE ATT&CK, Search Victim-Owned Websites (T1594): https://attack.mitre.org/techniques/T1594
