Reconnaissance, Scanning and Enumeration · PTES: Intelligence gathering
L12. Service and Web Content Enumeration
Enumerate services and web content within scope, using robots.txt, directory discovery and careful interpretation of results.
Lesson 12 of 21 · About 30 minutes including the lab
By the end of this lesson you can
- Use robots.txt and common paths as enumeration clues.
- Run a small, scoped directory discovery against a local lab.
- Distinguish clues from confirmed issues.
- Stop when results fall outside the agreed scope.
Video transcript
A web port is open. What is actually behind it? Enumeration digs into each service to list its pages, versions, users and shares. For a web app, it often means finding paths that are not linked. Start with what the server publishes, like robots.txt and the page source. A tool such as Gobuster requests paths from a small wordlist and records each status code. Every path that answers becomes a lead to check by hand, inside the scope. Open it and read what it reveals, or follow the redirect. Missing paths are filtered out, so you can focus. Agree the request rate and wordlist size first. Start with a small wordlist, so you understand the noise. Check every hit by hand before you report it. Good enumeration decides what you test next. Practice in the knowledge check.
Course outlineLesson 12 of 21
Lesson notes
Enumeration means turning “something is open” into “here is what it is and what it exposes”. For web applications that usually means reading robots.txt and sitemap files, walking the site through a proxy, reviewing JavaScript for API paths, and carefully probing for directories and files.
robots.txt is a polite list of paths a site asks crawlers not to index. It is not access control. A Disallow: /admin/ line is a clue that /admin/ might exist; it is not authorization to attack it on a real engagement, and it is not proof that it is sensitive. Always fetch it with a normal GET first.
Directory discovery tools such as Gobuster or ffuf send many requests for candidate paths. On a client engagement, agree the rate and the wordlist size first, and exclude destructive methods. Start with a tiny wordlist so you understand the noise. Treat every new path as a lead to investigate, not as a finding by itself.
When a path returns 200, open it and see what it is. When it returns 301 or 302, note the Location. When it returns 403, the path exists but is denied; that is still useful. When it returns 404, move on. Keep your scope in mind: if a discovered path points at a different host, stop and ask.
Commands and expected output
Read robots.txt
curl -s http://127.0.0.1:8000/robots.txtOutput
User-agent: *
Disallow: /admin/Real output from a local test server. The Disallow line is a clue. Next you would check whether /admin/ responds, still only inside your lab.
Small directory discovery against a local lab
printf 'admin\nbackup\nimages\nlogin\nrobots.txt\nuploads\n' > words-web.txt
gobuster dir -u http://127.0.0.1:8000 -w words-web.txt -qOutput
/backup (Status: 301) [Size: 0] [--> /backup/]
/robots.txt (Status: 200) [Size: 32]
/admin (Status: 301) [Size: 0] [--> /admin/]Real Gobuster 3.6 output against a local test server that had /admin and /backup directories. A six-word wordlist is enough to learn the tool. On Juice Shop or DVWA, point -u at http://127.0.0.1:3000 or http://127.0.0.1:4280 and use a wordlist the RoE allows.
Hands-on labs
Practice only on these legal targets: your own local lab or machines a platform assigns to you. Never use these techniques on systems you do not have written permission to test.
OWASP Juice Shop
Map the APIWith Burp or ZAP recording, browse Juice Shop and list every /rest/ and /api/ path you see. Note which ones require authentication.
DVWA
Enumerate DVWA pagesAfter logging in, list the vulnerability modules from the left menu. That menu is intentional enumeration; compare it with what Gobuster finds.
TryHackMe
Content discovery practiceSearch TryHackMe for a content discovery room on your plan; otherwise repeat the Gobuster exercise against Juice Shop with a larger allowed wordlist.
Practice task
Against Juice Shop on localhost, fetch /robots.txt and run Gobuster with a wordlist of at most 50 entries. Record three interesting paths and what each HTTP status told you.
Key takeaways
- robots.txt is a clue, not an access-control mechanism.
- Directory discovery is noisy: agree rate and wordlist before client work.
- 200, 301, 403 and 404 each mean something different; investigate before reporting.
- New hosts discovered via redirects stay out of scope until approved.
Framework mapping
Sources and further reading
1. What does Disallow: /admin/ in robots.txt mean?
2. Gobuster reports /backup with status 301. What should you do next?
3. Why start directory discovery with a tiny wordlist?
4. A discovered path redirects to a hostname that is not in scope. What do you do?