sitemap

I have already talked about converting a sitemap.xml file into a urllist.txt file, but what if you want to create a HTML sitemap? If you have a sitemap.xml file then you can use this to spider your site, scrape the contents of each page and populate the HTML file with this information.

The following code does this. For every page it looks for the title tag, the description meta tag and the first h2 tag on the page. These items are then used to construct a segment of HTML for that page.

If you create a script that produces a sitemap.xml file there is no point in adapting this script so that it creates a urllist.txt file. The best solution is to use this sitemap.xml file to create the urllist.txt. The following script will do exactly this.