How to Create Sitemaps for Millions of URL’s

Creating sitemaps for millions of URL’s can be a daunting task. There are a few ways to do this.

  1. You can run a crawler tool on your site to pull millions of URL’s which will take loads of time.
  2. Create a script that will generate the sitemaps for you provided you have logical page names and parameters.

A guide to create your sitemap:

  1. Define all the page types and URL parameters.
  2. Read all about sitemaps.org protocols.
  3. Use multiple sitemaps and group them into relevant categories such as media type, country or product type.
  4. Make sure the URL’s are non sequential. Google is not a fan of ABC and 123 order so mix it up.
  5. Find a way to prioritize your URL’s. What pages are most important?
  6. Use sitemaps index files.
  7. Stay within 50K max urls in a sitemap or sitemap index file.
  8. Compress the files with gZip.
  9. Load them to your site and submit them to the search engines.

A sitemap for millions of URL’s can be over 10MB+ in size. Compressing the files and finding a way to create dynamic Sitemaps will help you use less space. It can get tricky but remember it CAN be done.

Comments

comments