How to Generate Large Sitemap Automatically?

How to Generate Large Sitemap Automatically?

Large sitemap.xml files should always be generated automatically. Typically it’s a CMS or ecommerce system on the backend. Typically there are plugins or modules available that will do it or you can write your own. We would HIGHLY recommend utilizing the <priority> element in the sitemap.xml to prioritize your pages so the engines know which pages are most important to be indexed, second most important to be indexed, third most important to be indexed, etc.

We agree that typically sitemap.xml files are not needed by most sites, they are indeed very helpful for most large sites. Ecommerce sites, for instance, may have hundreds of thousands of products. Those products might be linked to primarily from category pages DEEP in the site (could be 5,6,7, etc.) clicks from the home page.

Each site has both a crawl budget (amount of time and resources that the engine is going to dedicate to find pages on your site). The engines are typically also going to index only a certain number of pages on your site. Both of these are based primarily on the number and quality of backlinks the site has (i.e. PageRank).

So anything you can do to help the engines like Google crawl your site more efficiently (spend your crawl budget crawling pages you think are most important like product pages on ecommerce sites) and to influence which pages get indexed before others (using <priority> in a sitemap.xml) is advantageous for large sites EVEN if the site has great navigation.

Google has lots of limitations on sitemaps. 50,000 URLs per sitemap file. A sitemap index file can contain at most 50,000 sitemap.xml files. We believe the size of the sitemap.xml cannot exceed more than 50MB uncompressed and possible 10MB compressed.

How we can create/generate xml-sitemap file for the same.
Within Categories Options, set Frequency to Daily and Priority to 1.

Within Products Options, set Frequency to Daily and Priority to 0.8 (or anything less than 1 and more than we are about to set the CMS pages to).

Within CMS Pages Options, set Frequency to Weekly and Priority to 0.25.

Within Generation Settings, set Enabled to Yes, Start Time to 01 00 00 (01:00 a.m. or you can use another time that your traffic is at it’s lowest), and Frequency to Daily, and enter your e-mail address into the Error Email Recipient field.

What role does sitemap plays in improving SEO rankings

What role does sitemap plays in improving SEO rankings

The sitemap not only helps for indexation and site discovery it also helps to de-index obsolete pages more quickly. In case a page no longer exists on the site (returning a 404) and in parallel also no longer is mentioned in the sitemap. Google will de-index it more quickly. I.e. I will re-crawl the page less often before it is convinced it is actually gone.

So it appears Google keeps “a state” of the sitemap and looks at differences.

That 2nd use of the sitemap makes it a powerful tool in combating index size issues and ensuring a domain’s “crawl capacity” is used on pages that the webmaster wants to get crawled.

HTML site map is for visitors and help them in navigating through the website (which Google likes). So both of them have importance in their own ways and helps your website in SERP ranking. Make sure your site maps include all pages and their sub pages. Make the site map like a index page of a book. It needs to be clear and logical.



Subscribe For Free To Get Free Stuff

Subscribe For Free To Get Free Stuff

Join our mailing list to receive the latest updated completely free.

You have Successfully Subscribed!

Pin It on Pinterest