SEO is one of the most powerful weapons of online stores. Magento, undoubtedly, provides its users with one of the most solid SEO out-of-box features. Moreover, you don’t need to be an expert to configure all the Magento SEO functionality.
Today I will share with you some tips on how to use Magento SEO built-in functionality on all cylinders. The first part of our guide is dedicated to Sitemap and Robots.txt configuration. Let’s deal with it!
Magento XML Sitemap Configuration
You should update your sitemap as frequent as your site’s content changes. It may happen daily, weekly, monthly, etc. It depends on how often you make changes to your store. Magento sitemap configuration allows you to set update frequency and content priority.
Step 1: Set the Frequency and Priority of Content Updates
— Admin Sidebar → Stores → Settings → Configuration → Catalog → XML Sitemap
— Expand the Category Options and do the following:
a) Set the Frequency: Always, Hourly, Daily, Weekly*, Monthly, Yearly, Never
* Usually, merchants don’t change the set of Categories often, so it’d better set “Weekly” frequency. But it depends on the website activity.
b) Fill in the Priority field with a value between 0.0 and 1.0. (**we recommend 0.9)
** Configure category pages after Homepage
— Expand the Product Options, complete Frequency & Priority settings as you need.
Configure Product pages after Home and Category pages.
Frequency – Daily. Usually, eCommerce websites update their products every day. However, if you rarely do this you can set “Weekly” frequency.
Priority – 0.7
— Include images to a sitemap by setting Add Images into Sitemap to one of the following: None, Base Only*, All**
*Set the value to “Base” only if you have a really huge store with lots of images due to sitemap size requirements (see above). The base image is the main product image (a screenshot is below).
**Set the value to “All” if your online store is small or mid-size.
This setting is important for merchants who want their images to appear in Images Search Results (=almost for everyone). If you sell some complex products such as equipment, construction parts and others, it’s insignificant. Nevertheless, if you sell apparel, shoes or jewelry, this setting is vital for you.
— Expand the CMS Pages Options, complete Frequency & Priority settings as you need.
Usually, cms pages don’t need frequent updates. “Monthly” frequency and “0.5 Priority” are ok.
— Don’t forget to save config
Step 2: Complete the Generation Settings
— Expand the Generation Settings and set Enabled = Yes, to generate a sitemap. Do the following:
a) Set the Start Time.
b) Set the Frequency: Daily (select this), Weekly, Monthly.
c) Fill in the Error Email Recipient field. If something will occur during an update, this person will receive a notification.
d) Set Error Email Sender. A recipient of the notification will see the sender of the notification.
e) Set Error Email Template for the error notification.
Step 3: Set the Sitemap File Limits
— Expand the Sitemap File Limits and do the following:
a) Enter the max number of URLs that can be included in the sitemap, completing the Maximum No of URLs per File field (default limit is 50.000 which is ok).
b) Enter the largest size allocated for the sitemap, completing the Maximum File Size field (default limit is 10,485,760 bytes).
Step 4: Adding Sitemap to Robots.txt
— Expand Search Engine Submission Settings. If you use a robots.txt file, set Enable Submission to Robots.txt to Yes.
— Save Config
What is a sitemap and why do you need It?
A sitemap is a vital SEO part of every website. It helps search engines to understand and index your content making it easier to find on the Net. A sitemap or XML (extensible markup language) sitemap is a list of all the URLs on your website that contains some information about them.
Why is it so important? The answer is search engine bots like Googlebot. It’s a “crawler” program that moves from page to page, recording data about the links it visits. The obtained information helps Google to create search results and define content relevancy.
If you don’t have a sitemap, Googlebot may not see some pieces of content you offer to users. This content won’t appear in Google search results which leads to traffic and conversion losses.
A sitemap guarantee that this crawler bot will see and explore everything placed on your website as it sees how you organized your pages. Moreover, if you add your sitemap to Search Console, it helps to reveal some internal errors like 404, 500, etc.
Do you need HTML sitemap for Magento?
Short answer: No
Long answer: Google always focuses on the usability and usefulness for a user. The recommendation to create html sitemap for the website comes from old times when webpages were not even close to what they are now. The purpose of the html sitemap was to give a user the option to navigate between all the pages located on the domain.
But what about now? Usability requirements have changed a lot since early 2000s. HTML sitemap doesn’t seem to be a usable method of navigation. Website header menu structure, layered navigation (filtering options), categories and subcategories structure – these are things that you have to be focused on now from a usability point of view.
If to speak about crawlers, sitemap xml file is enough for them.
Robots.txt is a file that contains instructions for search engine bots. When you create it, you forbid website crawlers to search some areas of your online store. You can exclude entire domain, directories & subdirectories, internal files from web crawling, etc. You can block crawling unimportant media files.
If you want to exclude specific pages from the Google index, use:
<meta name=”robots” content=”noindex”> or Noindex directive in an HTTP header.
Using robots.txt avoids the next problems:
- Crawling of numerous and unimportant folders and files
- Server overloads due to the high number of crawlers
According to Robots Exclusion Standard Protocol, every search engine robot should read robots.txt file before indexing web pages. The most important search engines like Google, Yahoo and Bing strictly follow REP and robots.txt instructions.
Robots.txt in Magento
Magento generates robots.txt file to avoid indexing internal files that are used by the system. You can use default settings or write your own custom instructions for all or specific search engine bots.
Default Robots.txt for Magento 2:
- Disallow: /lib/
- Disallow: /*.php$
- Disallow: /pkginfo/
- Disallow: /report/
- Disallow: /var/
- Disallow: /catalog/
- Disallow: /customer/
- Disallow: /sendfriend/
- Disallow: /review/
- Disallow: /*SID=
Configuring Robots.txt in Magento 2.3
— Content → Configuration (under Design) → find Global → Edit → choose section Search Engine Robots
Enetr needed instructions in the section “Edit custom instructions of robots.txt file”
Configuring Meta Name=”robots” Magento 2.3
— Admin Sidebar → Content→ Design→ Configurations → Global→ Edit
— Expand Search Engine Robots and do the following:
a) Set Default Robots to one of the following
- INDEX, FOLLOW (allow website indexing)
- NOINDEX, FOLLOW (do not index, but follow the links in HTML)
- INDEX, NOFOLLOW (index website, do not follow the links in HTML)
- NOINDEX, NOFOLLOW (do not index website, do not follow the links in HTML)
b) Use the Edit Custom instruction of robots.txt file field to enter custom instructions if it’s needed
c) If you want to restore default instructions, tap Reset to Default
— Save Config
These tips should help you to configure your Sitemap and Robots.txt in the right way. Keep for updates and don’t miss the next part of our Magento SEO Guide.
We provide both basic and advanced SEO audits. Basic SEO audit is free and includes URLs, 301 redirects, robot.txt and sitemap.txt, meta tags, alt attributes and page load speed. If you’re searching for a more advanced SEO audit, contact our experts to get the details.
We reply within 24 hours
If you have an urgent issue and need to get response faster you may contact us:
+37 525 944 33 88 (Viber, Telegram, WhatsApp)
SEO Expert at Mobecls
5+ years experience
Feel free to ask me anything about SEO