SEO is one of the most powerful weapons of online stores. Magento | Adobe Commerce, undoubtedly, provides its users with one of the most solid SEO out-of-box features. Moreover, you don’t need to be an expert to configure all the Magento | Adobe Commerce SEO functionality.
Today I will share with you some tips on how to use Magento | Adobe Commerce built-in SEO functionality on all cylinders. The first part of our guide is dedicated to Sitemap and Robots.txt configuration.
Magento | Adobe Commerce XML Sitemap Configuration
You should update your sitemap as frequently as your site’s content changes. It may happen daily, weekly, monthly, etc. It depends on how often you make changes to your store. Magento | Adobe Commerce sitemap configuration allows you to set update frequency and content priority.
Step 1: Set the Frequency and Priority of Content Updates
— Admin Sidebar → Stores → Settings → Configuration → Catalog → XML Sitemap
— Expand the Category Options and do the following:
a) Set the Frequency: Always, Hourly, Daily, Weekly*, Monthly, Yearly, Never
* Usually, merchants don’t change the set of Categories often, so it’d better set “Weekly” frequency. But it depends on the website activity.
b) Fill in the Priority field with a value between 0.0 and 1.0. (**we recommend 0.9)
** Configure category pages after Homepage
— Expand the Product Options, complete Frequency & Priority settings as you need.
We recommend:
Configure Product pages after Home and Category pages.
Frequency – Daily. Usually, eCommerce websites update their products every day. However, if you rarely do this you can set “Weekly” frequency.
Priority – 0.7
— Include images to a sitemap by setting Add Images into Sitemap to one of the following: None, Base Only*, All**
*Set the value to “Base” only if you have a really huge store with lots of images due to sitemap size requirements (see above). The base image is the main product image (a screenshot is below).
**Set the value to “All” if your online store is small or mid-size.
This setting is important for merchants who want their images to appear in Images Search Results (=almost for everyone). If you sell some complex products such as equipment, construction parts and others, it’s insignificant. Nevertheless, if you sell apparel, shoes or jewelry, this setting is vital for you.
— Expand the CMS Pages Options, complete Frequency & Priority settings as you need.
Usually, cms pages don’t need frequent updates. “Monthly” frequency and “0.5 Priority” are ok.
— Don’t forget to save the config.
Step 2: Complete the Generation Settings
— Expand the Generation Settings and set Enabled = Yes, to generate a sitemap. Do the following:
a) Set the Start Time.
b) Set the Frequency: Daily (select this), Weekly, Monthly.
c) Fill in the Error Email Recipient field. If something will occur during an update, this person will receive a notification.
d) Set Error Email Sender. A recipient of the notification will see the sender of the notification.
e) Set Error Email Template for the error notification.
Step 3: Set the Sitemap File Limits
— Expand the Sitemap File Limits and do the following:
a) Enter the max number of URLs that can be included in the sitemap, completing the Maximum No of URLs per File field (default limit is 50.000 which is ok).
b) Enter the largest size allocated for the sitemap, completing the Maximum File Size field (default limit is 10,485,760 bytes).
Step 4: Adding Sitemap to Robots.txt
— Expand Search Engine Submission Settings. If you use a robots.txt file, set Enable Submission to Robots.txt to Yes.
— Save Config
What Is an XML Sitemap and Why Do You Need It?
A sitemap is a vital SEO part of every website. It helps search engines to understand and index your content making it easier to find on the Net. A sitemap or XML (extensible markup language) sitemap is a list of all the URLs on your website that contains some information about them.
Why is it so important? The answer is search engine bots like Googlebot. It’s a “crawler” program that moves from page to page, recording data about the links it visits. The obtained information helps Google to create search results and define content relevancy.
If you don’t have a sitemap, Googlebot may not see some pieces of content you offer to users. This content won’t appear in Google search results which leads to traffic and conversion losses.
A sitemap guarantee that this crawler bot will see and explore everything placed on your website as it sees how you organized your pages. Moreover, if you add your sitemap to Search Console, it helps to reveal some internal errors like 404, 500, etc.
Robots.txt
Robots.txt is a file that contains instructions for search engine bots. When you create it, you forbid website crawlers to search some areas of your online store. You can exclude the entire domains, directories & subdirectories, internal files from web crawling, etc. You can block crawling unimportant media files.
If you want to exclude specific pages from the Google index, use:
<meta name=”robots” content=”noindex”> or Noindex directive in an HTTP header.
Using robots.txt avoids the next problems:
- Crawling of numerous and unimportant folders and files
- Server overloads due to the high number of crawlers
According to Robots Exclusion Standard Protocol, every search engine robot should read robots.txt file before indexing web pages. The most important search engines like Google, Yahoo, and Bing strictly follow REP and robots.txt instructions.
Robots.txt in Magento | Adobe Commerce
Magento | Adobe Commerce generates robots.txt files to avoid indexing internal files that are used by the system. You can use default settings or write your own custom instructions for all or specific search engine bots.
Default Robots.txt for Magento | Adobe Commerce:
- Disallow: /lib/
- Disallow: /*.php$
- Disallow: /pkginfo/
- Disallow: /report/
- Disallow: /var/
- Disallow: /catalog/
- Disallow: /customer/
- Disallow: /sendfriend/
- Disallow: /review/
- Disallow: /*SID=
Configuring Robots.txt in Magento | Adobe Commerce 2.3
— Content → Configuration (under Design) → find Global → Edit → choose section Search Engine Robots
Enter needed instructions in the section “Edit custom instructions of robots.txt file”
Configuring Meta Name=”robots” Magento | Adobe Commerce 2.3
— Admin Sidebar → Content→ Design→ Configurations → Global→ Edit
— Expand Search Engine Robots and do the following:
a) Set Default Robots to one of the following
- INDEX, FOLLOW (allow website indexing)
- NOINDEX, FOLLOW (do not index, but follow the links in HTML)
- INDEX, NOFOLLOW (index website, do not follow the links in HTML)
- NOINDEX, NOFOLLOW (do not index website, do not follow the links in HTML)
b) Use the Edit Custom instruction of robots.txt file field to enter custom instructions if it’s needed
c) If you want to restore default instructions, tap Reset to Default
— Save Config
Mobecls team provides basic and advanced SEO audits, SEO and performance optimization services. We are SEO-focused development company that specializes in Magento | Adobe Commerce. It means that we know what and how to fix SEO issues, taking into account your UX, Marketing and Business specifics.
