This article and linked github provide Magento 2 cli commands to dedup (remove duplicate) catalog product & category URL keys in Magento 2.1.x. This solution changes duplicate url_key fields to ensure each is unique.

Click here to view the GitHub repository for this extension

Overview

The Problem

Magento 2 is a very powerful ecommerce platform that gives the merchant total control over their store; the double edge to this sword is that it’s much easier to break things! If you have imported your catalog (products or categories) from a CSV, or have created a number of them using the Duplicate feature, you may find yourself with duplicate url_key’s. A duplicate URL key is:

• Product entity with the same url_key as another product in the same store.
• Category entity with the same url_key as another category in the same store AND with the same category path (same parent category)

It makes sense if you think about it: the url_key is a duplicate if the final URL constructed from it would identically match another. As I mentioned above, importing your URLs or using the Duplicate feature can quietly create duplicate urls without the merchant noticing. Unlike in Magento 1 (which would attempt to create a random url suffix) Magento 2 is not as forgiving. You will end up with non-seo friendly URLs like:

https://DOMAIN.com/catalog/product/view/id/[id]/s/[url_key]/

The Solution

Our extension is composed of two cli scripts which attempt to do the following:

• Search for any duplicate url_keys in categories OR products
• If any duplicates are found, “dedup” (de-duplicate) those url_keys by appending the product or category ID to one or more of the duplicates, ensuring all products & categories have a unique url_key field.

So if we find 3 products like the below:

Our tool will change them to be:

Store Views

Note that this extension does account for different URL key scopes — meaning it will account for conflicts between a default scope on one product and a store-level scope on another. In this situation, the default product is treated as the “primary” and the store-level products as the duplicates.

Prerequisites

***Make a Backup****

Please make a backup of your database prior to running this tool — it will make permanent changes!

olegkoval/magento2-regenerate_url_rewrites

You should use the extension found here: https://github.com/olegkoval/magento2-regenerate_url_rewrites by Oleg Koval to regenerate the product & category URLs after deduping them.

Install Extension

Install manually using github:

https://github.com/cadencelabs/urldedup

mkdir -p app/code/Cadence/UrlDedup && git clone https://github.com/cadencelabs/urldedup app/code/Cadence/UrlDedup

php bin/magento setup:upgrade

Note that this extension is licensed under the GPL 3.0 open-source license. Click here to read the license.

** Don’t forget to install the olegkoval/magento2-regenerate_url_rewrites extension as well!

Run Dedup Commands

Once the extension is installed and your database upgraded, you should see 2 commands from the below output:

php bin/magento | grep cadence

cadence
 cadence:urldedup:categories               Dedup Category url rewrites across stores
 cadence:urldedup:products                 Dedup Product url rewrites across stores

Depending on what entities you need to dedup, run one or both of the above commands.

** WARNING – Make sure you have a database backup! This tool will make permanent changes!

You should see output like the below:

Dedup Product Url rewrites:

[Store ID: 0, Store View code: admin]

[Dedup url_key for 0 store id: explicit conflicts.]
[Dedup url_key for 0 store id: implicit URL conflicts from cascading default value. Default values will be left unchanged, and store-view level entities will have their URLs changed.]
[Refusing to process cascading duplicates for root store]

Dedup Category Url rewrites:

[Store ID: 0, Store View code: admin]

[Dedup url_key for 0 store id: explicit conflicts.]
[Dedup url_key for 0 store id: implicit URL conflicts from cascading default value. Default values will be left unchanged, and store-view level entities will have their URLs changed.]
[Refusing to process cascading duplicates for root store]

That’s it! The UrlDedup script has run. You’ll notice that it checks for both explicit conflicts (conflicts at the same scope level) and conflicts caused by the default scope of one entity conflicting with a store-view scope of another. As I mentioned above, categories also take into account the category path.

You’ll also notice both scripts will reindex and clear your installation’s cache.

Run Regenerate URL Command

Once you’ve deduped, you need to then regenerate your catalog. Regenerate it using the olegkoval/magento-2_regenerate_url_rewrites command:

php bin/magento ok:urlrewrites:regenerate

Conclusion

At this point you should be all set! You should see new URLs on the frontend matching the [url_key]-[entity_id] format.

Alan Barber is the Lead Web Developer at Cadence Labs and a Magento Certified developer.

Need help with Magento 2 Catalog URLs?

We’ve had a lot of experience fixing problems with Magento 2 catalogs. If you need help, head over to the Cadence Labs contact page, or email us at [email protected]. We offer affordable rates for our Magento development services.