This article and linked github provide Magento 2 cli commands to dedup (remove duplicate) catalog product & category URL keys in Magento 2.1.x. This solution changes duplicate
url_key fields to ensure each is unique.
Magento 2 is a very powerful ecommerce platform that gives the merchant total control over their store; the double edge to this sword is that it’s much easier to break things! If you have imported your catalog (products or categories) from a CSV, or have created a number of them using the Duplicate feature, you may find yourself with duplicate url_key’s. A duplicate URL key is:
- Product entity with the same url_key as another product in the same store.
- Category entity with the same url_key as another category in the same store AND with the same category path (same parent category)
It makes sense if you think about it: the url_key is a duplicate if the final URL constructed from it would identically match another. As I mentioned above, importing your URLs or using the Duplicate feature can quietly create duplicate urls without the merchant noticing. Unlike in Magento 1 (which would attempt to create a random url suffix) Magento 2 is not as forgiving. You will end up with non-seo friendly URLs like:
Our extension is composed of two cli scripts which attempt to do the following:
- Search for any duplicate url_keys in categories OR products
- If any duplicates are found, “dedup” (de-duplicate) those url_keys by appending the product or category ID to one or more of the duplicates, ensuring all products & categories have a unique url_key field.
So if we find 3 products like the below:
Our tool will change them to be:
Note that this extension does account for different URL key scopes — meaning it will account for conflicts between a default scope on one product and a store-level scope on another. In this situation, the default product is treated as the “primary” and the store-level products as the duplicates.
***Make a Backup****
Please make a backup of your database prior to running this tool — it will make permanent changes!
You should use the extension found here: https://github.com/olegkoval/magento2-regenerate_url_rewrites by Oleg Koval to regenerate the product & category URLs after deduping them.
Install manually using github:
mkdir -p app/code/Cadence/UrlDedup && git clone https://github.com/cadencelabs/urldedup app/code/Cadence/UrlDedup
php bin/magento setup:upgrade
Note that this extension is licensed under the GPL 3.0 open-source license. Click here to read the license.
** Don’t forget to install the olegkoval/magento2-regenerate_url_rewrites extension as well!
Run Dedup Commands
Once the extension is installed and your database upgraded, you should see 2 commands from the below output:
php bin/magento | grep cadence cadence cadence:urldedup:categories Dedup Category url rewrites across stores cadence:urldedup:products Dedup Product url rewrites across stores
Depending on what entities you need to dedup, run one or both of the above commands.
** WARNING – Make sure you have a database backup! This tool will make permanent changes!
You should see output like the below:
Dedup Product Url rewrites: [Store ID: 0, Store View code: admin] [Dedup url_key for 0 store id: explicit conflicts.] [Dedup url_key for 0 store id: implicit URL conflicts from cascading default value. Default values will be left unchanged, and store-view level entities will have their URLs changed.] [Refusing to process cascading duplicates for root store]
Dedup Category Url rewrites: [Store ID: 0, Store View code: admin] [Dedup url_key for 0 store id: explicit conflicts.] [Dedup url_key for 0 store id: implicit URL conflicts from cascading default value. Default values will be left unchanged, and store-view level entities will have their URLs changed.] [Refusing to process cascading duplicates for root store]
That’s it! The UrlDedup script has run. You’ll notice that it checks for both explicit conflicts (conflicts at the same scope level) and conflicts caused by the default scope of one entity conflicting with a store-view scope of another. As I mentioned above, categories also take into account the category path.
You’ll also notice both scripts will reindex and clear your installation’s cache.
Run Regenerate URL Command
Once you’ve deduped, you need to then regenerate your catalog. Regenerate it using the olegkoval/magento-2_regenerate_url_rewrites command:
php bin/magento ok:urlrewrites:regenerate
At this point you should be all set! You should see new URLs on the frontend matching the [url_key]-[entity_id] format.
Need help with Magento 2 Catalog URLs?
We’ve had a lot of experience fixing problems with Magento 2 catalogs. If you need help, head over to the Cadence Labs contact page, or email us at firstname.lastname@example.org. We offer affordable rates for our Magento development services.