Excluding member-specific pages from sitemap Answered
Any ideas on how to exclude member-specific pages from sitemap? Our customers have custom dashboards and we use member-specific page generated by Memberstack IDs. It seems like they can be easily breached by malware. I tried using robots.txt but it didn’t work.
You can choose to create your own Sitemap on Webflow as shown in their article here (https://university.webflow.com/lesson/sitemap) or if you aren't using Webflow, some websites like https://www.xml-sitemaps.com/ could be used to generate a Sitemap for your website, then modify to your need.
Alternately just exclude that page from search indexing by using the <meta name="robots" content="noindex" /> tag in the head of that document.
I hope this helps.
The challenge in creating my own Sitemap is that I have many users who are creating their own pages during sign-up, which means the Sitemap needs to be updated automatically. I used the “noindex” meta tag, which may prevent the pages from being indexed by search engines, but it does not affect the Sitemap.
Well, probably writing a cron job which runs on daily basis on your server that would generate a new sitemap according to all newly generated content, return the sitemap in the form of an xml file, and would then overwrite the previous one could work in this case. I don't think changing sitemap for each newly added dynamic content is a good approach though.
However, I'll pass this to the dev team to see what they'll say about it and get back to you as soon as I get feedback.
We heard back from the dev team. They think this is likely a Webflow issue and recommend contacting their support.
I hope that helps!
Will do, thank you both for your help
Please sign in to leave a comment.