<html><head></head><body><div class="content educate_content"><section class="section section__article">
    <div class="breadcrumb" dir="ltr">
  <div class="link__arrow o__ltr">
    <a href="https://help.apify.com/en/">All Collections</a>
  </div>
      <div class="link__arrow o__ltr">
        <a href="https://help.apify.com/en/collections/1669746-extracting-data">Extracting data</a>
      </div>

  <div class="link__arrow o__ltr">How to scrape a website using sitemap.xml and Apify SDK</div>
</div>

  <div class="paper paper__large">
  <div class="content content__narrow">
    <div class="article intercom-force-break">
      <div class="article__meta" dir="ltr">
          <h1 class="t__h1">How to scrape a website using sitemap.xml and Apify SDK</h1>
          <div class="article__desc">
            Website with proper sitemap.xml = free jackpot for every web scraper
          </div>
          <div class="avatar">
  <div class="avatar__photo o__ltr">
      <img src="https://static.intercomassets.com/avatars/1227910/square_128/picture-3-1580399945.jpeg" alt="Marek Trunkát avatar" class="avatar__image">

  </div>
  <div class="avatar__info" style="vertical-align: top;">
    <div>
      Written by <span class="c__darker"> Marek Trunkát</span>
        </div>
  </div>
</div>

      </div>
      <article dir="ltr"><p class="no-margin"><i>Check out our <a href="https://apify.com/vaclavrut/sitemap-sniffer" class="intercom-content-link">Sitemap Sniffer</a> – it can discover sitemaps in hidden locations.</i></p><p class="no-margin"></p><p class="no-margin">A website's sitemap contains a list of all of its pages in XML format. We can use it to generate a list of pages to crawl. </p><ol><li><p class="no-margin">Locate the sitemap, using either the <code>/sitemap.xml</code> path or with the <a href="https://apify.com/vaclavrut/sitemap-sniffer" class="intercom-content-link">Sitemap Sniffer</a>. </p></li><li><p class="no-margin">Identify the URLs for the pages you want to scrape and <a href="https://docs.apify.com/tutorials/scrape-websites-using-the-sitemap" class="intercom-content-link">create a regular expression</a> to capture them.</p></li><li><p class="no-margin"><a href="https://docs.apify.com/tutorials/scrape-websites-using-the-sitemap" class="intercom-content-link">Import</a> the URLs into the Apify SDK's <a href="https://sdk.apify.com/docs/api/request-list#docsNav" class="intercom-content-link">RequestList</a>.</p></li><li><p class="no-margin">Use the created RequestList in <a href="https://sdk.apify.com/docs/api/puppeteer-crawler#docsNav" class="intercom-content-link">PuppeteerCrawler</a> and save the results to your dataset.</p></li></ol><p class="no-margin"><a href="https://docs.apify.com/tutorials/scrape-websites-using-the-sitemap" class="intercom-content-link">See the full tutorial and code examples</a> in our documentation.</p><p class="no-margin"></p></article>
    </div>
  </div>
      <div class="intercom-reaction-picker" dir="ltr">
    <div class="intercom-reaction-prompt">Did this answer your question?</div>

      <button class="intercom-reaction" data-reaction-text="disappointed" tabindex="0" aria-label="Disappointed Reaction">
        <span data-emoji="disappointed" title="Disappointed">
        </span>
      </button>
      <button class="intercom-reaction" data-reaction-text="neutral_face" tabindex="0" aria-label="Neutral face Reaction">
        <span data-emoji="neutral_face" title="Neutral face">
        </span>
      </button>
      <button class="intercom-reaction" data-reaction-text="smiley" tabindex="0" aria-label="Smiley Reaction">
        <span data-emoji="smiley" title="Smiley">
        </span>
      </button>
  </div>

</div>

</section>
</div>
    </body></html>

Website with proper sitemap.xml = free jackpot for every web scraper

Viewed Article

intercom:trackingEvent

https://www.googletagmanager.com/gtag/js?id=UA-67003981-4


        window.dataLayer = window.dataLayer || [];
        function gtag(){dataLayer.push(arguments);}
        gtag('js', new Date());
        gtag('config', 'UA-67003981-4');



    window.intercomSettings = {"app_id":"kod1r788","article_id":"1865554","anonymous_id":"8c8c96ce-06ec-4101-8fe8-52afd48b8917"};



    window.alexandriaArticleId = "1865554";
    window.alexandriaArticleContentId = "338772";


https://static.intercomassets.com/assets/help_center-d36cfac0eccacbe14d1a7ba68d9f0c90b39aba04a7f1d862292ab8a525bf4c8f.js