AutoRFP.ai allows you to sync content from websites directly into your content library through the Documentation import.
Understanding Scraping Modes
Before importing website content, you'll need to choose between two scraping modes based on your needs:
Entire Site
Perfect for comprehensive documentation imports, this mode:
Starts from a base URL you provide
Automatically discovers and follows all internal links
Creates individual content items for each discovered page
Ideal for importing complete help documentation or knowledge bases
Use this when: You want to import an entire documentation site or capture all pages within a specific section of a website.
Specific Pages
Designed for targeted content extraction, this mode:
Imports only the exact URLs you specify
Creates one content item per URL provided
Perfect for importing select content like case studies, policies, or specific product pages
Use this when: You need specific pieces of information from a website without importing surrounding pages.
Import Process
Access the Content Importer
Navigate to your content library and open the content importer tool.
Select Documentation Type
Choose "Documentation" as your content type. Website imports are exclusively available under this category.
Choose Website as Source
Select "Website" from the available source options.
Select Your Scraping Mode
Choose either "Entire Site" or "Specific Pages" based on your needs (see descriptions above).
Enter URLs
For Entire Site mode: Enter a single base URL (e.g.,
https://example.com/docs)For Specific Pages mode: Enter a list of complete URLs, one per line
Complete the Import
Click "Continue" and follow the remaining prompts to finalize your import.
After Import
Processing Time
Specific Pages: Usually complete within minutes
Entire Site: May take longer depending on site size and complexity
What to Expect
Each imported page appears as a separate item in your content library
Content is tagged with the source URL for easy identification
Daily syncs run automatically to capture any changes
Troubleshooting
If content doesn't appear in your library after several minutes:
Check that the URLs are publicly accessible
Verify you've selected the correct scraping mode
Contact support if issues persist
Best Practices
For Documentation Sites: Use Entire Site mode with the documentation root URL
For Marketing Content: Use Specific Pages mode to import only relevant pages
Access Requirements: Ensure all URLs are publicly accessible without login requirements
Need additional help? Contact our support team for assistance with your website imports.