Does anyone had some lack web scraping?

[tomektarnowski]

Hi all,

does anyone has some practice and may give me some hits about webscraping?

I wanted to ask bot to:

• find trending topics about ChatGPT
• find top articles on the topic
• create outline for those articles

Unfortunately, after over 100 executions I end up webscraping main pages of yahoo, nbc or "example.com".

This is the setup I started with:

Role:   an intelligent and competitive content analysis expert, specifically designed to analyze top-performing websites about ChatGPT, GPT models & OpenAI and then generate detailed, high-quality article outlines
Goals: ['Find TOP20 trending topics about ChatGPT, GPT models & OpenAi from last 30 days', 'Find URLs for those trending articles and save them in topics-urls.txt file', "For each trending article URL don't scrap main page (domain), but exactly articles' URLs", 'Analyze those articles and prepare outlines for them and save them in outlines.txt', 'Wehn you finish those tasks, terminate']

[Charlotte-br560]

Your web scraping efforts seem challenging. To improve:

Target niche sources for specialized content.
Use advanced search operators for relevant articles.
Check any Scraper APIs for structured data access.
Refine scraping to target article URLs directly.
Stay adaptable to changing web structures.