Tag: web crawler
-
Scrapy | Crawl WhoScored For Football Stats
Earlier, I have written code to crawl Google Play, iTunes AppStore and Goal.com websites. But every time I re-wrote the code to get content from website, parse it using BeautifulSoup while maintaining the list of crawled URLs to avoid crawling them again. This was a lot of work. A while ago I, discovered Scrapy. It’s…
-
Crawl iTunes AppStore To Get List of All Apps
Git Repo: https://github.com/anuvrat/scrape-google-play/ Thankfully Apple AppStore provides a nice index to look up all the apps. All the apps have been categorized into 23 broad categories. Within each category the apps have been indexed alphabetically. So, to discover all the apps in the iTunes AppStore, one only needs to crawl the main index page, find all the category…
-
Crawl Google Play to Get List of All Apps
Github Repo: https://github.com/anuvrat/scrape-google-play Unlike Apple, Google does not provide a list of all the apps in the Google Play store. There’s no index which has links to all active apps in their marketplace (Apple has a nice alphabetically index list per category of all apps in the iTunes AppStore). The only way to discover apps in…
Pensieve 