Crawl iTunes AppStore To Get List of All Apps

Git Repo: https://github.com/anuvrat/scrape-google-play/

Thankfully Apple AppStore provides a nice index to look up all the apps. All the apps have been categorized into 23 broad categories. Within each category the apps have been indexed alphabetically. So, to discover all the apps in the iTunes AppStore, one only needs to crawl the main index page, find all the category pages and then crawl the category alphabetical index.

The main index page of iTunes AppStore is at – https://itunes.apple.com/us/genre/ios/id36?mt=8.

From the main index page, one can get all url’s that match the regex: “^https://itunes.apple.com/us/genre/”. Category pages all have the url structure – https://itunes.apple.com/us/genre/ios-<category-name>/<category-id>?mt=8. For example, the apps categorized under Entertainment can be found at: https://itunes.apple.com/us/genre/ios-entertainment/id6016?mt=8.

In fact, one need not crawl the category page at all. The category page is alphabetically indexed, each alphabet page having the url structure: https://itunes.apple.com/us/genre/ios-<category-name>/<category-id>?mt=8&letter=<alphabet>&page=<page-number>. An example page being: https://itunes.apple.com/us/genre/ios-education/id6017?mt=8&letter=G&page=6.

I was able to discover 700K apps approximately! The script below also crawls each app page to extract information about the app.

Link: https://github.com/anuvrat/scrape-google-play/blob/master/src/crawlItunesStore.py

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: