Web scraping 101! Dive into the world of web scraping with Scott and Wes as they explore everything from tooling setup and navigating protected routes to effective data management. In this Tasty Treat episode, you'll gain invaluable insights and techniques to scrape (almost) any website with ease.

Show Notes

00:00 Welcome to Syntax!
03:13 Brought to you by Sentry.io.
05:00 What is scraping?
08:01 Examples of past scrapers.
10:06 Cloud app downloader.
16:13 Other use cases.
16:58 Scraping 101.
17:28 Client Side.
19:08 Private API.
22:40 Server rendered.
23:27 Initial state.
24:57 What format is the data in?
27:08 Working with the DOM.
27:12 Linkedom npm package.
29:02 querySelector everything.
31:28 How to find the elements without classes.
34:08 Use XPath selectors for select by word.
34:53 Make them as flexible as you can. Classes change!
35:10 AI is good at this!
36:26 File downloading.
38:20 Working with protected routes.
40:41 Programatically retrieve authentication keys because they are short-lived.
43:20 Deal-breakers.
44:58 What happened with Amazon?
46:42 Wes' portable refrigerator utopia.
47:25 Sick Picks & Shameless Plugs.

All links available at https://syntax.fm/763

------------------------------------------------------------------------------
Hit us up on Socials!

Scott:   / stolinski  
Wes:   / wesbos  
Randy:    / @randyrektor  
Syntax:   / syntaxfm  

http://www.syntax.fm

Brought to you by Sentry.io

#webdevelopment #webdeveloper #javascript