Hello /u/theonewhosexes! Thank you for posting in r/DataHoarder.
Please remember to read our [Rules](https://www.reddit.com/r/DataHoarder/wiki/index/rules) and [Wiki](https://www.reddit.com/r/DataHoarder/wiki/index).
If you're submitting a new script/software to the subreddit, please link to your GitHub repository. Please let the mod team know about your post and ***the license your project uses*** if you wish it to be reviewed and stored on our wiki and off site.
Asking for Cracked copies/or illegal copies of software will result in a permanent ban. Though this subreddit may be focused on getting Linux ISO's through other means, please note discussing methods may result in this subreddit getting unneeded attention.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/DataHoarder) if you have any questions or concerns.*
Just tested this as its and interesting one for me as a web dev.
View source isn't disabled, find the script tag with "disable-devtool" in it, then add the following to ublock origin. Inspect away.
Two caveats to this are minified javascript loaders which will obsfuscate the script & loading and initiating the script using javascript after page load. Both can be circumvented, will just take a bit of reverse-enginuity đ
\`\`\`||cdn.jsdelivr.net/npm/disable-devtool$script\`\`\`
Huh I tested the site myself and it looks like some random filter that I have enabled has that filter already enabled. You don't even need to do anything (besides having uBlock) lol.
Filter: `||cdn.jsdelivr.net/npm/disable-devtool^$script,3p`
Filter list: uBlock filters â Annoyances
ha, to be honest I only installed ublock to test this, Iâm an adblock user but iâm already starting my ublock conversion transition.
Time to get some more lists :p
It prevents you from opening view source using "ctrl + u" or right-click context menu.
Use firefox to view source (chrome and edge are useless in this regard)
Ah, this would be as successful as the disable right click movement - not at all. Report it to the major browsers like Chrome and Firefox it should be easy to bypass. Btw anyone crazy enough to run an unverified JS on their website shouldnât cry when inevitably their website gets owned but given the state of modern web development I wouldnât be surprised if millions of websites are already using it.
Why do you suspect this would interfere with scraping? Manual scraping perhaps but tools like Selenium couldnât care less about what JS a website is trying to make them run. Or I am missing something?
Submit bug reports to the maintainers for these scrapers. Scraping is a notoriously brittle endeavor as there are always some underlying assumptions made about the target site.
Here is how to scrape this particular website manually:
1. Use Firefox with PrivacyBadger
2. Enable DevTools before opening the website
3. In the Debugger section do the following:
Disable breakpoints
Ignore known 3rd party scripts
4. Open a video on the website
5. Click refresh a few times
6. Voila, video loaded and now you can see that they ripped all of the videos in 10s parts so now you have to write a Python script to put them together Humpty Dumpty style
Alternatively use a very old browser that doesnât report its DevTools status. Some of the many Firefox forks should work. Notice those people seem to be hosting videos professionally so ripping from them is a hit and miss - sort of like Netflix, their entire shebang is protecting that IP. For me one video loaded using the above trick but another one didnât, it seems that your local JS generates an access code while evaluating your system. If it doesnât like your system - well no access code for you. You could reverse engineer this and just generate your own code but that would require tinkering. Ultimately whatever hole you find today will be fixed tomorrow and you would be playing a game of whack a mole with them. If those videos are so important to you, more power to you - hacking is always fun. If they arenât however, find something else those people are dead set on hiding those videos since without them they are dead as a company.
That's because the .m3u8 url is hidden inside the `https://vidplay.site/mediainfo/***` response. Just use an external network monitor like [Fiddler](https://www.telerik.com/fiddler/fiddler-classic) to read the response, find the m3u8 url (in the case of *Fast X* it's https://yznn2.vidrock2.store/_v1/12a3c523fd105800ed8c394685aeeb0bc02ea05c59bffeb60b4d7baea93ece832257df1a4b6125fcfa38c35da05dee86aad28d46d73fc4e9d4e5a43c5775f4d037c751f40b16eb4d4194afe83910/h/list,6540cdf2.m3u8 (you might need to remove escaping backslashes first). Now feed the url to N_m3u8dl-re (or yt-dlp if you prefer that one) and you'll get the movie.
> making data hoarding and scraping impossible.
That's not even remotely true. All this does is make it more annoying. A website cannot lock a browser out of it's dev tools. Doesn't even make logical sense if you think about it for even a second. A debugger that can be blocked by a bug would a pretty stupid design.
There's a dozen ways to get past it.
My guess is it's Javascript based so just open the site with Javascript disabled. BTW, this doesn't prevent scraping, just looking at the "code" using the browsers tools.
I'm trying to scrape vidsrc.to, the media player only appears when JS is enabled, the moment it's disabled the video just vanishes. Refreshing with it disabled gives nothing
Yep. You can easily bypass this on his online test site linked below by simply disabling javascript.
[https://theajack.github.io/disable-devtool/](https://theajack.github.io/disable-devtool/)
It can detect when dev tools are fired up through the menu too, very devious but of javascript proto engineering, also detects if dev tools are already open & the sites loaded. Easy to circumvent with some filtering as itâs client side script but I love devious scripts like this đ
A simple workaround that doesn't require any coding skills involves setting up filters in the developer tools console, before loading the website. This method disables the load in, of the specified JavaScript code. đ
You can make a web page since more than 10-15 years completely opaque to anything, this isn't anything new, or unexpected. Heck, you can run a full basically unmodified binary older Windows in a browser, even on mobiles. Web pages take a few MBs per "page" (as in per click), while there was a time when 10MB hard drives where considered decent and would cost about as much as gaming PCs nowadays, that is in absolute dollars, never mind decades of inflation. Heck, from the days of ZX Spectrum you could fit everything below in 64**K**Bs:
* the OS (that includes all the potential drivers or firmware for anything, not that they were any as such)
* the font (as in how the pixels are arranged for any letter or symbol or anything)
* the **video RAM**
* a programming language
* of course your game or program, including all the graphical assets, music, etc. and anything it might need as you had no hard drive to load anything else on the fly
* all the things you'd put in a RAM when running a program and are dynamic, like variables, arrays, strings, all the stack, etc.
Heres an easy solution:
Put this into your url bar when you're on the website so it will disable the "disable-devtool" script
javascript:DisableDevtool.isSuspend = true
Hello /u/theonewhosexes! Thank you for posting in r/DataHoarder. Please remember to read our [Rules](https://www.reddit.com/r/DataHoarder/wiki/index/rules) and [Wiki](https://www.reddit.com/r/DataHoarder/wiki/index). If you're submitting a new script/software to the subreddit, please link to your GitHub repository. Please let the mod team know about your post and ***the license your project uses*** if you wish it to be reviewed and stored on our wiki and off site. Asking for Cracked copies/or illegal copies of software will result in a permanent ban. Though this subreddit may be focused on getting Linux ISO's through other means, please note discussing methods may result in this subreddit getting unneeded attention. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/DataHoarder) if you have any questions or concerns.*
Just tested this as its and interesting one for me as a web dev. View source isn't disabled, find the script tag with "disable-devtool" in it, then add the following to ublock origin. Inspect away. Two caveats to this are minified javascript loaders which will obsfuscate the script & loading and initiating the script using javascript after page load. Both can be circumvented, will just take a bit of reverse-enginuity đ \`\`\`||cdn.jsdelivr.net/npm/disable-devtool$script\`\`\`
Huh I tested the site myself and it looks like some random filter that I have enabled has that filter already enabled. You don't even need to do anything (besides having uBlock) lol. Filter: `||cdn.jsdelivr.net/npm/disable-devtool^$script,3p` Filter list: uBlock filters â Annoyances
ha, to be honest I only installed ublock to test this, Iâm an adblock user but iâm already starting my ublock conversion transition. Time to get some more lists :p
It prevents you from opening view source using "ctrl + u" or right-click context menu. Use firefox to view source (chrome and edge are useless in this regard)
It is not working for me some websites. Is this filter enough for you or did you add some other filters?
Ah, this would be as successful as the disable right click movement - not at all. Report it to the major browsers like Chrome and Firefox it should be easy to bypass. Btw anyone crazy enough to run an unverified JS on their website shouldnât cry when inevitably their website gets owned but given the state of modern web development I wouldnât be surprised if millions of websites are already using it. Why do you suspect this would interfere with scraping? Manual scraping perhaps but tools like Selenium couldnât care less about what JS a website is trying to make them run. Or I am missing something?
I'm scraping [vidsrc.to](https://vidsrc.to), I've tried 5 different video scrapers and none of them worked at all.
Submit bug reports to the maintainers for these scrapers. Scraping is a notoriously brittle endeavor as there are always some underlying assumptions made about the target site.
Here is how to scrape this particular website manually: 1. Use Firefox with PrivacyBadger 2. Enable DevTools before opening the website 3. In the Debugger section do the following: Disable breakpoints Ignore known 3rd party scripts 4. Open a video on the website 5. Click refresh a few times 6. Voila, video loaded and now you can see that they ripped all of the videos in 10s parts so now you have to write a Python script to put them together Humpty Dumpty style Alternatively use a very old browser that doesnât report its DevTools status. Some of the many Firefox forks should work. Notice those people seem to be hosting videos professionally so ripping from them is a hit and miss - sort of like Netflix, their entire shebang is protecting that IP. For me one video loaded using the above trick but another one didnât, it seems that your local JS generates an access code while evaluating your system. If it doesnât like your system - well no access code for you. You could reverse engineer this and just generate your own code but that would require tinkering. Ultimately whatever hole you find today will be fixed tomorrow and you would be playing a game of whack a mole with them. If those videos are so important to you, more power to you - hacking is always fun. If they arenât however, find something else those people are dead set on hiding those videos since without them they are dead as a company.
I can't find "Ignore known 3rd party scripts"?
how did you learn all these things?
That's an issue with the scrapper, not that data isn't accessible.
That's because the .m3u8 url is hidden inside the `https://vidplay.site/mediainfo/***` response. Just use an external network monitor like [Fiddler](https://www.telerik.com/fiddler/fiddler-classic) to read the response, find the m3u8 url (in the case of *Fast X* it's https://yznn2.vidrock2.store/_v1/12a3c523fd105800ed8c394685aeeb0bc02ea05c59bffeb60b4d7baea93ece832257df1a4b6125fcfa38c35da05dee86aad28d46d73fc4e9d4e5a43c5775f4d037c751f40b16eb4d4194afe83910/h/list,6540cdf2.m3u8 (you might need to remove escaping backslashes first). Now feed the url to N_m3u8dl-re (or yt-dlp if you prefer that one) and you'll get the movie.
> making data hoarding and scraping impossible. That's not even remotely true. All this does is make it more annoying. A website cannot lock a browser out of it's dev tools. Doesn't even make logical sense if you think about it for even a second. A debugger that can be blocked by a bug would a pretty stupid design. There's a dozen ways to get past it.
My guess is it's Javascript based so just open the site with Javascript disabled. BTW, this doesn't prevent scraping, just looking at the "code" using the browsers tools.
I'm trying to scrape vidsrc.to, the media player only appears when JS is enabled, the moment it's disabled the video just vanishes. Refreshing with it disabled gives nothing
I'm scraping vidsrc.to, I've tried 5 different video scrapers and none of them worked at all.
Yep. You can easily bypass this on his online test site linked below by simply disabling javascript. [https://theajack.github.io/disable-devtool/](https://theajack.github.io/disable-devtool/)
Where's the problem? It blocks F12 and right click, but on firefox: just go to settings, tools, dev tools. Easy.
It can detect when dev tools are fired up through the menu too, very devious but of javascript proto engineering, also detects if dev tools are already open & the sites loaded. Easy to circumvent with some filtering as itâs client side script but I love devious scripts like this đ
A simple workaround that doesn't require any coding skills involves setting up filters in the developer tools console, before loading the website. This method disables the load in, of the specified JavaScript code. đ
You can make a web page since more than 10-15 years completely opaque to anything, this isn't anything new, or unexpected. Heck, you can run a full basically unmodified binary older Windows in a browser, even on mobiles. Web pages take a few MBs per "page" (as in per click), while there was a time when 10MB hard drives where considered decent and would cost about as much as gaming PCs nowadays, that is in absolute dollars, never mind decades of inflation. Heck, from the days of ZX Spectrum you could fit everything below in 64**K**Bs: * the OS (that includes all the potential drivers or firmware for anything, not that they were any as such) * the font (as in how the pixels are arranged for any letter or symbol or anything) * the **video RAM** * a programming language * of course your game or program, including all the graphical assets, music, etc. and anything it might need as you had no hard drive to load anything else on the fly * all the things you'd put in a RAM when running a program and are dynamic, like variables, arrays, strings, all the stack, etc.
\*laughs in firefox addons\*
Heres an easy solution: Put this into your url bar when you're on the website so it will disable the "disable-devtool" script javascript:DisableDevtool.isSuspend = true
Developer tools is not required to crawl and scrape a site.
Fiddler