yt-dlp sure has gotten complicated

[Grogan]

I noticed Arch pulled in a javascript runtime called "deno" with yt-dlp. It's a fairly large package. When it started coming up in updates often, I thought: Bollocks!

At that time I was able to just remove deno (it wasn't a hard package dependency) and just use yt-dlp without it. Then, Arch being Arch, they split out a separate package called yt-dlp-ejs, made that a package dependency for yt-dlp and made deno a package dependency for it.

So bollocks... I just started rolling up my own yt-dlp. That's a waste of time for scripts and/or bytecode but I won't have some big redundant javascript interpreter that they can't leave alone, just for snagging the occasional youtube video.

It still works without javascript, but warns admonishingly:

Code: Select all

[grogan@nicetry Downloads]$ yt-dlp -t mp4 https://www.youtube.com/watch?v=QQd0BAQlL2M
[youtube] Extracting URL: https://www.youtube.com/watch?v=QQd0BAQlL2M
[youtube] QQd0BAQlL2M: Downloading webpage
WARNING: [youtube] No supported JavaScript runtime could be found. Only deno is enabled by default; to use another runtime add  --js-runtimes RUNTIME[:PATH]  to your command/config. YouTube extraction without a JS runtime has been deprecated, and some formats may be missing. See  https://github.com/yt-dlp/yt-dlp/wiki/EJS  for details on installing one
[youtube] QQd0BAQlL2M: Downloading android vr player API JSON
[info] QQd0BAQlL2M: Downloading 1 format(s): 137+140
[download] Destination: Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].f137.mp4
[download] 100% of   24.45MiB in 00:00:04 at 6.07MiB/s
[download] Destination: Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].f140.m4a
[download] 100% of    4.33MiB in 00:00:00 at 7.56MiB/s
[Merger] Merging formats into "Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].mp4"
Deleting original file Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].f140.m4a (pass -k to keep)
Deleting original file Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].f137.mp4 (pass -k to keep)
[VideoRemuxer] Not remuxing media file "Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].mp4"; already is in target format mp4

Alright, alright, I have a node.js installation that I built for compiling Firefox. There's no reason I can't use that, eh?

Code: Select all

[grogan@nicetry Downloads]$ yt-dlp -t mp4 --js-runtimes node:/storage2/shit/build/node/bin/node https://www.youtube.com/watch?v=QQd0BAQlL2M
[youtube] Extracting URL: https://www.youtube.com/watch?v=QQd0BAQlL2M
[youtube] QQd0BAQlL2M: Downloading webpage
[youtube] QQd0BAQlL2M: Downloading android vr player API JSON
[youtube] QQd0BAQlL2M: Downloading web safari player API JSON
[youtube] QQd0BAQlL2M: Downloading player 4e51e895-tv
[youtube] [jsc:node] Solving JS challenges using node
WARNING: [youtube] [jsc] Remote component challenge solver script (node) was skipped. It may be required to solve JS challenges. You can enable the download with  --remote-components ejs:github  (recommended). For more information and alternatives, refer to  https://github.com/yt-dlp/yt-dlp/wiki/EJS
WARNING: [youtube] QQd0BAQlL2M: Signature solving failed: Some formats may be missing. Ensure you have a supported JavaScript runtime and challenge solver script distribution installed. Review any warnings presented before this message. For more details, refer to  https://github.com/yt-dlp/yt-dlp/wiki/EJS
WARNING: [youtube] QQd0BAQlL2M: n challenge solving failed: Some formats may be missing. Ensure you have a supported JavaScript runtime and challenge solver script distribution installed. Review any warnings presented before this message. For more details, refer to  https://github.com/yt-dlp/yt-dlp/wiki/EJS
[info] QQd0BAQlL2M: Downloading 1 format(s): 137+140
[download] Destination: Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].f137.mp4
[download] 100% of   24.45MiB in 00:00:01 at 12.47MiB/s
[download] Destination: Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].f140.m4a
[download] 100% of    4.33MiB in 00:00:00 at 12.55MiB/s
[Merger] Merging formats into "Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].mp4"
Deleting original file Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].f137.mp4 (pass -k to keep)
Deleting original file Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].f140.m4a (pass -k to keep)
[VideoRemuxer] Not remuxing media file "Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].mp4"; already is in target format mp4

Even that's not good enough, it admonishes for the solver javascript code (ejs). Sigh... OK, I'll just do it temporary like, since it downloads player shit every time anyway. (This stuff gets cached in ~/.cache/yt-dlp so it doesn't actually have to download those players and solver script every time)

Code: Select all

[grogan@nicetry Downloads]$ yt-dlp -t mp4 --js-runtimes node:/storage2/shit/build/node/bin/node --remote-components ejs:github https://www.youtube.com/watch?v=QQd0BAQlL2M
[youtube] Extracting URL: https://www.youtube.com/watch?v=QQd0BAQlL2M
[youtube] QQd0BAQlL2M: Downloading webpage
[youtube] QQd0BAQlL2M: Downloading android vr player API JSON
[youtube] QQd0BAQlL2M: Downloading web safari player API JSON
[youtube] QQd0BAQlL2M: Downloading player 4e51e895-tv
[youtube] [jsc:node] Solving JS challenges using node
[youtube] [jsc:node] Downloading challenge solver lib script from  https://github.com/yt-dlp/ejs/releases/download/0.4.0/yt.solver.lib.min.js
[youtube] QQd0BAQlL2M: Downloading m3u8 information
[info] QQd0BAQlL2M: Downloading 1 format(s): 96
[download] Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].mp4 has already been downloaded
[download] 100% of   28.90MiB
[VideoRemuxer] Not remuxing media file "Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].mp4"; already is in target format mp4

That shuts it up but now I've got an awful command line. Fine... I'll make a wrapper script. Very simple.

I have it as /usr/local/bin/yt-dlp-get so it's in PATH.

Code: Select all

#! /bin/bash

read -p "Paste youtube video URL here: " VIDURL

yt-dlp -t mp4 --js-runtimes node:/storage2/shit/build/node/bin/node --remote-components ejs:github $VIDURL

[Grogan]

Reading on their git pages, I noticed they distribute a standalone x86_64 ELF binary that has everything built in, the python interpreter, dependencies and everything (except glibc and zlib) but not the external javascript interpreter or ffmpeg. I need yt-dlp for Crux but I don't want to install all the python dependencies etc. to build it. Using someone else's binaries isn't my style (if I can't build it, I'm not using it) so I set out to build one of these on Arch. Since I'm binary compatible now (glibc 2.42+) this binary will run on Crux as well.

This is awesome, old school shit. I'm surprised they have this option tooled up like this, really. pyinstaller is pretty frowny these days (you'll get admonishments that you "don't need it on GNU/Linux!")

I was successful, though I had to install "pyinstaller" system wide (the build scripts use sys.path and won't use the one I installed in a virtual env and I don't know enough python to rewrite their scripts). To do that I had to use the --break-system-packages switch with pip (It didn't, I watched what it did and for everything I already had, it said "requirements already satisfied"). Arch does not provide this, and the AUR PKGBUILD is a 10 year old version.

It built a 20 Mb binary:

Code: Select all

[grogan@nicetry dist]$ ls -l
total 19940
-rwxr-xr-x 1 grogan grogan 20416104 Feb  6 17:26 yt-dlp_linux

[grogan@nicetry dist]$ file yt-dlp_linux
yt-dlp_linux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=3d4132e6aade8fe199dfd898058aa768ebd7f197, stripped
[grogan@nicetry dist]$ ldd yt-dlp_linux
	linux-vdso.so.1 (0x00007f2670fd0000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f2670f97000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007f2670f7e000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f2670f79000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f2670d98000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f2670fd2000)

Which I renamed to yt-dlp and stuck it in /usr/local/bin (I uninstalled the yt-dlp package from the system... good riddance to that bollocks. I'll upgrade it like this from now on, despite the wasteful binary)

It werks! (this command is using my little wrapper in the post above)

Code: Select all

[grogan@nicetry ~]$ yt-dlp-get
Paste youtube video URL here: https://www.youtube.com/watch?v=QQd0BAQlL2M    
[youtube] Extracting URL: https://www.youtube.com/watch?v=QQd0BAQlL2M
[youtube] QQd0BAQlL2M: Downloading webpage
[youtube] QQd0BAQlL2M: Downloading android vr player API JSON
[youtube] QQd0BAQlL2M: Downloading web safari player API JSON
[youtube] QQd0BAQlL2M: Downloading player 4e51e895-tv
[youtube] [jsc:node] Solving JS challenges using node
[youtube] [jsc:node] Downloading challenge solver core script from  https://github.com/yt-dlp/ejs/releases/download/0.4.0/yt.solver.core.min.js
[youtube] QQd0BAQlL2M: Downloading m3u8 information
[info] QQd0BAQlL2M: Downloading 1 format(s): 96
[download] Sleeping 5.00 seconds as required by the site...
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 54
[download] Destination: Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].mp4
[download] 100% of   32.27MiB in 00:00:07 at 4.35MiB/s
[FixupM3u8] Fixing MPEG-TS in MP4 container of "Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].mp4"
[VideoRemuxer] Not remuxing media file "Uriah Heep -  Dreammare (high quality audio) [QQd0BAQlL2M].mp4"; already is in target format mp4

------------------------------------

How I built this:

Code: Select all

# pip install --break-system-packages pyinstaller

(Bad toad... but this particular thing was safe to do, and the risk is mine, I know how to fix shit if I break it and/or after I upgrade python)

Super easy after that, it's all scripted (and I already have the rest of the dependencies on Arch)

Code: Select all

git clone https://github.com/yt-dlp/yt-dlp.git

cd yt-dlp
python devscripts/make_lazy_extractors.py
python -m bundle.pyinstaller

and it compiles it. The result is in yt-dlp/dist

[Grogan]

So I found out that this in the command:

--remote-components ejs:github

does NOT upgrade the cached scripts in ~/.cache/yt-dlp when they already exist. I was getting errors that the js challenge solver wasn't working with node anymore (though it wasn't actually failing to download any videos that I was getting) and suggesting to "upgrade" (something).

So I upgraded my yt-dlp build as above and it was still doing it. I thought maybe it needed a newer node.js, but no, I just had to delete ~/.cache/yt-dlp/challenge-solver and let it download a new one.

Then, with the new build I was getting warnings (but it wasn't affecting anything because I didn't need any charset normalization). There's a new dependency since my last build.

requests/__init__.py:86: RequestsDependencyWarning: Unable to find acceptable character detection dependency (chardet or charset_normalizer).

Sigh... I had to install python-charset-normalizer and recompile my yt-dlp binary again (because it's got all that built in statically). The thing about yt-dlp is that a lot of the dependencies are optional (the program just won't have that functionality) when you roll it up yourself, but I might as well get rid of errors and warnings.