purl purl, youtube perl one liner downloaderLast time I explained how YouTube videos can be downloaded with gawk programming language by getting the YouTube page where the video is displayed and finding out how the flash video player retrieves the FLV (flash video) media file.

This time I'll use Perl programming language which is my favorite language at the moment and write a one-liner which downloads a YouTube video.

Instead of parsing the YouTube video page, let's look how an embedded YouTube video player on a 3rd party website gets the video.

Let's go to this cool video and look at the embed html code:

html code for embedded youtube video

For this video it looks as following:

<object width="425" height="350"><param name="movie" value="http://www.youtube.com/v/qg1ckCkm8YI"></param><param name="wmode" value="transparent"></param><embed src="http://www.youtube.com/v/qg1ckCkm8YI" type="application/x-shockwave-flash" wmode="transparent" width="425" height="350"></embed></object>

95% of this code is boring, the only interesting part is this URL:

http://www.youtube.com/v/qg1ckCkm8YI

Let's load this in a browser, and as we do it, we get redirected to some other URL:

http://www.youtube.com/jp.swf?video_id=qg1ckCkm8YI&eurl=&iurl=http://img.youtube.com/vi/qg1ckCkm8YI/default.jpg&t=OEgsToPDskJCPW5DvMKeM3srnQ5e0LSY

So far we have no information how the flash player will retrieve the video, the only thing we know that 'iurl' stands for 'image url' and is the location of the thumbnail image.

Let's sniff the traffic again, this time with an excellent (though, commercial) Internet Explorer plugin 'HttpWatch Professional'.
This plugin displays all the requests the browser makes no matter if it's HTTP or HTTPS traffic and displays in a nice manner which makes our job much quicker than by using Ethereal.
The FireFox's alternative to this tool is Live HTTP Headers extension which basically does the same as HttpWatch Professional but it takes more time to understand the output.

Here is what we see with HttpWatch Professional when we load the URL in the browser:

output of httpwatch professional sniffer (sane thumbnail (not the wordpress insanity))

We see that to get a video browser first requested:

http://www.youtube.com/get_video?video_id=qg1ckCkm8YI&t=OEgsToPDskJ3bp4DEiMuxUmjx7oumUec&eurl=

then got redirected to:

http://cache.googlevideo.com/get_video?video_id=qg1ckCkm8YI

and then another time to:

http://74.125.13.83/get_video?video_id=qg1ckCkm8YI

This is exactly what what we saw in the previous article on downloading videos with gawk!

Now let's write a Perl one-liner that retrieves this video file!

What is a one-liner you might ask? Well, my definition of one liner is that it is a program you are willing to type out without saving it to disk.

First of all we will need some perl packages (modules) which will ease working with HTTP protocol. There are two widely used available on Perl's module archive (CPAN) - LWP and WWW::Mechanize.

WWW::Mechanize is built on top of LWP, so let's go to a higher level of abstraction and use this module.

The WWW::Mechanize package does not come as Perl's core package by default, so you'll have to get it installed.
To do it, type

perl -MCPAN -eshell

In your console and when the CPAN shell appears, type

install WWW::Mechanize

to get the module installed.

If everything goes fine, the CPAN will tell you that the module got installed.

I don't want to go into Perl language's details again, also I don't want to go into WWW::Mechanize package's details.

If you want to learn Perl I recommend this article as a starter, these books and of course perldoc. Once you learn the basics you can quickly pick up the WWW::Mechanize package by reading the documentation, faq and trying examples.

Now finally let's write the one-liner. So what do we have to do?

First we have to retrieve

http://www.youtube.com/v/qg1ckCkm8YI

then follow the redirect (which WWW::Mechanize will do for us), then get the 't' identifier from query string and finally request and save output of

http://www.youtube.com/get_video?video_id=qg1ckCkm8YI&t=OEgsToPDskJ3bp4DEiMuxUmjx7oumUec&eurl=

That's it!

So here is the final version which can probably be made even shorter:

perl -MWWW::Mechanize -e '$_ = shift; s#http://|www\.|youtube\.com/|watch\?|v=|##g; $m = WWW::Mechanize->new; ($t = $m->get("http://www.youtube.com/v/$_")->request->uri) =~ s/.*&t=(.+)/$1/; $m->get("http://www.youtube.com/get_video?video_id=$_&t=$t", ":content_file" => "$_.flv")'

A little longer than a usual one-liner but does the job nicely. To keep it short, there is no error checking!

To use this one-liner just copy it to command line and specify the URL of a YouTube video (or just the ID of the video, or a variation of URL (like without 'http://'). Like this:

perl -MWWW::Mechanize -e '...' http://www.youtube.com/watch?v=l69Vi5IDc0g

or just

perl -MWWW::Mechanize -e '...' l69Vi5IDc0g

Let's spread this one liner to multiple lines and see what it does as it is not documented.
One could do the spreading out to multiple lines by hand, but that's not what humans are for, let's make Perl do it. By adding -MO=Deparse to the command line list we get the output of the Perl generated source code (i added line numbers myself):

use WWW::Mechanize;
1) $_ = shift @ARGV;
2) s[http://|www\.|youtube\.com/|watch\?|v=|][]g;
3) $m = 'WWW::Mechanize'->new;
4) ($t = $m->get("http://www.youtube.com/v/$_")->request->uri) =~ s/.*&t=(.+)/$1/;
5) $m->get("http://www.youtube.com/get_video?video_id=$_&t=$t", ':content_file', "$_.flv");

So our one liner is actually 5 lines.
On line 1 we put the first argument of ARGV variable into special variable $_ so we could use advantage of it and save some typing.
On line 2 we just leave the ID of the video by removing parts from the URL one by one so a user could specify the video URL in various formats like 'www.youtube.com/watch?v=ID, or just 'youtube.com?v=ID' or just 'v=ID' or even just 'ID'. The ID gets stored in the special $_ variable.
On line 3 we create a WWW::Mechanize object we are going to use twice.
Line 4 needs more explanation because we are doing so much in it. First it retrieves that embedded video URL I talked about earlier, the server actually redirects us away, so we have to look at the last request's location. We save this location into variable $t and then extract the 't' YouTube ID out.
As a YouTube video is uniquely specifed with two IDs, the video ID and 't' ID, on line 5 we retrieve the file and tell WWW::Mechanize to save contents to the ID.flv file. WWW::Mechanize handles redirects for us so everything should work. Indeed, I tested it out and it worked.

Can you golf it shorter? :)

I golfed it a little myself, here is what I came up with:

perl -MWWW::Mechanize -e '$_ = shift; ($y, $i) = m#(http://www\.youtube\.com)/watch\?v=(.+)#; $m = WWW::Mechanize->new; ($t = $m->get("$y/v/$i")->request->uri) =~ s/.*&t=(.+)/$1/; $m->get("$y/get_video?video_id=$i&t=$t", ":content_file" => "$i.flv")'

To use this one liner you must specify the full URL to youtube video, like this one:

http://www.youtube.com/watch?v=l69Vi5IDc0g

This one liner saves the "http://www.youtube.com" string in variable $y and the ID of the video in variable $i. The $y comes handy because we don't have to use the full YouTube URL, instead we use use $y.

Update 2009.12.05: YouTube has changed the way it displays videos several times! The current one-liner is here:

perl -MWWW::Mechanize -e '$m = WWW::Mechanize->new; $_=shift; ($i) = /v=(.+)/; s/%(..)/chr(hex($1))/ge for (($u) = $m->get($_)->content =~ /l_map": .+(?:%2C)?5%7C(.+?)"/); print $i, "\n"; $m->get($u, ":content_file" => "$i.flv")'

Also, are you interested in Perl programming language? Here are three excellent books on Perl from Amazon (recommended by me):

Comments

Saldane Permalink
July 27, 2007, 07:34

All of the methods you used from WWW::Mechanize were inherited from LWP...

Here's a first go at a golf, very similar to yours:

perl -MLWP -e '($y,$i)=shift=~/^(.+m)\/.+v=(.+)/;($m=LWP::UserAgent->new)->get("$y/get_video?video_id=$i&t=".($m->get("$y/v/$i")->request->uri=~/&t=(.+)/)[0],":content_file"=>"$i.flv")'
'http://www.youtube.com/watch?v=l69Vi5IDc0g'
Saldane Permalink
July 27, 2007, 07:38

With some whitespace:

perl -MLWP -e '($y,$i) = shift =~ /^(.+m)\/.+v=(.+)/; ($m = LWP::UserAgent->new) ->get("$y/get_video?video_id=$i&t=" . ($m->get("$y/v/$i") ->request->uri =~ /&t=(.+)/)[0], ":content_file" => "$i.flv")'

Saldane Permalink
July 27, 2007, 07:46

Sorry to spam, I'm new to the intertubes.

Third time's the charm (perltidied):

perl -MLWP -e'
( $y, $i ) = shift =~ /^(.+m)\/.+v=(.+)/;
( $m = LWP::UserAgent->new )->get(
    "$y/get_video?video_id=$i&t="
      . ( $m->get("$y/v/$i")->request->uri 
             =~ /&t=(.+)/ 
        )[0],
    ":content_file" => "$i.flv"
  )
'
Shawn Permalink
July 27, 2007, 07:52

A little shorter:

perl -MWWW::Mechanize -e'$y="http://youtube.com";($i)=pop=~/\w+$/g;$m=new WWW::Mechanize;$m->get("$y/v/$i")->request->uri=~/&t=.+/;$m->get("$y/get_video?video_id=$i$&",":content_file"=>"$i.flv")'
Shawn Permalink
July 27, 2007, 07:54

heh, same problem as Saldane. here it is with unnecessary \ns after semicolons:

perl -MWWW::Mechanize -e'$y="http://youtube.com";
($i)=pop=~/\w+$/g;
$m=new WWW::Mechanize;
$m->get("$y/v/$i")->request->uri=~/&t=.+/;
$m->get("$y/get_video?video_id=$i$&",":content_file"=>"$i.flv")'
ani625 Permalink
July 27, 2007, 08:13

Intermediate Perl
http://www.flazx.com/ebook4407.php

Saldane Permalink
July 27, 2007, 18:42

nice.. and quoted for windows shell:

perl -MLWP -e"$y='http://youtube.com';($i)=pop=~/\w+$/g;($m=new LWP::UserAgent)->get(qq{$y/v/$i})->request->uri=~/&t=.+/;$m->get(qq{$y/get_video?video_id=$i$&},':content_file',$i.'.flv')" "l69Vi5IDc0g"
July 27, 2007, 18:54

Sweet! Thanks for golfing ;)

I noticed the comments do not look good at all. I will fix the design so that the code did not get cut off if it runs over the edge :)

July 28, 2007, 01:44

That is nice. A friend of mine always boasted about Perl and how good it is.

________________
http://www.FreeOpenMoko.com

August 12, 2007, 03:29

I have to say, that I could not agree with you in 100% regarding o.us poetry, but it's just my opinion, which could be wrong :)

August 12, 2007, 06:50

Daniel, what do you mean by 'o.us poetry'?

August 15, 2007, 09:31

problem, video -CrLh0xR3FM causes problems

there is a 0xR in it - and perl recognizes this as unicode. Anyway you can quote it to take in the whole string?

Greg Permalink
November 04, 2007, 02:54

I put together the following korn script from your perl code... It downloads the video and converts it to a DVD-style MPEG. Good work; I hope others will find it useful!

#!/bin/ksh

set -e

if [ -z "$1" ]; then
  echo "Please supply quoted URL as argument."
  exit 1
fi

URL="$1"
FILE=`echo "$URL" | awk -F 'v=' '{print $2}'`

perl -MWWW::Mechanize -e '$_ = shift; ($y, $i) = m#(http://www\.youtube\.com)/watch\?v=(.+)#; $m = WWW::Mechanize->new; ($t = $m->get("$y/v/$i")->request->uri) =~ s/.*&t=(.+)/$1/; $m->get("$y/get_video?video_id=$i&t=$t", ":content_file" => "$i.flv")' "$URL"

mencoder -of mpeg -mpegopts format=dvd -ofps 30000/1001 -oac lavc -ovc lavc -srate 48000 -af lavcresample=48000 -vf scale=704:480,expand=720:480 -lavcopts acodec=ac3:abitrate=192:vcodec=mpeg2video:vrc_buf_size=1835:vrc_maxrate=9800:vbitrate=1856:keyint=18:aspect=4/3 -o "$FILE.mpeg" "$FILE.flv"
November 04, 2007, 06:26

Greg, thanks for the script :)

Kankani Permalink
November 07, 2007, 11:06

I am not able to download youtube vieo using VBScript file. I am still getting the .dll error even though I opened IE and did the required change? Could you tell me why is that?

Thanks.
-Manish

Kankani Permalink
November 07, 2007, 11:07

I am not able to download youtube video using VBScript file. I am still getting the .dll error even though I opened IE and did the required change? Could you tell me why is that?

Thanks.
-Manish

November 07, 2007, 22:34

Kankani, what .dll error?

February 03, 2008, 05:06

I'm not a programmer so I guess I'll stick to How To Download YouTube Videos The Easy Way For Free. :)

April 01, 2008, 18:05

Hi,
this is great and it works perfectly (I'm wonrking on Windows). I have one question, is it possible to have the percentage accomplished being displayed to know how long it will take to complete ? I it could be done that'd awesome.

Thanks a lot

Parth Patil Permalink
April 19, 2008, 09:48

After reading the comment from Vinniemc I thought of taking a stab at showing some kind of progress indicator.
For showing progress indicator I had to find a mechanism where LWP::UserAgent would call my function after it received each chunk of file. I was delighted to find in LWP::UserAgent's perldoc that its possible to specify a call back method to LWP::UserAgent's get() method via the special field name ":content_cb". After trying unsuccessfully to use this call back functionality I went back to the LWP perldoc. On re-reading I found that its not possible to use the option ":content_file" & ":content_cb" at the same time !
After searching some more I found the lwp cookbook which has an example of manually processing http responses. So based on that I was able to hack up the progress indicator. Unfortunately it hardly qualifies as a one liner anymore! In my attempt to still make the script small it has become a little obfuscated and might be difficult to understand. So here is the code

perl -MLWP -e '$_ = shift; ($y, $i) = m#(http://www\.youtube\.com)/watch\?v=(.+)#; $m = LWP::UserAgent->new; ($t = $m->get("$y/v/$i")->request->uri) =~ s/.*&t=(.+)/$1/; open($fh,">$i.flv");binmode($fh);$t1=$t2=time;print "\n";$res = $m->request(HTTP::Request->new(GET => "$y/get_video?video_id=$i&t=$t"),sub { ($c,$res) = @_;$br += length($c);$t2 = time;if($t2 > $t1){if ($res->content_length) {printf STDERR "%d%% - ",100*$br/$res->content_length;$t1= $t2;}}print $fh $c;});close($fh);print "\n";' http://www.youtube.com/watch?v=l69Vi5IDc0g
July 03, 2008, 03:49

I want to download videos and movies.

July 09, 2008, 06:55

Why Perl? You can even do it with bash!
And you also get to download the mp4 format and a download o-meter for free.

#!/bin/bash
#
# getvideo.sh
# Copyright (C) 2008 by Quandary
# Contact: http://www.nixcoders.org/forum/index.php?showuser=215
# Released under the GPL 3.O
#
# Download youtube videos with bash and wget;-)
#
# run: chmod +x getvideo.sh
# run:  ./getvideo.sh YOUTUBE_URL
# e.g.  ./getvideo.sh http://www.youtube.com/watch?v=00KYPsQSG10
#
# Last modified: Jul 09, 2008


#youtubeURL="http://www.youtube.com/watch?v=43sUEFcD4Ro" # only has flv
#youtubeURL="http://www.youtube.com/watch?v=00KYPsQSG10" # has mp4 & flv
youtubeURL=$1; # get command line argument 1
youtubeURL=$(echo "$youtubeURL") # argh, damn bash! Remove invisible double quote

reset
echo "Downloading video: $youtubeURL"
echo ""
youtubeHTML=$(wget -O - "$youtubeURL" 2> /dev/null)
#youtubeHTML=$(cat "$(echo ~)/Desktop/teletubbie.htm") # cat it from an existing file

#echo "HTML Page:"
#echo "$youtubeHTML"
#echo "$youtubeHTML" >> youtube.htm # For review, if the keyword changes


#flvURL=$(echo "$youtubeHTML" | grep -o -E 'player2\.swf?[^\"]+' | head -n 1 | sed 's/^player2\.swf?//g')
flvURL=$(echo "$youtubeHTML" | grep 'fullscreenUrl' | sed -re 's/^.+video_id=//;s/&hl=.+//') # sed -re enables regex search syntax extensions
echo "Encrypted URL = $flvURL"
echo ""

VideoTitle=$(echo "$youtubeHTML" | grep "" | sed 's///;s///;s/YouTube - //;s/^\s*//g')
FLVfile=$(echo "$youtubeHTML" | grep "" | sed 's///;s//.flv/;s/YouTube - //;s/^\s*//g')
MP4file=$(echo "$youtubeHTML" | grep "" | sed 's///;s//.mp4/;s/YouTube - //;s/^\s*//g')
#sed 's/^\s*//g' # finally, trim trailing whitespaces
echo "Video title: $VideoTitle"
echo ""

echo "Starting wget: "
FLVdownloadURL="http://www.youtube.com/get_video.php?hl=en&video_id=$flvURL"
MP4downloadURL="http://www.youtube.com/get_video.php?hl=en&video_id=$flvURL&fmt=18"
# append &fmt=6    FLV
# append &fmt=18   MP4
echo "FLV download URL: $FLVdownloadURL"
echo "MP4 download URL: $MP4downloadURL"
echo ""

wget -O "$FLVfile" "$FLVdownloadURL"
wget -O "$MP4file" "$MP4downloadURL"
reset

echo ""
echo "VoilĂ , download of \"$VideoTitle\" completed."
July 23, 2008, 10:24

wow,perl is so strong.

Paulo Permalink
August 03, 2008, 18:34

Very good..
It was very useful. ;)

September 12, 2008, 05:12

Download videos from popular video sharing sites like youtube.com,
blip.tv, break.com, google video, ifilm.com, spike.com etc. Simply
copy the video url and paste it to the video url box at vidmaza.com
and click to download.Also vidmaza.com try to find different video
formats automaticly(If avaliable).

Tired of copying and pasting urls of video files, try vidmaza.com search
videos function, it searches entirely in youtube.com for videos according
to your search criteria.Simply choose the format of video at results page
and download.

nikonlover Permalink
September 21, 2008, 07:55

Trying to develop a one liner to download the video from CNBC video pages without success.
for example from:
http://www.cnbc.com/id/15840232?video=861445025&play=1
It generally plays an ad and then the video. Sometimes the ad is missing and it plays the video directly. No way to tell.
Any pointers ?

October 16, 2008, 19:58

The price for HttpWatch starts from $300, there are some other good http analyzers. For example http debugger or fiddler.

Chirantan Permalink
October 24, 2008, 05:41

Hi, I tried to do the same thing using Ruby. Not the one liner though. Its a piece of code. It worked great initially as I was able to extract value of 't' from the redirected url.

But now, after a few day, the script stops working. I investigated and found out that the redirected url which initially held the value of 't' does not contain the value of 't' anymore and hence the regex that was used to retrieve it was fetching nil.

How to go about it now? As value of t is unknown. (Or atleast I could not figure out a way to find it out)

Chirantan Permalink
October 24, 2008, 06:00

I am getting redirected to

http://www.youtube.com/swf/l.swf?swf=http%3A//s.ytimg.com/yt/swf/cps-vfl61304.swf&video_id=AP5VIhbJwFs&rel=1&showsearch=0&eurl=&iurl=http%3A//i2.ytimg.com/vi/AP5VIhbJwFs/hqdefault.jpg&use_get_video_info=1&load_modules=1&hqt=1

If that helps. The redirection stops here. Earlier this kind url used to have the 't' value. But it does not anymore.

October 25, 2008, 19:04

Chirantan,

You'll have to use tcpdump, then download the video using a browser with flash. When the browser views the video the output of tcpdump will show the final url.

max Permalink
December 07, 2008, 15:25

Beautiful!!!
But how do you write this code for running it on your own pc, e.g. with ActivePerl?

Muhammad Permalink
January 11, 2009, 17:03

Hi peter,The script Doesnt work with me.dont know why!!
it just run for 2 secs and end with nothing!!!.any clues?

February 04, 2009, 02:28

youtube must have changed its format since you posted this, since what was being returned did not have a t get variable. The t value was stored in the javascript of the page, so i used the following regex to get it:

/, "t": "([^"]+)"/

which i found out after looking at a python script youtube-dl at http://www.arrakis.es/~rggi3/youtube-dl/ but while their downloader is very robust and object oriented, this one is quick and perl-like but gets the job done. Heres one that works as of february 2009:

use WWW::Mechanize;
use Number::Bytes::Human qw(format_bytes);

for (@ARGV) {
    s{http://|www\.|youtube\.com/|watch\?|v=|}{}g;

    $m = WWW::Mechanize->new;
    $m->get("http://www.youtube.com/watch?v=$_&gl=US&hl=en");
    ($t) = $m->content =~ /, "t": "([^"]+)"/;

    open $f, "> $_.flv";
    binmode $f;

    $m->get(
        "http://youtube.com/get_video?video_id=$_&t=$t",
        ':content_cb' => sub {
            ($c, $r) = @_;
            $b += length($c);
            if ($r->content_length) {
                printf STDERR "$_: %.2f%%: %s of %s            \r",
                    100. * $b / $r->content_length,
                    format_bytes($b),
                    format_bytes($r->content_length);
            }
            print $f $c;
        });
    
    $b = 0;
    print "\n";
    close $f;
}

I combined some things that i liked from the previous posts (such as the callback).

Beefcake Permalink
March 23, 2009, 19:25

$_ = shift; ?

just say "shift;"

ding Permalink
April 07, 2009, 10:34

what????

April 07, 2009, 10:35

what???

Josh Permalink
May 24, 2009, 17:26

Thank you...

But i found "Zillatube" program download video quickly, and also easy to play those videos

too.

found it at http://www.zillatube.com

June 03, 2009, 05:16

PS - If you have to use a browser with flash support, you might as well use a plugin like the author mentioned: e.g. LiveHTTPheaders for Firefox, which gives can handle regex and you a formatted URL. Unless you're piping the tcpdump output into something else.
YouTube has made it easier to download directly. I think because so much effort has been put by users into making this easier, whether with Perl, Awk, Python, VB, whatever.
If someone publishes an article on how to get the URL without using a flash browser, then I think we will really see a change in how sites use flash. Because ultimately people just want the video; they don't care how they get it. And not everyone across the globe has a fast internet connection. Streaming is great if you have the bandwidth for it. But not everyone in the world has the bandwidth.

June 05, 2009, 00:27

Addendum:

There is a Microsoft Research (command line) tool (actually it's several) that is quite good for nicely formatted HTTP conversations: it's called STRACE. It comes with an HTTP replay utility as well. You can also use a special wininet.dll with MSIE that MS Research released many years ago.

The tool we really need though is one that has all the 'handshaking' (redirections, javascript, swf, etc) capabilities of Mozilla coupled with built-in 'livehttpeaders' functions, and _nothing else_. No rendering engines and other bloat. This tools would get you the URL to the content. That is its sole job. Then you download the content with a tool that works (unlike the download mamagers of the browsers). And finally you play the content offline with a standalone tool that works on any media file (you should know which ones can do that from your own experience trying different ones).

Alas, the developers seems to think they can include all these steps in one application ("plugins") and have it work, seamlessly... instead of following the UNIX way of letting each tool do its job and piping the task from one application to the next.

Advertising (after all we do need that right?) can be 'attached' to the content.
An example is the TED videos, where you see a BMW ad at the start of each video.

This has been said before by people smarter than me: all the html tags flowing through the 'wire' are largely unnecessary, and slow things down. Almost all value is in plain text and links to more text or to content. 'Typesetting' (e.g. html gimmicks) is only residual value.

June 05, 2009, 00:34

s/all the html tags/many of the html tags/

What I mean is that there is 'useful' markup and then there are 'gimmicks'.

vishnu Permalink
July 05, 2009, 20:20

no commands

July 29, 2009, 07:32

It is a very good thing,

argv Permalink
August 17, 2009, 03:07

I give the edge to quandary based on non-'base system' lines of code needed.

A shell script plus the wget binary (or whatever small http client binary you use)

vs.

WWW::Mechanize and all it dependencies plus the perl binary plus the oneliner

And re: the 'whatever-zilla' spam by 'Josh', in addition to pk's impressive demos, we already have
- statically-linked cclive (curl)
- youtube-dl (python)

Both free and very reliable, even on the slowest connections.

Matrixx Permalink
August 22, 2009, 12:40

http://video.deshibinodon.com is a dynamic video portal from where you can search, watch & download video in different format. No need any software.

August 31, 2009, 10:34

Here is another very useful website for downloading YouTube videos in many formats. There are a lot of other great options so it's worth visiting!

December 12, 2009, 05:04

I appreciated the step by step. Have you had a chance to try this on a win32 platform? Good catch on the 12/5/2009 youtube changes. I am not sure the mechanize extraction will work on the win32 platform. What I am more interested in is identifying the embed code box on any website screen and extracting the object embed code, opening a flat file and adding the embed code at either the end or back where it came from. The flat file then becomes a database with numerical pointer reference index to video feed. The technology I have developed also hides address reference to tools like HTTPwatch which I know google is pursuing. You can see some of it in action on my site at http://nwufochasers.com or barring all of the above I could simply use a perl script that grabs data off a google searched screen. You got what I want for christmas, better programing skills and understanding of object linkage. Anyway thanks maybe we can chat sometime.

Michael Permalink
December 14, 2009, 02:30

Today I made my first Perl steps, inspired by this one-liner. I added a direkt pipe to convert the file to mp3 with ffmpeg, a name extraction from the youtube page, like in the awk script, and a download status bar, to have some more convenience than with the one-liner. Comments for better code style anytime welcome.

#!/usr/bin/perl
# Requires some packages
# For debian/ubuntu ->
# sudo apt-get install libwww-perl libwww-mechanize-perl ffmpeg

$outdir = $ENV{HOME}."/incoming/youtube";
chdir $outdir or die "Chdir \"$outdir\": $!";

use WWW::Mechanize;
use feature 'state';

$_ = shift;
my $m = WWW::Mechanize->new;
s!http://|www\.|youtube\.(com|de)/|watch\?|v=|&[^v][^=][^&]*!!g; # remove all except id
s/%(..)/chr(hex($1))/ge for 
	(($u) = $m->get("http://www.youtube.com/watch?v=$_")->content =~ /l_map": .+(?:%2C)?5%7C(.+?)"/);

my $stream = HTML::TokeParser->new(\$m->content);
$stream->get_tag("title");
my $name = $stream->get_trimmed_text("/title");
$name =~ s/^YouTube\s*-\s*//;
$name =~ s![\\/]!-!g;
$name =~ s![^\w,.-]! !g;
$name =~ s/^\s+|\s+$//g;

STDOUT->autoflush(1);
open(CONVERT, "| ffmpeg -i - -y -acodec copy \"$outdir/$name.mp3\" 2>/dev/null") 
	or die "Can't fork ffmpeg: $!";
$m->get($u, ":content_cb" => \&get_chunk) or die "Get \"$u\": $!";
close(CONVERT);
print "\nDone.\n";

sub get_chunk {
	($data, $response) = @_;
	print CONVERT $data;
	state $b = 0;
	state $l = $response->content_length;
	if ($l) {
		$b += length($data);
		$percent = 100. * $b / $l;
		print STDOUT sprintf("Downloading \"$name\": %.0f%%  \r", $percent);
	}         
}
Vangran Permalink
January 10, 2010, 14:39

/dev/null what is this in Win32?

or how do we translate this line of code in Win32?

open(CONVERT, "| ffmpeg -i - -y -acodec copy "$outdir/$name.mp3" 2>/dev/null")
or die "Can't fork ffmpeg: $!";

Also I noticed some minor modifications like in line 8 from chdir $outdir or die "Chdir "$outdir": $!"; to chdir $outdir or die "Chdir ".$outdir.": $!"; Basically the String concatination wont work without the dot (.) on both sides of $outdir.

February 24, 2010, 04:21

Thanks but there's a very easy way to download & play Youtube on your computer.
It's FLV Direct. You can learn more & download at: http://bit.ly/935tJJ for free
Good luck!

jim Permalink
April 17, 2010, 09:00

Anyone have an update since Youtube updated their code in April 2010?

June 12, 2010, 13:54

Try Getv: http://getv-en.super-idol.com/

Detailed tutorial to download youtoube videos by Getv:
http://getvmaster.blogspot.com/2010/06/how-to-use-getv-to-download-youtube.html

Just like Keepvid, Youtube Downloader,
Getv is a free, easy, safe site to download youtube videos.
Paste the video URLs and you can get your favorite videos very quickly.
Videos can be downloaded as FLV, MP4, 3GP(good quality as well), etc,
You can also download videos on facebook, DailyMotion, Break, Mofile,
Tudou and more on Getv.

July 06, 2010, 18:17

very useful stuff. Thanks a lot

asas Permalink
July 23, 2010, 21:10

get_video does not work anymore since middle of july 2010

July 24, 2010, 09:34

Yeah, it's all broken. But I am no longer supporting these projects.

September 13, 2010, 09:54

Youtube Downloader is a fab youtube video downloading tool. I love youtube clips and have a huge collection stored on my hard drive.

February 20, 2011, 14:08

10 lines Youtube Downloader (new)

Downloads Youtube videos with wget at the best quality available with Youtube title as file name.

[code]
use LWP;
use URI::Escape;
foreach $_ (@ARGV) {
$_ =~ s[^.*(v|embed)[=/]+([\w\-]{11}).*][$2];
($c = 'LWP::UserAgent'->new->get("http://www.youtube.com/get_video_info?&video_id=$_&el=detailpage&ps=default&eurl=&gl=US&hl=en")->content) ? $c = uri_unescape($c) : '';
$c =~ /&title=([^&]+)/ ? $t = $1 : '';
$t =~ s[/|\||`|"|\+][ ]g;
$c =~ s/.*fmt_url_map=[\d\|]+([^,]+).*/$1/;
exec qq[wget "$c" -O "$t.mp4"];
}
[/code]

Enjoy!

Jay Permalink
March 24, 2011, 02:41

Thanks Trizen. That works perfectly for me. Quick question: where is the 'best quality' part? I actually want the quickest download/lowest quality. How would I modify your query?

dhunnapotha Permalink
March 08, 2011, 08:49

nice post.

@Trizen:
thanks for the code. It works !!
Looks like your site has lot of useful tools. Will go thru them when i get time :)

March 08, 2011, 19:41

Or with support for multiple URLs or video codes:

use LWP;
use URI::Escape;
foreach $_ (@ARGV) {
$_ =~ s[^.*[=/]?([\w\-]{11}).*][$1];
$c = uri_unescape($t = 'LWP::UserAgent'->new->get("http://www.youtube.com/get_video_info?&video_id=$_&el=detailpage&ps=default&eurl=&gl=US&hl=en")->content);
$t =~ /&title=([^&]+)&/ ? $t = uri_unescape($1) : '';
$t =~ s[\+|/][ ]g ? $t = quotemeta $t : '';
$c =~ s/.*fmt_url_map=[\d\|]+([^,]+).*/$1/;
system qq[wget "$c" -O $t.mp4];
}

* Sorry for the double post...

March 21, 2011, 06:18

Peteris,
Right now, i am studying Perl with a book "Minimal Perl". Its a guide to people who already know basics of scripting in Linux. The book is really amazing. Linux admins who wants to learn perl can fastly and easily learn Perl. Jus have a look at it. If u feel that its worth, recommending that in ur page wud benefit many.

Thanks,
Ashok

March 21, 2011, 06:40

Thanks, I'll look into it!

andy Permalink
August 18, 2011, 15:29

Good post!!
I also find it good to use http://www.wantyoutube.com/
It's totally free and easy to download.
It is also available for youtube, dailymotion, matecafe, and myspace, etc.

August 27, 2011, 21:31

New youtube downloader code based entirely on LWP::UserAgent

use strict;
use LWP::UserAgent;
use URI::Escape qw(uri_unescape);
my $lwp = LWP::UserAgent->new;
foreach $_ (@ARGV) {
s[.*[=/]?([\w-]{11}).*][$1];
my $c = uri_unescape(my $t = $lwp->get("http://www.youtube.com/get_video_info?&video_id=$_&el=detailpage&ps=default&eurl=&gl=US&hl=en")->content);
do { $t = uri_unescape($1); $t =~ s[\+|/][ ]g; print $t. $/ } if $t =~ /&title=([^&]+)&/;
$lwp->show_progress(1), $lwp->mirror(uri_unescape(($c =~ /map=url=([^&]+)&/)), "$t.mp4")
unless -e "$t.mp4", $lwp->show_progress(0);
}

September 22, 2011, 20:45

Update code (the above code has been broken by Youtube changes)

#!/usr/bin/perl -w

#################### Coded by TRIZEN under the GPL. ####################
use strict;use LWP::UserAgent;use URI::Escape('uri_unescape');my $lwp=new
LWP::UserAgent;foreach $_(@ARGV){($_) = m[[=/]?([\w-]{11})] or next;my $c
=uri_unescape(my $t = $lwp->get("http://www.youtube.com/get_video_info?&"
."video_id=$_&el=detailpage&ps=default&eurl=&gl=US&hl=en")->content);do{#
$t=uri_unescape $1;$t=~tr[\+/][ ];print"** $t\n"}if$t=~/&title=([^&]+)/;#
$lwp->show_progress(1),$lwp->mirror(uri_unescape(($c=~/map=url=([^&]+)&/)
),"$t.mp4")unless -e "$t.mp4";$lwp->show_progress(0)}# Youtube Downloader

September 25, 2011, 08:45

Awesome. Thanks for posting these!

September 26, 2011, 16:11

My pleasure. I came back with few shorter youtube-downloaders.

For Perl>=5.14.1
use LWP;use URI::Escape;$u=\&uri_unescape;for(@ARGV){($y,$_)=m[(.+m)/.+[=/]([\w-]{11})] or next;&$u(($l=new LWP::UserAgent)->
get("$y/get_video_info?&video_id=$_")->content)=~/ap=url=([^&]+).+&title=([^&]+)/;$l->mirror(&$u($1),&$u("$2.mp4"=~y#+/# #r))}

For all versions of Perl
use LWP;use URI::Escape;$u=\&uri_unescape;for(@ARGV){($y,$_)=m[(.+m)/.+[=/]([\w-]{11})] or next;&$u(($l=new LWP::UserAgent)->get(
"$y/get_video_info?&video_id=$_")->content)=~/ap=url=([^&]+).+&title=([^&]+)/;$l->mirror(&$u($1),do{$t=&$u($2);$t=~y#+/# #;"$t.mp4"})}

One-liner version:
perl -MLWP -MURI::Escape -e '$u=\&uri_unescape;for(@ARGV){($y,$_)=m[(.+m)/.+[=/]([\w-]{11})] or next;&$u(($l=new LWP::UserAgent)->get("$y/get_video_info?&video_id=$_")->content)=~/ap=url=([^&]+).+&title=([^&]+)/;$l->mirror(&$u($1),do{$t=&$u($2);$t=~y#+/# #;"$t.mp4"})}'

November 04, 2012, 15:32

Update:

perl -MLWP::Simple -MURI::Escape -e'$u=\&uri_unescape;for(@ARGV){($y,$_)=m[(.+m)/.+?[=/]([\w-]{11})] or next;($_=&$u(get("$y/get_video_info?video_id=$_")))=~/g=\d+&url=([^&]+).*?&sig=([^&]+)/;mirror(&$u("$1&signature=$2"),do{/tle=([^&]+)/;"$1.mp4"=~tr#/+# #r})}'

Leave a new comment

(why do I need your e-mail?)

(Your twitter name, if you have one. (I'm @pkrumins, btw.))

Type the word "security": (just to make sure you're a human)

Please preview the comment before submitting to make sure it's OK.

Advertisements