This article is part of the article series "GUI Video Downloader."
<- previous article next article ->

video downloader software, vim, mingw, wxwidgetsIn my previous posts I have shown you how to download YouTube videos and how to convert them to a better format. Wouldn't it be great to use the skills learned and create a small, cool desktop application which would download YouTube videos (and later extend to more video sites) and convert them to whatever format we want? I bet it would!

Let's get hands on it!

This will be a multi-part article on how to create this application. As the title of my blog "good coders code, great reuse" suggests, I will reuse as many software components as possible. I will use a free compiler, free gui framework and any free libraries I can find useful. Also I will try to take as many screenshots as possible to make this tutorial newbie friendly. For this reason I just installed a fresh copy of Windows XP on my VMware virtual machine (for those of you who don't know what virtual machine is, here is a quote from Intro to Virtualization: "Virtualization allows multiple virtual machines, with heterogeneous operating systems to run in isolation, side-by-side on the same physical machine.") which I will be developing the software from scratch on.

Since I am using the "good coders code, great reuse" principle, naturally I want parts of my code to be reusable. That's why I will first create a neat video downloading library in C which I will reuse later in the test command line application and the final GUI application which will probably be written in C++ using some cross-platform GUI framework/toolkit.

Part I: Preparation of the Tools

Since I am doing this project from scratch on a naked WinXP machine, the first thing I personally need need is a good browser. I will be searching for tools, libraries and will read documentation a lot, so I need a good browser. My choice - FireFox.

Next I will need a text editor to write the code in. My choice for this project - Vim. Since I want to create this project from scratch, I will need some essential plugins for vim, otherwise editing the source files would take me forever. From all the plugins I usually use in my vim configuration, for this project I'll use: a to quickly switch between .c and .h source files, bufexplorer to quickly navigate around buffers, bufpos to quickly switch between buffers, c-support to quickly write C/C++ code (it has key mapping cheat-sheet in the plugin's archive), surround to quickly change/add/insert quotes, parens etc, and taglist to navigate around source files. To use the last plugin (taglist) we also need Exuberant Ctags tool. Installation process of all these programs and plugins is straight forward.

I installed vim to "C:\program files\vim" and ctags to "C:\ctags56".

To compile the project we will also need a descent compiler. I will choose MinGW which is "a collection of freely available and freely distributable Windows specific header files and import libraries combined with GNU toolsets that allow one to produce native Windows programs that do not rely on any 3rd-party C runtime DLLs." Getting to run MinGW is not as easy as the previous tools so if you are not sure just download the automatic installer and it will do everything for you. The automatic installer can be downloaded from 'Current' package version release of MinGW at SourceForge.

I installed mingw using automated installer to "C:\mingw".

We still need a debugger. Going quickly through MinGW's files at SourceForge I couldn't find the GDB package. So I joined #mingw on FreeNode IRC network and in a few minutes someone helped me by suggesting that it was located in snapshot release. Thank you!

[17:15:49] * Now talking in #mingw
[17:16:07] <pkrumins> hi, what debugger could I use with mingw? I don't see gdb coming with any of the packages, did i miss it?
[17:20:30] <Vinky> pkrumins, snapshot->gdb

IRC is a powerful resource to get help at. Learn to use it.

Next we need a cross platform GUI toolkit. I am pretty familiar with wxWidgets and have used it in a few projects, so I will use for this project as well. I will not compile it at the moment because first I want to create a library and then the command line tool and only then the GUI app.

We are now ready to create the video downloading library. Before we do it, let's test if our development environment works.
Let's write the hello world application, compile it and run it. Click, click...

#include <stdio.h>

int main(void) {
    (void)printf("hello world\n");
    return 0;
}

We are using vim with c-support plugin, so compiling the hello world from vim is done (by looking up in the c-support plugin's keyboard mapping cheat sheet) by typing \rc and the program can be run by typing \rr. Quick and neato.

video downloader programming environment screenshot - vim, gcc, hello world

This is end of part one. In part two I will create the library and the command line test program. Until next time, kthxbye!

sed as a superman, the unix stream editorI present to you my cheat sheet of sed, the Superman of stream editing! It has come handy 101 times for me because sed is not what I use daily and after some time of not using the sed commands tend to fade away from my mind and I have to review them quickly. This cheat sheet is ideal for that!

You can't really be a good hacker if you don't know this tool. I always love to think of various situations where knowing many, many different tools makes you the ultimate guru.
Suppose you had a situation where you were on a box which had some problems and the only tool available to you was sed, and you had to fix a configuration file problem really quickly or bad, bad things would happen. If you never knew sed, you'd be in deep trouble because it can't be learned from the man page, because it only lists all the commands and you have no understanding how the commands work.

When I first learned sed, I remember the joy when I understood how it worked, that it operated on input stream, output stream, pattern space and hold buffer. Once you know this, the rest can be understood from the man page, but before that, I doubt it.

If you want to learn the sed editor I recommend this wonderful tutorial and these books.

This sed cheat sheet contains:

  • command line summary
  • command description, if they take single address or pattern, or a range of addresses, and what they modify (input stream, output stream, pattern space or hold buffer)
  • command line argument summary
  • extensions
  • short summary of adderss range format

If you notice any inaccuracies, mistakes or see that I have left something out, comment on this page, please!

Download Sed Stream Editor Cheat-Sheet

PDF:
Download link: sed stream editor cheat sheet (.pdf)
Downloaded: 134440 times

Plain Text (.txt):
Download link: sed stream editor cheat sheet (.txt)
Downloaded: 37913 times

Microsoft Word 2000 format (.doc):
Download link: sed stream editor cheat sheet (.doc)
Downloaded: 7368 times

Are you interested in sed and unix text editing power tools? Here are three great books on this topic from Amazon:

ffmpeg video converterIn the few previous posts we have been downloading YouTube videos with Awk and Perl. What we ended up with were .flv (Flash Media Video) files which do not play with the usual video players like Windows Media Player which comes with Windows out-of-the-box.

To play these .flv's we either need a special video player which supports this video file format such as VLC Media Player or FLV Player, or get a good video codec pack (which I do not recommend since it eventually messes the whole multimedia subsystem up, not gonna even link to it), or get a ffdshow all-in-one video codec (I recommend it).

The ffdshow solution is pretty good and that is what I use myself together with the excellent Media Player Classic video player which replaces the horrific Windows Media Player once and forever.

Now suppose you wanted to send the video file to your friend who didn't know about all these video codecs and players and didn't want to have them installed. What would you do?

Or suppose you wanted to put the video on your brand new iPhone or video iPod, or your cellphone.

Let's learn how to convert the video to a better format such as windows media video (.wmv), .avi, mpeg or divx, or mobile phone's 3gp format (or even others).

The tool we are going to use is called ffmpeg. ffmpeg is a command line tool to convert one video and audio file format to another.

The official homepage of the tool is at ffmpeg.mplayerhq.hu which is more often down than not. The official download page is here, alternative download here.

I went to the alternative site, downloaded the latest version (ffmpeg.rev9767.7z) and extracted it with WinRar.

The other cool thing about ffmpeg is that it is a command line tool and it's cross platfrom, so you can run it anywhere and it does not need the GUI.

Once you have it downloaded, unpack it and run the tool without any command line options:

usage: ffmpeg [[infile options] -i infile]... {[outfile options] outfile}...
Hyper fast Audio and Video encoder

Main options:
-L                  show license
-h                  show help
-version            show version
-formats            show available formats, codecs, protocols, ...
-f fmt              force format
...

Okay, from the first usage line we now know that to convert a video we will run the tool as following:

> ffmpeg -i our_flash_video.flv ... options ... our_desired_video.avi (or .mpg, or whatever)

Besides this valuable information we also get hundreds of various options which do not yet make much sense to us.
We need documentation of the tool to understand it. Quickly typing 'ffmpeg documentation' into Google we find this ffmpeg documentation page.

Before going into documentation let's first just try converting the video:

> ffmpeg.exe -i youtube_flash_video.flv out.avi
FFmpeg version SVN-r9767, Copyright (c) 2000-2007 Fabrice Bellard, et al.
  configuration: --enable-gpl --enable-pp --enable-swscaler --enable-pthreads --
enable-liba52 --enable-avisynth --enable-libamr-nb --enable-libamr-wb --enable-l
ibfaac --enable-libfaad --enable-libgsm --enable-libmp3lame --enable-libnut --en
able-libogg --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libx
vid --cpu=i686 --enable-memalign-hack --extra-ldflags=-static
  libavutil version: 49.4.1
  libavcodec version: 51.40.4
  libavformat version: 51.12.1
  built on Jul 20 2007 18:03:34, gcc: 4.2.0

Seems stream 0 codec frame rate differs from container frame rate: 1000.00 (1000
/1) -> 15.00 (15/1)
Input #0, flv, from 'youtube_flash_video.flv':
  Duration: 00:01:23.9, start: 0.000000, bitrate: 64 kb/s
  Stream #0.0: Video: flv, yuv420p, 320x240, 15.00 fps(r)
  Stream #0.1: Audio: mp3, 22050 Hz, mono, 64 kb/s
Output #0, avi, to 'out.avi':
  Stream #0.0: Video: mpeg4, yuv420p, 320x240, q=2-31, 200 kb/s, 15.00 fps(c)
  Stream #0.1: Audio: mp2, 22050 Hz, mono, 64 kb/s
Stream mapping:
  Stream #0.0 -> #0.0
  Stream #0.1 -> #0.1
Press [q] to stop encoding
frame= 1259 fps=219 q=2.0 Lsize=    2780kB time=83.4 bitrate= 272.9kbits/s
video:2050kB audio:652kB global headers:0kB muxing overhead 2.860901%

Hey, wow! It worked, we just converted our .flv to an mpeg4 video! Also the sound got converted from mp3 to mp2.

We would like to have more control over these parameters, right?

Let's go through documentation and find how to change audio and video bitrate, video framerate, video and codecs and output video size.
For various video options there is a section in documentation called 'Video Options', there we find that '-b bitrate' option changes video bitrate, '-s wxh' changes video width and height, '-vcodec codec' changes the video codec and '-r framerate' changes the video frame rate.

The audio options are listed here and we find that to change the audio bitrate we need to specify '-ab bitrate' and '-acodec codec' to change the audio codec.

The other interesting option is '-formats' which lists the supported codecs and protocols.

The videos we get from YouTube are usually 320x240 in size and at 15 fps with 64kbit/s mp3 sound.

If we did not specify the vid size, framerate and other parameters the input video's ones would be used for output video.

Just to illustrate how to use those command options, let's convert a YouTube video to 3gp format for viewing on a mobile phone. What is this 3gp format? Looking in wikipedia for 3gp we find that this format stores video streams as MPEG-4 Part 2 or H.263 and audio streams as AMR-NB, AMR-WB, AMR-WB+ or AAC-LC.
Also it turned out that H.263 can only take a number of video sizes which are listed on this page.
All we have to do is just to specify the right command line flags - and voila!

> ffmpeg.exe -i youtube_flash_video.flv -r 15 -b 200kbit/s -s 176x144 -vcodec h263 -ar 8000Hz -ab 10.2k -acodec libamr_nb for_mobile.3gp

Yeap, it worked, I uploaded the file to my Nokia N73 and it played perfectly!

To convert it to DivX and leave all the other parameters as is, use the following simple command line:

> ffmpeg.exe -i youtube_flash_video.flv -vcodec mpeg4 divx_youtube_movie.avi

To convert to other formats, just look at the supported encoding formats via '-formats' command line option. For example, to convert a video to .wmv, you would specify '-vcodec wmv2' or '-vocdec wmv1', etc.

Now we know how to download youtube videos and how to convert them. It seems that a perfect time has come to create a free, quick and cool desktop software application which will do this job for us.

Thanks for reading and until next time!

Perl√Ęs Camel and Special Variables Trying to eat him In my previous post - downloading youtube videos with a Perl one-liner - I used Perl's special variable $_. This is just one of many Perl's special variables.

One day last year I decided to go through all the Perl's special variables carefully so I could become a better Perl programmer and make a cheat sheet which I could always keep on my desk and look things up when needed.

As I explained in the previous cheat-sheet post (awk's cheat sheet), the cheat sheets are for learning things better and not doing things blindly and looking the stuff each time up. Here is another example why cheat sheets are helpful - suppose you programmed for 5 hours and then sat back in your chair to relax for 10 minutes. You could relax by taking cheat sheet in your hands and just scan through it and remember a thing or two.

This cheat sheet contains all the perl's special variables, their description and examples where possible.

It can be nicely printed on one sheet of paper by having two pages per side. Two on one side and two on other. That's how I have printed it.

Available as usual in .doc and .pdf formats.

Download Perl's Special Variable Cheat-Sheet

PDF:
Download link: perl's special variable cheat sheet (.pdf)
Downloaded: 108697 times

Microsoft Word 2000 format (.doc):
Download link: perl's special variable cheat sheet (.doc)
Downloaded: 6089 times

Are you interested in Perl programming language? Here are three excellent books on Perl from Amazon (recommended by me):

purl purl, youtube perl one liner downloaderLast time I explained how YouTube videos can be downloaded with gawk programming language by getting the YouTube page where the video is displayed and finding out how the flash video player retrieves the FLV (flash video) media file.

This time I'll use Perl programming language which is my favorite language at the moment and write a one-liner which downloads a YouTube video.

Instead of parsing the YouTube video page, let's look how an embedded YouTube video player on a 3rd party website gets the video.

Let's go to this cool video and look at the embed html code:

html code for embedded youtube video

For this video it looks as following:

<object width="425" height="350"><param name="movie" value="http://www.youtube.com/v/qg1ckCkm8YI"></param><param name="wmode" value="transparent"></param><embed src="http://www.youtube.com/v/qg1ckCkm8YI" type="application/x-shockwave-flash" wmode="transparent" width="425" height="350"></embed></object>

95% of this code is boring, the only interesting part is this URL:

http://www.youtube.com/v/qg1ckCkm8YI

Let's load this in a browser, and as we do it, we get redirected to some other URL:

http://www.youtube.com/jp.swf?video_id=qg1ckCkm8YI&eurl=&iurl=http://img.youtube.com/vi/qg1ckCkm8YI/default.jpg&t=OEgsToPDskJCPW5DvMKeM3srnQ5e0LSY

So far we have no information how the flash player will retrieve the video, the only thing we know that 'iurl' stands for 'image url' and is the location of the thumbnail image.

Let's sniff the traffic again, this time with an excellent (though, commercial) Internet Explorer plugin 'HttpWatch Professional'.
This plugin displays all the requests the browser makes no matter if it's HTTP or HTTPS traffic and displays in a nice manner which makes our job much quicker than by using Ethereal.
The FireFox's alternative to this tool is Live HTTP Headers extension which basically does the same as HttpWatch Professional but it takes more time to understand the output.

Here is what we see with HttpWatch Professional when we load the URL in the browser:

output of httpwatch professional sniffer (sane thumbnail (not the wordpress insanity))

We see that to get a video browser first requested:

http://www.youtube.com/get_video?video_id=qg1ckCkm8YI&t=OEgsToPDskJ3bp4DEiMuxUmjx7oumUec&eurl=

then got redirected to:

http://cache.googlevideo.com/get_video?video_id=qg1ckCkm8YI

and then another time to:

http://74.125.13.83/get_video?video_id=qg1ckCkm8YI

This is exactly what what we saw in the previous article on downloading videos with gawk!

Now let's write a Perl one-liner that retrieves this video file!

What is a one-liner you might ask? Well, my definition of one liner is that it is a program you are willing to type out without saving it to disk.

First of all we will need some perl packages (modules) which will ease working with HTTP protocol. There are two widely used available on Perl's module archive (CPAN) - LWP and WWW::Mechanize.

WWW::Mechanize is built on top of LWP, so let's go to a higher level of abstraction and use this module.

The WWW::Mechanize package does not come as Perl's core package by default, so you'll have to get it installed.
To do it, type

perl -MCPAN -eshell

In your console and when the CPAN shell appears, type

install WWW::Mechanize

to get the module installed.

If everything goes fine, the CPAN will tell you that the module got installed.

I don't want to go into Perl language's details again, also I don't want to go into WWW::Mechanize package's details.

If you want to learn Perl I recommend this article as a starter, these books and of course perldoc. Once you learn the basics you can quickly pick up the WWW::Mechanize package by reading the documentation, faq and trying examples.

Now finally let's write the one-liner. So what do we have to do?

First we have to retrieve

http://www.youtube.com/v/qg1ckCkm8YI

then follow the redirect (which WWW::Mechanize will do for us), then get the 't' identifier from query string and finally request and save output of

http://www.youtube.com/get_video?video_id=qg1ckCkm8YI&t=OEgsToPDskJ3bp4DEiMuxUmjx7oumUec&eurl=

That's it!

So here is the final version which can probably be made even shorter:

perl -MWWW::Mechanize -e '$_ = shift; s#http://|www\.|youtube\.com/|watch\?|v=|##g; $m = WWW::Mechanize->new; ($t = $m->get("http://www.youtube.com/v/$_")->request->uri) =~ s/.*&t=(.+)/$1/; $m->get("http://www.youtube.com/get_video?video_id=$_&t=$t", ":content_file" => "$_.flv")'

A little longer than a usual one-liner but does the job nicely. To keep it short, there is no error checking!

To use this one-liner just copy it to command line and specify the URL of a YouTube video (or just the ID of the video, or a variation of URL (like without 'http://'). Like this:

perl -MWWW::Mechanize -e '...' http://www.youtube.com/watch?v=l69Vi5IDc0g

or just

perl -MWWW::Mechanize -e '...' l69Vi5IDc0g

Let's spread this one liner to multiple lines and see what it does as it is not documented.
One could do the spreading out to multiple lines by hand, but that's not what humans are for, let's make Perl do it. By adding -MO=Deparse to the command line list we get the output of the Perl generated source code (i added line numbers myself):

use WWW::Mechanize;
1) $_ = shift @ARGV;
2) s[http://|www\.|youtube\.com/|watch\?|v=|][]g;
3) $m = 'WWW::Mechanize'->new;
4) ($t = $m->get("http://www.youtube.com/v/$_")->request->uri) =~ s/.*&t=(.+)/$1/;
5) $m->get("http://www.youtube.com/get_video?video_id=$_&t=$t", ':content_file', "$_.flv");

So our one liner is actually 5 lines.
On line 1 we put the first argument of ARGV variable into special variable $_ so we could use advantage of it and save some typing.
On line 2 we just leave the ID of the video by removing parts from the URL one by one so a user could specify the video URL in various formats like 'www.youtube.com/watch?v=ID, or just 'youtube.com?v=ID' or just 'v=ID' or even just 'ID'. The ID gets stored in the special $_ variable.
On line 3 we create a WWW::Mechanize object we are going to use twice.
Line 4 needs more explanation because we are doing so much in it. First it retrieves that embedded video URL I talked about earlier, the server actually redirects us away, so we have to look at the last request's location. We save this location into variable $t and then extract the 't' YouTube ID out.
As a YouTube video is uniquely specifed with two IDs, the video ID and 't' ID, on line 5 we retrieve the file and tell WWW::Mechanize to save contents to the ID.flv file. WWW::Mechanize handles redirects for us so everything should work. Indeed, I tested it out and it worked.

Can you golf it shorter? :)

I golfed it a little myself, here is what I came up with:

perl -MWWW::Mechanize -e '$_ = shift; ($y, $i) = m#(http://www\.youtube\.com)/watch\?v=(.+)#; $m = WWW::Mechanize->new; ($t = $m->get("$y/v/$i")->request->uri) =~ s/.*&t=(.+)/$1/; $m->get("$y/get_video?video_id=$i&t=$t", ":content_file" => "$i.flv")'

To use this one liner you must specify the full URL to youtube video, like this one:

http://www.youtube.com/watch?v=l69Vi5IDc0g

This one liner saves the "http://www.youtube.com" string in variable $y and the ID of the video in variable $i. The $y comes handy because we don't have to use the full YouTube URL, instead we use use $y.

Update 2009.12.05: YouTube has changed the way it displays videos several times! The current one-liner is here:

perl -MWWW::Mechanize -e '$m = WWW::Mechanize->new; $_=shift; ($i) = /v=(.+)/; s/%(..)/chr(hex($1))/ge for (($u) = $m->get($_)->content =~ /l_map": .+(?:%2C)?5%7C(.+?)"/); print $i, "\n"; $m->get($u, ":content_file" => "$i.flv")'

Also, are you interested in Perl programming language? Here are three excellent books on Perl from Amazon (recommended by me):