This article is part of the article series "GUI Video Downloader."
<- previous article next article ->

video downloader software, vim, mingw, wxwidgetsThis is a continuation of multi-part series on how to create a cross-platform desktop video downloading and converting application. In part I we prepared our coding environment with vim, various vim plugins, mingw and gdb. In this part we will start creating a C library for downloading just the YouTube videos with modularity in mind, so that we could create a nice desktop application capable of downloading videos from other video sharing sites like break, metacafe, dailymotion and others. Complementary to this library we will be also be creating a command line application which will use this C library. The command line application will serve us as a test tool for the library, so we knew the library worked ok. Once we have these two components, we will build GUI around the library and create the desktop app.

Writing the Library

I have chosen a name libvd for the library which stands for video download library.

The library will be written in C programming language because:

  • it is the number one cross platform language,
  • I haven't programming in C for a while, so it will be a great refresher,
  • the GUI framework which I will be using is written in C++ which will have no problems with using C code
  • the application will be small and will require no unnecessary library/toolkit dependencies

Downloading a video means working with HTTP protocol. Let's use "good coders code, great reuse" approach and seek for a C library which would ease the usage of HTTP protocol.

Having worked on a few projects in C programming language involving HTTP protocol, I know an excellent library. It's called libcurl. I even implemented cookie interface for this library which was missing two years ago.

What is libcurl? Quoting the author of libcurl: "libcurl is a free and easy-to-use client-side URL transfer library, supporting FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, FILE and LDAP. libcurl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, Kerberos4), file transfer resume, http proxy tunneling and more!

libcurl is highly portable, it builds and works identically on numerous platforms, including Solaris, NetBSD, FreeBSD, OpenBSD, Darwin, HPUX, IRIX, AIX, Tru64, Linux, UnixWare, HURD, Windows, Amiga, OS/2, BeOs, Mac OS X, Ultrix, QNX, OpenVMS, RISC OS, Novell NetWare, DOS and more..."

Woah! Much more than we need! And notice how cross platform it is! Exactly what we are looking for!

Let's download the libcurl library. Since I am developing the program on a fresh Windows XP virtual machine, I chose to download curl-7.16.4.zip version of the library. It really does not matter what operating system I am developing this program on, I could have as well chosen Linux, then I would have downloaded the library ending with extension .gz or .bz2.

download libcurl for windows

Compiling libcurl

Once we have downloaded libcurl, let's unpack it and look for instructions on how to compile it with MinGW on a Windows machine. I chose c:\vd as the root directory for the project on the windows machine because I am afraid of those whitespace characters windows has in path. I unpacked the libcurl library to c:\vd\curl.

At this point we have no idea how to build the libcurl on Windows, right? So let's look in the docs subdirectory (C:\vd\curl\docs) for a clue. Usually the instructions on how to build a library on a particular platform can be found in files like README.platform or INSTALL.platform or just INSTALL. Indeed, libcurl has a file README.win32. Let's read it.

libcurl's win32 readme file

Oops, it says to read README first, we look in the same docs subdirectory but there is no README! We look in the parent directory, whew, yeah there is a README. But it is just 1KB in size and says nothing...

The only reasonable file left is INSTALL, let's look at it. Oh yeah, it's 29K in size! Must be it!

Indeed, scrolling a few screens down by pressing vim's Ctrl_D, we find instructions on how to build libcurl on win32 with MinGW compiler!

libcurl's win32 install file with mingw

Libcurl also provides support for compressed HTTP content with zlib, and secure connections over HTTPS with OpenSSL. The INSTALL file mentions what to do to get support for these thingies. We do not need them, so we ignore them.

Don't rush yet compiling the library! We also want our application to be small, not just portable. Libcurl has support for so many protocols but our project need just the HTTP protocol!

Looking a little further in the INSTALL file, we find the following instruction:

   Disabling Specific Protocols in Win32 builds
   --------------------------------------------

   The configure utility, unfortunately, is not available for the Windows
   environment, therefore, you cannot use the various disable-protocol
   options of the configure utility on this platform.

   However, you can use the following defines to disable specific
   protocols:

   HTTP_ONLY             disables all protocols except HTTP
   [...]

   If you want to set any of these defines you have the following
   possibilities:

   - Modify lib/setup.h
   [...]

We really want just the HTTP protocol, so let's follow this instruction and modify setup.h file in lib directory (file C:\vd\curl\lib\setup.h) and add the following line right after the comments:

#define HTTP_ONLY

The file should now look like this:

modifying libcurl to make it support HTTP only!

Okay, now we are ready to compile libcurl!

The INSTALL file says that to compile the libcurl library we first need to run 'mingw32.bat' and then 'make mingw32'.

   MingW32
   -------

   Run the 'mingw32.bat' file to get the proper environment variables set,
   then run 'make mingw32' in the root dir. Use 'make mingw32-ssl' to build
   curl SSL enabled.

mingw32.bat? I don't have such file in my mingw32 distribution! Now what? Let's be real developers and compile the library ourselves then. Let's make sure the MinGW's bin directory (in my case C:\MinGW\bin) is in our PATH environment variable and let's run the second command - 'make mingw32'.

Oops:

C:\vd\curl>make mingw32
'make' is not recognized as an internal or external command,
operable program or batch file.

There is no program named 'make' in MinGW's distribution, there is only mingw32-make.exe, let's run it:

C:\vd\curl>c:\MinGW\bin\mingw32-make.exe mingw32
c:/MinGW/bin/mingw32-make -C lib -f Makefile.m32 ZLIB=1
mingw32-make[1]: Entering directory `C:/vd/curl/lib'
gcc -I. -I../include -I"../../zlib-1.2.3" -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -DHAVE_LIBZ -DHAVE_ZLIB_H -c file.c
gcc -I. -I../include -I"../../zlib-1.2.3" -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -DHAVE_LIBZ -DHAVE_ZLIB_H -c timeval.c
gcc -I. -I../include -I"../../zlib-1.2.3" -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -DHAVE_LIBZ -DHAVE_ZLIB_H -c base64.c
In file included from base64.c:43:
urldata.h:92:59: zlib.h: No such file or directory
In file included from base64.c:43:

Huh? I never wanted my libcurl to be built with zlib support. Let's get rid of it by editing the Makefile (C:\vd\curl\Makefile) directly. Search for mingw32 target (string "mingw32:") and you'll find:

mingw32:
	$(MAKE) -C lib -f Makefile.m32 ZLIB=1
	$(MAKE) -C src -f Makefile.m32 ZLIB=1

We really don't want to build the curl executable, so let's get rid of the last line and also get rid of "ZLIB=1" variable definition.
The lines should look now like this:

mingw32:
	$(MAKE) -C lib -f Makefile.m32

Now let's run the mingw32-make again:

C:\vd\curl>c:\MinGW\bin\mingw32-make.exe mingw32
c:/MinGW/bin/mingw32-make -C lib -f Makefile.m32
mingw32-make[1]: Entering directory `C:/vd/curl/lib'
gcc -I. -I../include -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -c base64.c
gcc -I. -I../include -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -c hostip.c
[...]
rm -f libcurl.a
process_begin: CreateProcess(NULL, rm -f libcurl.a, ...) failed.
make (e=2): The system cannot find the file specified.
mingw32-make[1]: *** [libcurl.a] Error 2
mingw32-make[1]: Leaving directory `C:/vd/curl/lib'
mingw32-make: *** [mingw32] Error 2

Argh! There is no 'rm' on Windows, at least not with MinGW development tools package. Let's modify the Makefile.m32 file (C:\vd\curl\lib\Makefile.m32) and change 'rm' to Windows 'del' command.

Find this line in Makefile.m32:

RM = rm -f

and change it to

RM = del /F

Perfect! Running the mingw32-make.exe again finishes with no errors! Lookin in the C:\vd\curl\lib directory we find the static library libcurl.a and the dynamic library libcurl.dll. Had we compiled it on Linux, the dynamic library would have been named libcurl.so.

Testing libcurl

I wrote the following test program which outputs the website of google.com to the standard output if libcurl works correctly:

#include <stdio.h>
#include <stdlib.h>
#include "curl/curl.h"

int main(void) {
    CURLcode ret;
    CURL *curl = curl_easy_init();
    if (curl == NULL) {
        fprintf(stderr, "Failed creating CURL easy handle!\n");
        exit(EXIT_FAILURE);
    }

    /* let's get google.com because I love google */
    ret = curl_easy_setopt(curl, CURLOPT_URL, "http://www.google.com");
    if (ret != CURLE_OK) {
        fprintf(stderr, "Failed getting http://www.google.com: %s\n",
                curl_easy_strerror(ret));
        exit(EXIT_FAILURE);
    }

    ret = curl_easy_perform(curl);
    if (ret != 0) {
        fprintf(stderr, "Failed getting http://www.google.com: %s\n",
                curl_easy_strerror(ret));
        exit(EXIT_FAILURE);
    }

    return 0;
}

I placed this program in C:\vd\tests\curltest directory and named the source file testcurl.c. Our curl library is located in C:\vd\curl\lib and the curl header files in C:\vd\curl\include. Assuming out currenly working directory is C:\vd\tests\curltest, the following command line will compile the C source file to an executable:

gcc testcurl.c -o testcurl -DCURL_STATICLIB -I..\..\curl\include -L..\..\curl\lib -lcurl -lwsock32 -lwinmm -Wall

It will create testcurl.exe executable which can be run. Let's try running it:

C:\vd\tests\libcurl>testcurl.exe
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>Google</title>
[...]

Yeah! It works! Now we are ready to write the libvd (video download) library which will use the libcurl for communicating over HTTP protocol.

I'll do this in Part III of this series. Until then, have fun compiling libcurl!

ed text editor and mr. edI have always had historic interest of various UNIX tools and the UNIX itself. One day when I was learning sed, the stream editor, (sed cheat sheet) I got interested how much the most powerful text editing tools on UNIX - 'ed', 'ex', 'sed' and 'vim' (or just 'vi') had in common and what they borrowed from one another.

I was not perfectly sure if 'sed' has originated from 'ed', so I Google searched for 'sed history' which turned up some interesting results, one of them being "The original users manual for sed, by Lee E. McMahon, from the 7th edition UNIX Manual (1978)".

Quoting the manual "Sed is a lineal descendant of the UNIX editor, ed." Great! I knew that 'sed' was non-interactive text editor whereas 'ed' was an interactive one. Also I knew that 'ex' was improved 'ed' and that 'sed' was a non-interactive 'ed' and that 'ex' had become 'vi' and was used in 'vi' command line mode. To boost my vi/vim skills and get a touch on how text editing was done with just 'ed', and satisfy my historical curiosity I set myself a goal to learn it.

As I mentioned in awk and sed cheat sheet posts, the best method for me to learn a new tool is to have all commands in front of me, so when I am doing experimentation I can quickly find the command and also see all the other commands at the same time, blowing them in subconsciousness.

Since 'ed' is line-oriented text editor, it is important to understand the line addressing, this cheat sheet summarizes the line addressing.

Most of 'ed' command are single letter in length. This cheat sheet summarizes all the commands, showing the supported address ranges and a detailed description of each command.

From my experience, once I had completed this cheat sheet and had it in front of me, I picked 'ed' up in 30 minutes. And then spent a few more hours experimenting and trying various constructs.

Good luck with learning ed! :)

And don't forget: ED IS THE TRUE PATH TO NIRVANA!

ps. if you notice any inaccuracies, spelling or grammar errors, please leave a comment here, so I can fix it. Thanks!
pss. logo for this article from mr. ed comedy ;)

Download Ed UNIX Text Editor's Cheat-Sheet

Plain Text (.txt):
Download link: ed text editor cheat sheet (.txt)
Downloaded: 19786 times

PDF:
Download link: ed text editor cheat sheet (.pdf)
Downloaded: 32591 times

Microsoft Word 2000 format (.doc):
Download link: ed text editor cheat sheet (.doc)
Downloaded: 3199 times

upload youtube perlWe have been downloading and converting YouTube videos but have not looked at how to upload them.

I will teach you how to upload YouTube videos via a command line. To find out how a video is being uploaded we will need just the Firefox browser!

Here is a typical scenario when you would want to automatically upload your videos to YouTube. Suppose you were using some other video sharing site and had already uploaded like 100 of your videos there. To get more popular on the net you'd also want to get your videos on YouTube, right? Doing it manually is a boring and tedious job, you want a program to do it for you while you sit back and relax.

Finding the YouTube Upload Form's Elements

Log into your account and go to "My Account" menu in the upper right. Then press the "Upload New Video" button.

how to upload youtube videos

Now let's use Firefox's "Page Info" tool which is located under "Tools" menu.

firefox's page info built in tool

When the "Page Info" tool pops up, select the "Forms" tab. The tool will list all the HTML forms on the page. The one named "theForm" with action's url "http://www.youtube.com/my_videos_upload" is the upload form! When a user uploads the video, it gets submitted. Our program will submit this form for us.

youtube upload video page's forms, fields, values

Now that we have found all the fields we need to submit, we can write the tool itself. I'll use my favorite programming language - Perl, again. It is perfect for this job because it has extremely good packages for working with HTTP protocol.
I will probably write another post just about the Perl programming language - how to get it working on Windows Operating System. For now, if you are running Windows, then you can download ActiveState's ActivePerl which is Perl's port to win.

Writing the Program

As I said, we will be creating the tool in Perl programming language. In the previous post about YouTube I used the WWW::Mechanize package. I can tell you in advance that it will not work this time because there will be some unexpected surprises, such as hidden form elements being created by Javascript when the form is submitted. Since WWW::Mechanize stores all the forms internally, it will not be able to submit this form with these dynamic form elements. That's why we will use the base class of WWW::Mechanize - the LWP::UserAgent.

LWP::UserAgent package can be installed with the following command:

perl -MCPAN -e 'install LWP::UserAgent'

Once you have it installed, we can start writing the program.

Step 1: Logging in to YouTube

Before we can upload a video, we need to login to YouTube. Let's use the same procedure as before to find out what form do we need to submit to login. Let's go to this url:

http://www.youtube.com/login

and look at all the forms on this page. One of them is named "loginForm" with four fields. We will need to submit this form and save the cookies. Luckily LWP::UserAgent can do it for us and we will not have to parse HTTP headers or anything like that.

YouTube's login form fields, types and their values

We see that there is no action URL provided for the loginForm, that means that the form should be POSTed to the same URL it was located on, in our case:

http://www.youtube.com/login

We also see that the form has 4 elements which should be submitted:

  • 'current_form' with value 'loginForm'
  • 'username' with our youtube login/username
  • 'password' with our password, and
  • 'action_login' with value 'Log In'

How would we submit this form with LWP::UserAgent? Easy - it has a member function $ua->post( $url, \%form, $header_field_name => $value, ... ), where %form is a hash of key, value pairs of form elements and $header_field_name => $value is a pair of HTTP headers we would like to add to the request.

Given an object of LWP::UserAgent named $ua, code which would login us to YouTube is the following:

$ua->post('http://www.youtube.com/login',
    {
        current_form => 'loginForm',
        username     => 'youtube_login',
        password     => 'youtube_password',
        action_login => 'Log In'
    },
);

See how easy it was? I love it.

The only problem is how to detect if we have really logged in. I noticed that when we log in, the upper right menu has a 'Log Out' button. So if we logged in succesfully, we should find this button.

Step 2: Uploading the Video

Now, once we have logged in, let's upload the video.

Here is the image with all the fields we have to submit to upload a video, once more:

youtube upload video page's forms, fields, values

There are 17 fields totally:

  • "field_myvideo_title" - title of the video
  • "field_myvideo_descr" - description of the video
  • "field_myvideo_keywords" - comma separated tags
  • "field_myvideo_categories" - video category
  • "language" - language of the video, i'll leave it "EN" for English
  • "action_upload" - a string "Upload a video..."
  • "session_token" - we will have to extract the session token for this upload
  • "allow_embeddings" - should the video be allowed to be embedded on other sites, i'll leave the default "Yes"
  • "allow_responses" - should we allow video responses, i'll leave the default "Yes"
  • "allow_comments" - should we allow comments on the video, i'll leave the default "Yes"
  • "allow_ratings" - should we allow users to rate the vide, i'll leave the default "Yes"
  • "location" - latitude and longitude of the location video was shot, i'll leave it empty
  • "field_date_mon" - month the video was shot, i'll use default 0 (unknown)
  • "field_date_day" - day the video was shot, i'll use 0 (unknown)
  • "field_date_yr" - year the video was shot, i'll use 0 (unknown)
  • "field_privacy" - privacy of video (public or private), i'll use the default - "public"
  • "ignore_broadcast_settings" - no idea what this is, i'll use the default value 0

The trickiest part here is to extract the "session_token" which is some kind of unique id.
Going through the HTML source of upload page:

http://www.youtube.com/my_videos_upload

We find a javascript function called "dynamic_append_session_token" which dynamically creates a hidden form element containing the session ID:

finding the session id of youtube upload form

We have to extract this session ID from the line in red. It can be done with the following regular expression:

token_elem\.setAttribute\('value', '(.+?)'\);

Update 2008.03.12: This is no longer true! A session cookie is represented by a variable now:

var gXSRF_token = 'OTxXiSg8O-oQFB-PWTSaDv7oX2V8MTIwNTQ1NzYxNQ==';

When we have done this, we can submit the form. For this form we will need to add a "Content-Type: form-data" header to the request because the upload form has explicitly stated that its "enctype" is "multipart/form-data".

As it turns out the actual video upload process is a two step process. This was the first step which set the video information. After we submit this form, we get redirected to a page which contains the upload file selection form. It might seem that that is the only element in this form but not so. Actually this 2nd step form has a few new fields and it also has all the fields from 1st step as hidden elements. Here are the modified and added fields in the 2nd step:

  • "contact" - no idea what this field is for
  • "field_command" - always "myvideo_submit"
  • "field_uploadfile" - our video file we want to upload
  • "field_private_share_entities" - no idea
  • "action_upload" - submit button's caption "Upload Video"
  • "addresser" - another mysterious unique id

Here is what page info tool shows us about 2nd step's form elements:

youtube upload video 2nd step

Notice that the form's action URL changes, we have to extract it from the html source. Someone might suggest not to use regexes for parsing HTML but it is perfectly ok for this tiny project.

The trickiest thing here is telling LWP::UserAgent that 'field_uploadfile' is a form element of 'file' type. If we look in the LWP::UserAgent's documentation we find that the three argument post() subroutine uses "POST() function from HTTP::Request::Common to build the request." Further looking in the HTTP::Request::Common documentation we find that to specify a file to be uploaded the field has to be an array ref looking like this: [ $file, $filename, Header => Value... ].

That's it. When we submit this form the video should have been uploaded. Now we just need to detect if the upload was successful. I noticed that when the video has uploaded successfully, YouTube thanks me for uploading. It looks like this:

youtube video upload complete

So checking if a video uploaded successfully is reduced to checking for the thank you message.

The program can be used from command line and takes a few mandatory arguments:

Usage: perl ytup.pl -l [login] -p [password] -f <video file> -c <category> -t <title> -d <description> -x <comma, separated, tags>

The login (-l) and password (-p) arguments are not mandatory because not to expose your password on a publically shared machine (by looking at process list) they can be specified in the source file by changing YT_LOGIN and YT_PASS constants.

Possible categories (for -c switch):
2    - Autos & Vehicles
23   - Comedy
24   - Entertainment
1    - Film & Animation
20   - Gadgets & Games
26   - Howto & DIY
10   - Music
25   - News & Politics
22   - People & Blogs
15   - Pets & Animals
17   - Sports
19   - Travel & Places

Here is the final code:

#!/usr/bin/perl 
#
# Version 1.0: 2007.07.30
# Version 1.1: 2007.10.12: youtube changed html
# Version 1.2: 2008.03.13: youtube changed html
# Version 1.3: 2009.12.02: youtube changed html and now logins with google login
#
# Peteris Krumins (peter@catonmat.net)
# http://www.catonmat.net  --  good coders code, great reuse
#

use constant VERSION => "1.3";

use strict;
use warnings;

use LWP::UserAgent;
use HTML::Entities 'decode_entities';
use Getopt::Std;
$Getopt::Std::STANDARD_HELP_VERSION = 1;

# debug?
use constant DEBUG => 0;

# set these values for default -l (login) and -p (pass) values
#
use constant YT_LOGIN => "";
use constant YT_PASS  => "";

# video categories
#
my %cats = (
    2   =>  'Autos & Vehicles',
    23  =>  'Comedy',
    27  =>  'Education',
    24  =>  'Entertainment',
    1   =>  'Film & Animation',
    20  =>  'Gaming',
    26  =>  'Howto & Style',
    10  =>  'Music',
    25  =>  'News & Politics',
    29  =>  'Nonprofits & Activism',
    22  =>  'People & Blogs',
    15  =>  'Pets & Animals',
    28  =>  'Science & Technology',
    17  =>  'Sports',
    19  =>  'Travel & Places'
);

# various urls
my $login_url      = 'http://www.youtube.com/login';
my $login_post_url = 'https://www.google.com/accounts/ServiceLoginAuth?service=youtube';
my $login_cont_url = 'http://www.youtube.com/';
my $upload_url     = 'http://upload.youtube.com/my_videos_upload?restrict=html_form';
my $upload_video_url1 = 'http://www.youtube.com/gen_204?a=multi_up_queue&si=%SI%&uk=%UK%&ac=1&gbe=1&fl=0&b=0&fn=0&ti=1&d=0&ta=0&pv=0&c=0&m=0&t=0&ft=0&dn=upload.youtube.com&fe=scotty&ut=html_form';
my $upload_video_url2 = 'http://upload.youtube.com/upload/rupio';
my $upload_video_url3 = 'http://www.youtube.com/gen_204?a=multi_up_start&si=%SI%&uk=%UK%&ac=1&gbe=1&fl=0&b=0&fn=0&ti=1&d=0&ta=0&pv=0&c=0&m=0&t=0&ft=0&dn=upload.youtube.com&fe=scotty&ut=html_form';
my $upload_video_url4 = 'http://www.youtube.com/gen_204?a=multi_up_finish&si=%SI%&uk=%UK%&ac=1&gbe=1&fl=0&b=0&fn=0&ti=1&d=0&ta=0&pv=0&c=0&m=0&t=23001&ft=0&dn=upload.youtube.com&fe=scotty&ut=html_form';
my $upload_video_set_info = 'http://upload.youtube.com/my_videos_upload_json';


unless (@ARGV) {
    HELP_MESSAGE();
    exit 1;
}

my %opts;
getopts('l:p:f:c:t:d:x:', \%opts);

# if -l or -p are not given, try to use YT_LOGIN and YT_PASS constants
unless (defined $opts{l}) {
    unless (length YT_LOGIN) {
        preamble();
 print "Youtube username/login as neither defined nor passed as an argument\n";
        print "Use -l switch to specify the username\n";
        print "Example: -l joe_random\n";
        exit 1;
    }
    else {
        $opts{l} = YT_LOGIN;
    }
}

unless (defined $opts{p}) {
    unless (length YT_PASS) {
        preamble();
        print "Password was neither defined nor passed as an argument\n";
        print "Use -p switch to specify the password\n";
        print "Example: -p secretPass\n";
        exit 1;
    }
    else {
        $opts{p} = YT_PASS;
    }
}

unless (defined $opts{f} && length $opts{f}) {
    preamble();
    print "No video file was specified\n";
    print "Use -f switch to specify the video file\n";
    print 'Example: -f "C:\Program Files\movie.avi"', "\n";
    print 'Example: -f "/home/pkrumins/super.cool.video.wmv"', "\n";
    exit 1;
}

unless (-r $opts{f}) {
    preamble();
    print "Video file is not readable or does not exist\n";
    print "Check the permissions and the path to the file\n";
    exit 1;
}

unless (defined $opts{c} && length $opts{c}) {
    preamble();
    print "No video category was specified\n";
    print "Use -c switch to set the category of the video\n";
    print "Example: -c 20, would set category to Gadgets & Games\n\n";
    print_cats();
    exit 1;
}

unless (defined $cats{$opts{c}}) {
    preamble();
    print "Category '$opts{c}' does not exist\n\n";
    print_cats();
    exit 1;
}

unless (defined $opts{t} && length $opts{t}) {
    preamble();
    print "No video title was specified\n";
    print "Use -t switch to set the title of the video\n";
    print 'Example: -t "Super Cool Video Title"', "\n";
    exit 1;
}

unless (defined $opts{d} && length $opts{d}) {
    preamble();
    print "No video description was specified\n";
    print "Use -d switch to set the description of the video\n";
    print 'Example: -d "The coolest video description"', "\n";
    exit 1;
}

unless (defined $opts{x} && length $opts{x}) {
    preamble();
    print "No tags were specified\n";
    print "Use -x switch to set the tags\n";
    print 'Example: -x "foo, bar, baz, hacker, purl"', "\n";
    exit 1;
}

# tags should be at least two chars, can't be just numbers
my @tags = split /,\s+/, $opts{x};
my @filtered_tags = grep { length > 2 && !/^\d+$/ } @tags;
unless (@filtered_tags) {
    preamble();
    print "Tags must at least two chars in length and must not be numeric!\n";
    print "For example, 'foo', 'bar', 'yo29' are valid tags, but ";
    print "'22222', 'hi', 'b9' are invalid\n";
    exit 1;
} 

$opts{x} = join ', ', @filtered_tags;
# create the user agent, have it store cookies and
# pretend to be a cool windows firefox browser
my $ua = LWP::UserAgent->new(
    cookie_jar => {},
    agent => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.4) '.
             'Gecko/20070515 Firefox/2.0.0.4'
);

if (DEBUG) {
    $ua->add_handler("request_send",  sub { print "Request: \n"; shift->dump; print "\n"; return });
    $ua->add_handler("response_done", sub { print "Response: \n"; shift->dump; print "---\n"; return });
}

# let the user agent follow redirects after a POST request
push @{$ua->requests_redirectable}, 'POST';

print "Logging in to YouTube...\n";
login();

print "Uploading the video ($opts{t})...\n";
upload();

print "Done!\n";

sub login {
    # go to login page to get redirected to google sign in page
    my $res = $ua->get($login_url);
    unless ($res->is_success) {
        die "Failed going to YouTube's login URL: ", $res->status_line;
    }

    # extract GALX id
    my $GALX = extract_field($res->content, qr/name="GALX".+?value="([^"]+)"/s);
    unless ($GALX) {
        die "Failed logging in. Unable to extract GALX identifier from Google's login page.";
    }

    # submit the login form
    $res = $ua->post($login_post_url,
        {
            ltmpl    => 'sso',
            continue => 'http://www.youtube.com/signin?action_handle_signin=true&nomobiletemp=1&hl=en_US&next=%2F',
            service  => 'youtube',
            uilel    => 3,
            hl       => 'en_US',
            GALX     => $GALX,
            Email    => $opts{l},
            Passwd   => $opts{p},
            rmShown  => 1,
            signIn   => 'Sign in',
            asts     => ''
        }
    );
    unless ($res->is_success) {
        die "Failed logging in: failed submitting login form: ", 
            $res->status_line;
    }

    # Get the meta http-equiv="refresh" url
    my $next_url = extract_field($res->content, qr/http-equiv="refresh" content="0; url='(.+?)'/);
    unless ($next_url) {
        die "Failed logging in. Getting next url from http-equiv failed.";
    }
    $next_url = decode_entities($next_url);

    $res = $ua->get($next_url);
    unless ($res->is_success) {
        die "Failed logging in. Extraction of next_url failed: ",
            $res->status_line;
    }

    $res = $ua->get($login_cont_url);
    unless ($res->is_success) {
        die "Failed logging in. Navigation to YouTube.com failed: ", 
            $res->status_line;
    }

    # We have no good way to check if we really logged in.
    # I found that when you have logged in the upper right menu changes
    # and you have access to 'Sign Out' option.
    # We check for this string to see if we have logged in.
    unless ($res->content =~ />Sign Out</) {
        die "Failed logging in: username/password incorrect";
    }
}

sub upload {
    # upload is actually a multistep process
    #
    
    # get upload page to extract some gibberish info
    my $resp = $ua->get($upload_url);
    unless ($resp->is_success) {
        die "Failed getting $upload_url: ", $resp->status_line;
    }

    my $SI = extract_field($resp->content, qr/"sessionKey": "([^"]+)"/);
    unless ($SI) {
        die "Failed extracting sessionKey. YouTube might have redesigned!";
    }

    my $UK = extract_field($resp->content, qr/"uploadKey": "([^"]+)"/);
    unless ($UK) {
        die "Failed extracting uploadKey. YouTube might have redesigned!";
    }

    my $session_token = extract_field($resp->content, qr/"session_token" value="([^"]+)"/);
    unless ($session_token) {
        die "Failed extracting session_token. YouTube might have redesigned!";
    }

    prepare_upload_urls($SI, $UK);

    # tell the server that we are up for uploads
    $resp = $ua->get($upload_video_url1);
    unless ($resp->is_success) {
        die "Failed getting upload_video_url1: ", $resp->status_line;
    }

    # now lets post some more gibberish data
my $post_data_gibberish =<<EOL;
{"protocolVersion":"0.7","createSessionRequest":{"fields":[{"external":{"name":"file","filename":"$opts{f}","formPost":{}}},{"inlined":{"name":"return_address","content":"upload.youtube.com","contentType":"text/plain"}},{"inlined":{"name":"upload_key","content":"$UK","contentType":"text/plain"}},{"inlined":{"name":"action_postvideo","content":"1","contentType":"text/plain"}},{"inlined":{"name":"live_thumbnail_id","content":"$SI.0","contentType":"text/plain"}},{"inlined":{"name":"parent_video_id","content":"","contentType":"text/plain"}},{"inlined":{"name":"allow_offweb","content":"True","contentType":"text/plain"}},{"inlined":{"name":"uploader_type","content":"Web_HTML","contentType":"text/plain"}}]},"clientId":"scotty html form"}
EOL

    $resp = $ua->post($upload_video_url2, Content => $post_data_gibberish);
    unless ($resp->is_success) {
        die "Failed posting gibberish to upload_video_url2: ", $resp->status_line
    }

    # extract the file upload url
    my $file_upload_url = extract_field($resp->content, qr/"url":"([^"]+)"/);
    unless ($file_upload_url) {
        die "Failed extracting file upload url. YouTube might have redesigned!";
    }

    # now lets tell the server that we are starting to upload
    $resp = $ua->get($upload_video_url3);
    unless ($resp->is_success) {
        die "Failed getting upload_video_url3: ", $resp->status_line;
    }

    # now lets post the video
    $resp = $ua->post($file_upload_url, 
        {
            Filedata => [ $opts{f} ]
        },
        "Content_Type" => "form-data"
    );

    unless ($resp->is_success) {
        die "Failed uploading the file: ", $resp->status_line;
    }

    my $video_id = extract_field($resp->content, qr/"video_id":"([^"]+)"/);

    # tell the server that we are done
    $resp = $ua->get($upload_video_url4);
    unless ($resp->is_success) {
        die "Failed getting upload_video_url4: ", $resp->status_line;
    }

    # finally set the video info
    $resp = $ua->post($upload_video_set_info,
        {
            session_token     => $session_token,
            action_edit_video => 1,
            updated_flag      => 0,
            video_id          => $video_id,
            title             => $opts{t},
            description       => $opts{d},
            keywords          => $opts{x},
            category          => $opts{c},
            privacy           => 'public'
        }
    );

    unless ($resp->is_success) {
        die "Failed setting video info (but it uploaded ok!): ", $resp->status_line;
    }

    if ($resp->content =~ /"errors":\s*\[(.+?)\]/) {
        die "The video uploaded OK, but there were errors setting the video info:\n", $1;
    }
}

sub prepare_upload_urls {
    my ($SI, $UK) = @_;
    for ($upload_video_url1, $upload_video_url3, $upload_video_url4)
    {
        s/%SI%/$SI/;
        s/%UK%/$UK/;
    }
}

sub extract_field {
    my ($content, $rx) = @_;
    if ($content =~ /$rx/) {
        return $1
    }
}

sub HELP_MESSAGE {
    preamble();
    print "Usage: $0 ",
          "-l [login] ", 
          "-p [password] ",
          "-f <video file> ",
          "-c <category> ",
          "-t <title> ",
          "-d <description> ",
          "-x <comma, separated, tags>\n\n";
    print_cats();
}

sub print_cats {
    print "Possible categories (for -c switch):\n";
    printf "%-4s - %s\n", $_, $cats{$_} foreach (sort {
        $cats{$a} cmp $cats{$b}
    } keys %cats);
}

sub VERSION_MESSAGE {
    preamble();
    print "Version: v", VERSION, "\n";
}

sub preamble {
    print "YouTube video uploader by Peteris Krumins (peter\@catonmat.net)\n";
    print "http://www.catonmat.net  --  good coders code, great reuse\n";
    print "\n"
}

Download

Download link: youtube video uploader (perl script)
Total downloads: 8404 times

After you download the script ytup.perl, rename it back to ytup.pl. I could not get the webserver to stop interpreting .pl extension as a CGI script, so I renamed it to ytup.perl.

Thanks for reading the tutorial! If you notice any inaccuracies, bugs or just have something to say, comment on this article!

This article is part of the article series "GUI Video Downloader."
<- previous article next article ->

video downloader software, vim, mingw, wxwidgetsIn my previous posts I have shown you how to download YouTube videos and how to convert them to a better format. Wouldn't it be great to use the skills learned and create a small, cool desktop application which would download YouTube videos (and later extend to more video sites) and convert them to whatever format we want? I bet it would!

Let's get hands on it!

This will be a multi-part article on how to create this application. As the title of my blog "good coders code, great reuse" suggests, I will reuse as many software components as possible. I will use a free compiler, free gui framework and any free libraries I can find useful. Also I will try to take as many screenshots as possible to make this tutorial newbie friendly. For this reason I just installed a fresh copy of Windows XP on my VMware virtual machine (for those of you who don't know what virtual machine is, here is a quote from Intro to Virtualization: "Virtualization allows multiple virtual machines, with heterogeneous operating systems to run in isolation, side-by-side on the same physical machine.") which I will be developing the software from scratch on.

Since I am using the "good coders code, great reuse" principle, naturally I want parts of my code to be reusable. That's why I will first create a neat video downloading library in C which I will reuse later in the test command line application and the final GUI application which will probably be written in C++ using some cross-platform GUI framework/toolkit.

Part I: Preparation of the Tools

Since I am doing this project from scratch on a naked WinXP machine, the first thing I personally need need is a good browser. I will be searching for tools, libraries and will read documentation a lot, so I need a good browser. My choice - FireFox.

Next I will need a text editor to write the code in. My choice for this project - Vim. Since I want to create this project from scratch, I will need some essential plugins for vim, otherwise editing the source files would take me forever. From all the plugins I usually use in my vim configuration, for this project I'll use: a to quickly switch between .c and .h source files, bufexplorer to quickly navigate around buffers, bufpos to quickly switch between buffers, c-support to quickly write C/C++ code (it has key mapping cheat-sheet in the plugin's archive), surround to quickly change/add/insert quotes, parens etc, and taglist to navigate around source files. To use the last plugin (taglist) we also need Exuberant Ctags tool. Installation process of all these programs and plugins is straight forward.

I installed vim to "C:\program files\vim" and ctags to "C:\ctags56".

To compile the project we will also need a descent compiler. I will choose MinGW which is "a collection of freely available and freely distributable Windows specific header files and import libraries combined with GNU toolsets that allow one to produce native Windows programs that do not rely on any 3rd-party C runtime DLLs." Getting to run MinGW is not as easy as the previous tools so if you are not sure just download the automatic installer and it will do everything for you. The automatic installer can be downloaded from 'Current' package version release of MinGW at SourceForge.

I installed mingw using automated installer to "C:\mingw".

We still need a debugger. Going quickly through MinGW's files at SourceForge I couldn't find the GDB package. So I joined #mingw on FreeNode IRC network and in a few minutes someone helped me by suggesting that it was located in snapshot release. Thank you!

[17:15:49] * Now talking in #mingw
[17:16:07] <pkrumins> hi, what debugger could I use with mingw? I don't see gdb coming with any of the packages, did i miss it?
[17:20:30] <Vinky> pkrumins, snapshot->gdb

IRC is a powerful resource to get help at. Learn to use it.

Next we need a cross platform GUI toolkit. I am pretty familiar with wxWidgets and have used it in a few projects, so I will use for this project as well. I will not compile it at the moment because first I want to create a library and then the command line tool and only then the GUI app.

We are now ready to create the video downloading library. Before we do it, let's test if our development environment works.
Let's write the hello world application, compile it and run it. Click, click...

#include <stdio.h>

int main(void) {
    (void)printf("hello world\n");
    return 0;
}

We are using vim with c-support plugin, so compiling the hello world from vim is done (by looking up in the c-support plugin's keyboard mapping cheat sheet) by typing \rc and the program can be run by typing \rr. Quick and neato.

video downloader programming environment screenshot - vim, gcc, hello world

This is end of part one. In part two I will create the library and the command line test program. Until next time, kthxbye!

sed as a superman, the unix stream editorI present to you my cheat sheet of sed, the Superman of stream editing! It has come handy 101 times for me because sed is not what I use daily and after some time of not using the sed commands tend to fade away from my mind and I have to review them quickly. This cheat sheet is ideal for that!

You can't really be a good hacker if you don't know this tool. I always love to think of various situations where knowing many, many different tools makes you the ultimate guru.
Suppose you had a situation where you were on a box which had some problems and the only tool available to you was sed, and you had to fix a configuration file problem really quickly or bad, bad things would happen. If you never knew sed, you'd be in deep trouble because it can't be learned from the man page, because it only lists all the commands and you have no understanding how the commands work.

When I first learned sed, I remember the joy when I understood how it worked, that it operated on input stream, output stream, pattern space and hold buffer. Once you know this, the rest can be understood from the man page, but before that, I doubt it.

If you want to learn the sed editor I recommend this wonderful tutorial and these books.

This sed cheat sheet contains:

  • command line summary
  • command description, if they take single address or pattern, or a range of addresses, and what they modify (input stream, output stream, pattern space or hold buffer)
  • command line argument summary
  • extensions
  • short summary of adderss range format

If you notice any inaccuracies, mistakes or see that I have left something out, comment on this page, please!

Download Sed Stream Editor Cheat-Sheet

PDF:
Download link: sed stream editor cheat sheet (.pdf)
Downloaded: 133697 times

Plain Text (.txt):
Download link: sed stream editor cheat sheet (.txt)
Downloaded: 37752 times

Microsoft Word 2000 format (.doc):
Download link: sed stream editor cheat sheet (.doc)
Downloaded: 7318 times

Are you interested in sed and unix text editing power tools? Here are three great books on this topic from Amazon: