ed text editor and mr. edI have always had historic interest of various UNIX tools and the UNIX itself. One day when I was learning sed, the stream editor, (sed cheat sheet) I got interested how much the most powerful text editing tools on UNIX - 'ed', 'ex', 'sed' and 'vim' (or just 'vi') had in common and what they borrowed from one another.

I was not perfectly sure if 'sed' has originated from 'ed', so I Google searched for 'sed history' which turned up some interesting results, one of them being "The original users manual for sed, by Lee E. McMahon, from the 7th edition UNIX Manual (1978)".

Quoting the manual "Sed is a lineal descendant of the UNIX editor, ed." Great! I knew that 'sed' was non-interactive text editor whereas 'ed' was an interactive one. Also I knew that 'ex' was improved 'ed' and that 'sed' was a non-interactive 'ed' and that 'ex' had become 'vi' and was used in 'vi' command line mode. To boost my vi/vim skills and get a touch on how text editing was done with just 'ed', and satisfy my historical curiosity I set myself a goal to learn it.

As I mentioned in awk and sed cheat sheet posts, the best method for me to learn a new tool is to have all commands in front of me, so when I am doing experimentation I can quickly find the command and also see all the other commands at the same time, blowing them in subconsciousness.

Since 'ed' is line-oriented text editor, it is important to understand the line addressing, this cheat sheet summarizes the line addressing.

Most of 'ed' command are single letter in length. This cheat sheet summarizes all the commands, showing the supported address ranges and a detailed description of each command.

From my experience, once I had completed this cheat sheet and had it in front of me, I picked 'ed' up in 30 minutes. And then spent a few more hours experimenting and trying various constructs.

Good luck with learning ed! :)

And don't forget: ED IS THE TRUE PATH TO NIRVANA!

ps. if you notice any inaccuracies, spelling or grammar errors, please leave a comment here, so I can fix it. Thanks!
pss. logo for this article from mr. ed comedy ;)

Download Ed UNIX Text Editor's Cheat-Sheet

Plain Text (.txt):
Download link: ed text editor cheat sheet (.txt)
Downloaded: 19000 times

PDF:
Download link: ed text editor cheat sheet (.pdf)
Downloaded: 31563 times

Microsoft Word 2000 format (.doc):
Download link: ed text editor cheat sheet (.doc)
Downloaded: 3095 times

upload youtube perlWe have been downloading and converting YouTube videos but have not looked at how to upload them.

I will teach you how to upload YouTube videos via a command line. To find out how a video is being uploaded we will need just the Firefox browser!

Here is a typical scenario when you would want to automatically upload your videos to YouTube. Suppose you were using some other video sharing site and had already uploaded like 100 of your videos there. To get more popular on the net you'd also want to get your videos on YouTube, right? Doing it manually is a boring and tedious job, you want a program to do it for you while you sit back and relax.

Finding the YouTube Upload Form's Elements

Log into your account and go to "My Account" menu in the upper right. Then press the "Upload New Video" button.

how to upload youtube videos

Now let's use Firefox's "Page Info" tool which is located under "Tools" menu.

firefox's page info built in tool

When the "Page Info" tool pops up, select the "Forms" tab. The tool will list all the HTML forms on the page. The one named "theForm" with action's url "http://www.youtube.com/my_videos_upload" is the upload form! When a user uploads the video, it gets submitted. Our program will submit this form for us.

youtube upload video page's forms, fields, values

Now that we have found all the fields we need to submit, we can write the tool itself. I'll use my favorite programming language - Perl, again. It is perfect for this job because it has extremely good packages for working with HTTP protocol.
I will probably write another post just about the Perl programming language - how to get it working on Windows Operating System. For now, if you are running Windows, then you can download ActiveState's ActivePerl which is Perl's port to win.

Writing the Program

As I said, we will be creating the tool in Perl programming language. In the previous post about YouTube I used the WWW::Mechanize package. I can tell you in advance that it will not work this time because there will be some unexpected surprises, such as hidden form elements being created by Javascript when the form is submitted. Since WWW::Mechanize stores all the forms internally, it will not be able to submit this form with these dynamic form elements. That's why we will use the base class of WWW::Mechanize - the LWP::UserAgent.

LWP::UserAgent package can be installed with the following command:

perl -MCPAN -e 'install LWP::UserAgent'

Once you have it installed, we can start writing the program.

Step 1: Logging in to YouTube

Before we can upload a video, we need to login to YouTube. Let's use the same procedure as before to find out what form do we need to submit to login. Let's go to this url:

http://www.youtube.com/login

and look at all the forms on this page. One of them is named "loginForm" with four fields. We will need to submit this form and save the cookies. Luckily LWP::UserAgent can do it for us and we will not have to parse HTTP headers or anything like that.

YouTube's login form fields, types and their values

We see that there is no action URL provided for the loginForm, that means that the form should be POSTed to the same URL it was located on, in our case:

http://www.youtube.com/login

We also see that the form has 4 elements which should be submitted:

  • 'current_form' with value 'loginForm'
  • 'username' with our youtube login/username
  • 'password' with our password, and
  • 'action_login' with value 'Log In'

How would we submit this form with LWP::UserAgent? Easy - it has a member function $ua->post( $url, \%form, $header_field_name => $value, ... ), where %form is a hash of key, value pairs of form elements and $header_field_name => $value is a pair of HTTP headers we would like to add to the request.

Given an object of LWP::UserAgent named $ua, code which would login us to YouTube is the following:

$ua->post('http://www.youtube.com/login',
    {
        current_form => 'loginForm',
        username     => 'youtube_login',
        password     => 'youtube_password',
        action_login => 'Log In'
    },
);

See how easy it was? I love it.

The only problem is how to detect if we have really logged in. I noticed that when we log in, the upper right menu has a 'Log Out' button. So if we logged in succesfully, we should find this button.

Step 2: Uploading the Video

Now, once we have logged in, let's upload the video.

Here is the image with all the fields we have to submit to upload a video, once more:

youtube upload video page's forms, fields, values

There are 17 fields totally:

  • "field_myvideo_title" - title of the video
  • "field_myvideo_descr" - description of the video
  • "field_myvideo_keywords" - comma separated tags
  • "field_myvideo_categories" - video category
  • "language" - language of the video, i'll leave it "EN" for English
  • "action_upload" - a string "Upload a video..."
  • "session_token" - we will have to extract the session token for this upload
  • "allow_embeddings" - should the video be allowed to be embedded on other sites, i'll leave the default "Yes"
  • "allow_responses" - should we allow video responses, i'll leave the default "Yes"
  • "allow_comments" - should we allow comments on the video, i'll leave the default "Yes"
  • "allow_ratings" - should we allow users to rate the vide, i'll leave the default "Yes"
  • "location" - latitude and longitude of the location video was shot, i'll leave it empty
  • "field_date_mon" - month the video was shot, i'll use default 0 (unknown)
  • "field_date_day" - day the video was shot, i'll use 0 (unknown)
  • "field_date_yr" - year the video was shot, i'll use 0 (unknown)
  • "field_privacy" - privacy of video (public or private), i'll use the default - "public"
  • "ignore_broadcast_settings" - no idea what this is, i'll use the default value 0

The trickiest part here is to extract the "session_token" which is some kind of unique id.
Going through the HTML source of upload page:

http://www.youtube.com/my_videos_upload

We find a javascript function called "dynamic_append_session_token" which dynamically creates a hidden form element containing the session ID:

finding the session id of youtube upload form

We have to extract this session ID from the line in red. It can be done with the following regular expression:

token_elem\.setAttribute\('value', '(.+?)'\);

Update 2008.03.12: This is no longer true! A session cookie is represented by a variable now:

var gXSRF_token = 'OTxXiSg8O-oQFB-PWTSaDv7oX2V8MTIwNTQ1NzYxNQ==';

When we have done this, we can submit the form. For this form we will need to add a "Content-Type: form-data" header to the request because the upload form has explicitly stated that its "enctype" is "multipart/form-data".

As it turns out the actual video upload process is a two step process. This was the first step which set the video information. After we submit this form, we get redirected to a page which contains the upload file selection form. It might seem that that is the only element in this form but not so. Actually this 2nd step form has a few new fields and it also has all the fields from 1st step as hidden elements. Here are the modified and added fields in the 2nd step:

  • "contact" - no idea what this field is for
  • "field_command" - always "myvideo_submit"
  • "field_uploadfile" - our video file we want to upload
  • "field_private_share_entities" - no idea
  • "action_upload" - submit button's caption "Upload Video"
  • "addresser" - another mysterious unique id

Here is what page info tool shows us about 2nd step's form elements:

youtube upload video 2nd step

Notice that the form's action URL changes, we have to extract it from the html source. Someone might suggest not to use regexes for parsing HTML but it is perfectly ok for this tiny project.

The trickiest thing here is telling LWP::UserAgent that 'field_uploadfile' is a form element of 'file' type. If we look in the LWP::UserAgent's documentation we find that the three argument post() subroutine uses "POST() function from HTTP::Request::Common to build the request." Further looking in the HTTP::Request::Common documentation we find that to specify a file to be uploaded the field has to be an array ref looking like this: [ $file, $filename, Header => Value... ].

That's it. When we submit this form the video should have been uploaded. Now we just need to detect if the upload was successful. I noticed that when the video has uploaded successfully, YouTube thanks me for uploading. It looks like this:

youtube video upload complete

So checking if a video uploaded successfully is reduced to checking for the thank you message.

The program can be used from command line and takes a few mandatory arguments:

Usage: perl ytup.pl -l [login] -p [password] -f <video file> -c <category> -t <title> -d <description> -x <comma, separated, tags>

The login (-l) and password (-p) arguments are not mandatory because not to expose your password on a publically shared machine (by looking at process list) they can be specified in the source file by changing YT_LOGIN and YT_PASS constants.

Possible categories (for -c switch):
2    - Autos & Vehicles
23   - Comedy
24   - Entertainment
1    - Film & Animation
20   - Gadgets & Games
26   - Howto & DIY
10   - Music
25   - News & Politics
22   - People & Blogs
15   - Pets & Animals
17   - Sports
19   - Travel & Places

Here is the final code:

#!/usr/bin/perl 
#
# Version 1.0: 2007.07.30
# Version 1.1: 2007.10.12: youtube changed html
# Version 1.2: 2008.03.13: youtube changed html
# Version 1.3: 2009.12.02: youtube changed html and now logins with google login
#
# Peteris Krumins (peter@catonmat.net)
# http://www.catonmat.net  --  good coders code, great reuse
#

use constant VERSION => "1.3";

use strict;
use warnings;

use LWP::UserAgent;
use HTML::Entities 'decode_entities';
use Getopt::Std;
$Getopt::Std::STANDARD_HELP_VERSION = 1;

# debug?
use constant DEBUG => 0;

# set these values for default -l (login) and -p (pass) values
#
use constant YT_LOGIN => "";
use constant YT_PASS  => "";

# video categories
#
my %cats = (
    2   =>  'Autos & Vehicles',
    23  =>  'Comedy',
    27  =>  'Education',
    24  =>  'Entertainment',
    1   =>  'Film & Animation',
    20  =>  'Gaming',
    26  =>  'Howto & Style',
    10  =>  'Music',
    25  =>  'News & Politics',
    29  =>  'Nonprofits & Activism',
    22  =>  'People & Blogs',
    15  =>  'Pets & Animals',
    28  =>  'Science & Technology',
    17  =>  'Sports',
    19  =>  'Travel & Places'
);

# various urls
my $login_url      = 'http://www.youtube.com/login';
my $login_post_url = 'https://www.google.com/accounts/ServiceLoginAuth?service=youtube';
my $login_cont_url = 'http://www.youtube.com/';
my $upload_url     = 'http://upload.youtube.com/my_videos_upload?restrict=html_form';
my $upload_video_url1 = 'http://www.youtube.com/gen_204?a=multi_up_queue&si=%SI%&uk=%UK%&ac=1&gbe=1&fl=0&b=0&fn=0&ti=1&d=0&ta=0&pv=0&c=0&m=0&t=0&ft=0&dn=upload.youtube.com&fe=scotty&ut=html_form';
my $upload_video_url2 = 'http://upload.youtube.com/upload/rupio';
my $upload_video_url3 = 'http://www.youtube.com/gen_204?a=multi_up_start&si=%SI%&uk=%UK%&ac=1&gbe=1&fl=0&b=0&fn=0&ti=1&d=0&ta=0&pv=0&c=0&m=0&t=0&ft=0&dn=upload.youtube.com&fe=scotty&ut=html_form';
my $upload_video_url4 = 'http://www.youtube.com/gen_204?a=multi_up_finish&si=%SI%&uk=%UK%&ac=1&gbe=1&fl=0&b=0&fn=0&ti=1&d=0&ta=0&pv=0&c=0&m=0&t=23001&ft=0&dn=upload.youtube.com&fe=scotty&ut=html_form';
my $upload_video_set_info = 'http://upload.youtube.com/my_videos_upload_json';


unless (@ARGV) {
    HELP_MESSAGE();
    exit 1;
}

my %opts;
getopts('l:p:f:c:t:d:x:', \%opts);

# if -l or -p are not given, try to use YT_LOGIN and YT_PASS constants
unless (defined $opts{l}) {
    unless (length YT_LOGIN) {
        preamble();
 print "Youtube username/login as neither defined nor passed as an argument\n";
        print "Use -l switch to specify the username\n";
        print "Example: -l joe_random\n";
        exit 1;
    }
    else {
        $opts{l} = YT_LOGIN;
    }
}

unless (defined $opts{p}) {
    unless (length YT_PASS) {
        preamble();
        print "Password was neither defined nor passed as an argument\n";
        print "Use -p switch to specify the password\n";
        print "Example: -p secretPass\n";
        exit 1;
    }
    else {
        $opts{p} = YT_PASS;
    }
}

unless (defined $opts{f} && length $opts{f}) {
    preamble();
    print "No video file was specified\n";
    print "Use -f switch to specify the video file\n";
    print 'Example: -f "C:\Program Files\movie.avi"', "\n";
    print 'Example: -f "/home/pkrumins/super.cool.video.wmv"', "\n";
    exit 1;
}

unless (-r $opts{f}) {
    preamble();
    print "Video file is not readable or does not exist\n";
    print "Check the permissions and the path to the file\n";
    exit 1;
}

unless (defined $opts{c} && length $opts{c}) {
    preamble();
    print "No video category was specified\n";
    print "Use -c switch to set the category of the video\n";
    print "Example: -c 20, would set category to Gadgets & Games\n\n";
    print_cats();
    exit 1;
}

unless (defined $cats{$opts{c}}) {
    preamble();
    print "Category '$opts{c}' does not exist\n\n";
    print_cats();
    exit 1;
}

unless (defined $opts{t} && length $opts{t}) {
    preamble();
    print "No video title was specified\n";
    print "Use -t switch to set the title of the video\n";
    print 'Example: -t "Super Cool Video Title"', "\n";
    exit 1;
}

unless (defined $opts{d} && length $opts{d}) {
    preamble();
    print "No video description was specified\n";
    print "Use -d switch to set the description of the video\n";
    print 'Example: -d "The coolest video description"', "\n";
    exit 1;
}

unless (defined $opts{x} && length $opts{x}) {
    preamble();
    print "No tags were specified\n";
    print "Use -x switch to set the tags\n";
    print 'Example: -x "foo, bar, baz, hacker, purl"', "\n";
    exit 1;
}

# tags should be at least two chars, can't be just numbers
my @tags = split /,\s+/, $opts{x};
my @filtered_tags = grep { length > 2 && !/^\d+$/ } @tags;
unless (@filtered_tags) {
    preamble();
    print "Tags must at least two chars in length and must not be numeric!\n";
    print "For example, 'foo', 'bar', 'yo29' are valid tags, but ";
    print "'22222', 'hi', 'b9' are invalid\n";
    exit 1;
} 

$opts{x} = join ', ', @filtered_tags;
# create the user agent, have it store cookies and
# pretend to be a cool windows firefox browser
my $ua = LWP::UserAgent->new(
    cookie_jar => {},
    agent => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.4) '.
             'Gecko/20070515 Firefox/2.0.0.4'
);

if (DEBUG) {
    $ua->add_handler("request_send",  sub { print "Request: \n"; shift->dump; print "\n"; return });
    $ua->add_handler("response_done", sub { print "Response: \n"; shift->dump; print "---\n"; return });
}

# let the user agent follow redirects after a POST request
push @{$ua->requests_redirectable}, 'POST';

print "Logging in to YouTube...\n";
login();

print "Uploading the video ($opts{t})...\n";
upload();

print "Done!\n";

sub login {
    # go to login page to get redirected to google sign in page
    my $res = $ua->get($login_url);
    unless ($res->is_success) {
        die "Failed going to YouTube's login URL: ", $res->status_line;
    }

    # extract GALX id
    my $GALX = extract_field($res->content, qr/name="GALX".+?value="([^"]+)"/s);
    unless ($GALX) {
        die "Failed logging in. Unable to extract GALX identifier from Google's login page.";
    }

    # submit the login form
    $res = $ua->post($login_post_url,
        {
            ltmpl    => 'sso',
            continue => 'http://www.youtube.com/signin?action_handle_signin=true&nomobiletemp=1&hl=en_US&next=%2F',
            service  => 'youtube',
            uilel    => 3,
            hl       => 'en_US',
            GALX     => $GALX,
            Email    => $opts{l},
            Passwd   => $opts{p},
            rmShown  => 1,
            signIn   => 'Sign in',
            asts     => ''
        }
    );
    unless ($res->is_success) {
        die "Failed logging in: failed submitting login form: ", 
            $res->status_line;
    }

    # Get the meta http-equiv="refresh" url
    my $next_url = extract_field($res->content, qr/http-equiv="refresh" content="0; url='(.+?)'/);
    unless ($next_url) {
        die "Failed logging in. Getting next url from http-equiv failed.";
    }
    $next_url = decode_entities($next_url);

    $res = $ua->get($next_url);
    unless ($res->is_success) {
        die "Failed logging in. Extraction of next_url failed: ",
            $res->status_line;
    }

    $res = $ua->get($login_cont_url);
    unless ($res->is_success) {
        die "Failed logging in. Navigation to YouTube.com failed: ", 
            $res->status_line;
    }

    # We have no good way to check if we really logged in.
    # I found that when you have logged in the upper right menu changes
    # and you have access to 'Sign Out' option.
    # We check for this string to see if we have logged in.
    unless ($res->content =~ />Sign Out</) {
        die "Failed logging in: username/password incorrect";
    }
}

sub upload {
    # upload is actually a multistep process
    #
    
    # get upload page to extract some gibberish info
    my $resp = $ua->get($upload_url);
    unless ($resp->is_success) {
        die "Failed getting $upload_url: ", $resp->status_line;
    }

    my $SI = extract_field($resp->content, qr/"sessionKey": "([^"]+)"/);
    unless ($SI) {
        die "Failed extracting sessionKey. YouTube might have redesigned!";
    }

    my $UK = extract_field($resp->content, qr/"uploadKey": "([^"]+)"/);
    unless ($UK) {
        die "Failed extracting uploadKey. YouTube might have redesigned!";
    }

    my $session_token = extract_field($resp->content, qr/"session_token" value="([^"]+)"/);
    unless ($session_token) {
        die "Failed extracting session_token. YouTube might have redesigned!";
    }

    prepare_upload_urls($SI, $UK);

    # tell the server that we are up for uploads
    $resp = $ua->get($upload_video_url1);
    unless ($resp->is_success) {
        die "Failed getting upload_video_url1: ", $resp->status_line;
    }

    # now lets post some more gibberish data
my $post_data_gibberish =<<EOL;
{"protocolVersion":"0.7","createSessionRequest":{"fields":[{"external":{"name":"file","filename":"$opts{f}","formPost":{}}},{"inlined":{"name":"return_address","content":"upload.youtube.com","contentType":"text/plain"}},{"inlined":{"name":"upload_key","content":"$UK","contentType":"text/plain"}},{"inlined":{"name":"action_postvideo","content":"1","contentType":"text/plain"}},{"inlined":{"name":"live_thumbnail_id","content":"$SI.0","contentType":"text/plain"}},{"inlined":{"name":"parent_video_id","content":"","contentType":"text/plain"}},{"inlined":{"name":"allow_offweb","content":"True","contentType":"text/plain"}},{"inlined":{"name":"uploader_type","content":"Web_HTML","contentType":"text/plain"}}]},"clientId":"scotty html form"}
EOL

    $resp = $ua->post($upload_video_url2, Content => $post_data_gibberish);
    unless ($resp->is_success) {
        die "Failed posting gibberish to upload_video_url2: ", $resp->status_line
    }

    # extract the file upload url
    my $file_upload_url = extract_field($resp->content, qr/"url":"([^"]+)"/);
    unless ($file_upload_url) {
        die "Failed extracting file upload url. YouTube might have redesigned!";
    }

    # now lets tell the server that we are starting to upload
    $resp = $ua->get($upload_video_url3);
    unless ($resp->is_success) {
        die "Failed getting upload_video_url3: ", $resp->status_line;
    }

    # now lets post the video
    $resp = $ua->post($file_upload_url, 
        {
            Filedata => [ $opts{f} ]
        },
        "Content_Type" => "form-data"
    );

    unless ($resp->is_success) {
        die "Failed uploading the file: ", $resp->status_line;
    }

    my $video_id = extract_field($resp->content, qr/"video_id":"([^"]+)"/);

    # tell the server that we are done
    $resp = $ua->get($upload_video_url4);
    unless ($resp->is_success) {
        die "Failed getting upload_video_url4: ", $resp->status_line;
    }

    # finally set the video info
    $resp = $ua->post($upload_video_set_info,
        {
            session_token     => $session_token,
            action_edit_video => 1,
            updated_flag      => 0,
            video_id          => $video_id,
            title             => $opts{t},
            description       => $opts{d},
            keywords          => $opts{x},
            category          => $opts{c},
            privacy           => 'public'
        }
    );

    unless ($resp->is_success) {
        die "Failed setting video info (but it uploaded ok!): ", $resp->status_line;
    }

    if ($resp->content =~ /"errors":\s*\[(.+?)\]/) {
        die "The video uploaded OK, but there were errors setting the video info:\n", $1;
    }
}

sub prepare_upload_urls {
    my ($SI, $UK) = @_;
    for ($upload_video_url1, $upload_video_url3, $upload_video_url4)
    {
        s/%SI%/$SI/;
        s/%UK%/$UK/;
    }
}

sub extract_field {
    my ($content, $rx) = @_;
    if ($content =~ /$rx/) {
        return $1
    }
}

sub HELP_MESSAGE {
    preamble();
    print "Usage: $0 ",
          "-l [login] ", 
          "-p [password] ",
          "-f <video file> ",
          "-c <category> ",
          "-t <title> ",
          "-d <description> ",
          "-x <comma, separated, tags>\n\n";
    print_cats();
}

sub print_cats {
    print "Possible categories (for -c switch):\n";
    printf "%-4s - %s\n", $_, $cats{$_} foreach (sort {
        $cats{$a} cmp $cats{$b}
    } keys %cats);
}

sub VERSION_MESSAGE {
    preamble();
    print "Version: v", VERSION, "\n";
}

sub preamble {
    print "YouTube video uploader by Peteris Krumins (peter\@catonmat.net)\n";
    print "http://www.catonmat.net  --  good coders code, great reuse\n";
    print "\n"
}

Download

Download link: youtube video uploader (perl script)
Total downloads: 8128 times

After you download the script ytup.perl, rename it back to ytup.pl. I could not get the webserver to stop interpreting .pl extension as a CGI script, so I renamed it to ytup.perl.

Thanks for reading the tutorial! If you notice any inaccuracies, bugs or just have something to say, comment on this article!

This article is part of the article series "GUI Video Downloader."
<- previous article next article ->

video downloader software, vim, mingw, wxwidgetsIn my previous posts I have shown you how to download YouTube videos and how to convert them to a better format. Wouldn't it be great to use the skills learned and create a small, cool desktop application which would download YouTube videos (and later extend to more video sites) and convert them to whatever format we want? I bet it would!

Let's get hands on it!

This will be a multi-part article on how to create this application. As the title of my blog "good coders code, great reuse" suggests, I will reuse as many software components as possible. I will use a free compiler, free gui framework and any free libraries I can find useful. Also I will try to take as many screenshots as possible to make this tutorial newbie friendly. For this reason I just installed a fresh copy of Windows XP on my VMware virtual machine (for those of you who don't know what virtual machine is, here is a quote from Intro to Virtualization: "Virtualization allows multiple virtual machines, with heterogeneous operating systems to run in isolation, side-by-side on the same physical machine.") which I will be developing the software from scratch on.

Since I am using the "good coders code, great reuse" principle, naturally I want parts of my code to be reusable. That's why I will first create a neat video downloading library in C which I will reuse later in the test command line application and the final GUI application which will probably be written in C++ using some cross-platform GUI framework/toolkit.

Part I: Preparation of the Tools

Since I am doing this project from scratch on a naked WinXP machine, the first thing I personally need need is a good browser. I will be searching for tools, libraries and will read documentation a lot, so I need a good browser. My choice - FireFox.

Next I will need a text editor to write the code in. My choice for this project - Vim. Since I want to create this project from scratch, I will need some essential plugins for vim, otherwise editing the source files would take me forever. From all the plugins I usually use in my vim configuration, for this project I'll use: a to quickly switch between .c and .h source files, bufexplorer to quickly navigate around buffers, bufpos to quickly switch between buffers, c-support to quickly write C/C++ code (it has key mapping cheat-sheet in the plugin's archive), surround to quickly change/add/insert quotes, parens etc, and taglist to navigate around source files. To use the last plugin (taglist) we also need Exuberant Ctags tool. Installation process of all these programs and plugins is straight forward.

I installed vim to "C:\program files\vim" and ctags to "C:\ctags56".

To compile the project we will also need a descent compiler. I will choose MinGW which is "a collection of freely available and freely distributable Windows specific header files and import libraries combined with GNU toolsets that allow one to produce native Windows programs that do not rely on any 3rd-party C runtime DLLs." Getting to run MinGW is not as easy as the previous tools so if you are not sure just download the automatic installer and it will do everything for you. The automatic installer can be downloaded from 'Current' package version release of MinGW at SourceForge.

I installed mingw using automated installer to "C:\mingw".

We still need a debugger. Going quickly through MinGW's files at SourceForge I couldn't find the GDB package. So I joined #mingw on FreeNode IRC network and in a few minutes someone helped me by suggesting that it was located in snapshot release. Thank you!

[17:15:49] * Now talking in #mingw
[17:16:07] <pkrumins> hi, what debugger could I use with mingw? I don't see gdb coming with any of the packages, did i miss it?
[17:20:30] <Vinky> pkrumins, snapshot->gdb

IRC is a powerful resource to get help at. Learn to use it.

Next we need a cross platform GUI toolkit. I am pretty familiar with wxWidgets and have used it in a few projects, so I will use for this project as well. I will not compile it at the moment because first I want to create a library and then the command line tool and only then the GUI app.

We are now ready to create the video downloading library. Before we do it, let's test if our development environment works.
Let's write the hello world application, compile it and run it. Click, click...

#include <stdio.h>

int main(void) {
    (void)printf("hello world\n");
    return 0;
}

We are using vim with c-support plugin, so compiling the hello world from vim is done (by looking up in the c-support plugin's keyboard mapping cheat sheet) by typing \rc and the program can be run by typing \rr. Quick and neato.

video downloader programming environment screenshot - vim, gcc, hello world

This is end of part one. In part two I will create the library and the command line test program. Until next time, kthxbye!

sed as a superman, the unix stream editorI present to you my cheat sheet of sed, the Superman of stream editing! It has come handy 101 times for me because sed is not what I use daily and after some time of not using the sed commands tend to fade away from my mind and I have to review them quickly. This cheat sheet is ideal for that!

You can't really be a good hacker if you don't know this tool. I always love to think of various situations where knowing many, many different tools makes you the ultimate guru.
Suppose you had a situation where you were on a box which had some problems and the only tool available to you was sed, and you had to fix a configuration file problem really quickly or bad, bad things would happen. If you never knew sed, you'd be in deep trouble because it can't be learned from the man page, because it only lists all the commands and you have no understanding how the commands work.

When I first learned sed, I remember the joy when I understood how it worked, that it operated on input stream, output stream, pattern space and hold buffer. Once you know this, the rest can be understood from the man page, but before that, I doubt it.

If you want to learn the sed editor I recommend this wonderful tutorial and these books.

This sed cheat sheet contains:

  • command line summary
  • command description, if they take single address or pattern, or a range of addresses, and what they modify (input stream, output stream, pattern space or hold buffer)
  • command line argument summary
  • extensions
  • short summary of adderss range format

If you notice any inaccuracies, mistakes or see that I have left something out, comment on this page, please!

Download Sed Stream Editor Cheat-Sheet

PDF:
Download link: sed stream editor cheat sheet (.pdf)
Downloaded: 129741 times

Plain Text (.txt):
Download link: sed stream editor cheat sheet (.txt)
Downloaded: 36724 times

Microsoft Word 2000 format (.doc):
Download link: sed stream editor cheat sheet (.doc)
Downloaded: 7092 times

Are you interested in sed and unix text editing power tools? Here are three great books on this topic from Amazon:

ffmpeg video converterIn the few previous posts we have been downloading YouTube videos with Awk and Perl. What we ended up with were .flv (Flash Media Video) files which do not play with the usual video players like Windows Media Player which comes with Windows out-of-the-box.

To play these .flv's we either need a special video player which supports this video file format such as VLC Media Player or FLV Player, or get a good video codec pack (which I do not recommend since it eventually messes the whole multimedia subsystem up, not gonna even link to it), or get a ffdshow all-in-one video codec (I recommend it).

The ffdshow solution is pretty good and that is what I use myself together with the excellent Media Player Classic video player which replaces the horrific Windows Media Player once and forever.

Now suppose you wanted to send the video file to your friend who didn't know about all these video codecs and players and didn't want to have them installed. What would you do?

Or suppose you wanted to put the video on your brand new iPhone or video iPod, or your cellphone.

Let's learn how to convert the video to a better format such as windows media video (.wmv), .avi, mpeg or divx, or mobile phone's 3gp format (or even others).

The tool we are going to use is called ffmpeg. ffmpeg is a command line tool to convert one video and audio file format to another.

The official homepage of the tool is at ffmpeg.mplayerhq.hu which is more often down than not. The official download page is here, alternative download here.

I went to the alternative site, downloaded the latest version (ffmpeg.rev9767.7z) and extracted it with WinRar.

The other cool thing about ffmpeg is that it is a command line tool and it's cross platfrom, so you can run it anywhere and it does not need the GUI.

Once you have it downloaded, unpack it and run the tool without any command line options:

usage: ffmpeg [[infile options] -i infile]... {[outfile options] outfile}...
Hyper fast Audio and Video encoder

Main options:
-L                  show license
-h                  show help
-version            show version
-formats            show available formats, codecs, protocols, ...
-f fmt              force format
...

Okay, from the first usage line we now know that to convert a video we will run the tool as following:

> ffmpeg -i our_flash_video.flv ... options ... our_desired_video.avi (or .mpg, or whatever)

Besides this valuable information we also get hundreds of various options which do not yet make much sense to us.
We need documentation of the tool to understand it. Quickly typing 'ffmpeg documentation' into Google we find this ffmpeg documentation page.

Before going into documentation let's first just try converting the video:

> ffmpeg.exe -i youtube_flash_video.flv out.avi
FFmpeg version SVN-r9767, Copyright (c) 2000-2007 Fabrice Bellard, et al.
  configuration: --enable-gpl --enable-pp --enable-swscaler --enable-pthreads --
enable-liba52 --enable-avisynth --enable-libamr-nb --enable-libamr-wb --enable-l
ibfaac --enable-libfaad --enable-libgsm --enable-libmp3lame --enable-libnut --en
able-libogg --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libx
vid --cpu=i686 --enable-memalign-hack --extra-ldflags=-static
  libavutil version: 49.4.1
  libavcodec version: 51.40.4
  libavformat version: 51.12.1
  built on Jul 20 2007 18:03:34, gcc: 4.2.0

Seems stream 0 codec frame rate differs from container frame rate: 1000.00 (1000
/1) -> 15.00 (15/1)
Input #0, flv, from 'youtube_flash_video.flv':
  Duration: 00:01:23.9, start: 0.000000, bitrate: 64 kb/s
  Stream #0.0: Video: flv, yuv420p, 320x240, 15.00 fps(r)
  Stream #0.1: Audio: mp3, 22050 Hz, mono, 64 kb/s
Output #0, avi, to 'out.avi':
  Stream #0.0: Video: mpeg4, yuv420p, 320x240, q=2-31, 200 kb/s, 15.00 fps(c)
  Stream #0.1: Audio: mp2, 22050 Hz, mono, 64 kb/s
Stream mapping:
  Stream #0.0 -> #0.0
  Stream #0.1 -> #0.1
Press [q] to stop encoding
frame= 1259 fps=219 q=2.0 Lsize=    2780kB time=83.4 bitrate= 272.9kbits/s
video:2050kB audio:652kB global headers:0kB muxing overhead 2.860901%

Hey, wow! It worked, we just converted our .flv to an mpeg4 video! Also the sound got converted from mp3 to mp2.

We would like to have more control over these parameters, right?

Let's go through documentation and find how to change audio and video bitrate, video framerate, video and codecs and output video size.
For various video options there is a section in documentation called 'Video Options', there we find that '-b bitrate' option changes video bitrate, '-s wxh' changes video width and height, '-vcodec codec' changes the video codec and '-r framerate' changes the video frame rate.

The audio options are listed here and we find that to change the audio bitrate we need to specify '-ab bitrate' and '-acodec codec' to change the audio codec.

The other interesting option is '-formats' which lists the supported codecs and protocols.

The videos we get from YouTube are usually 320x240 in size and at 15 fps with 64kbit/s mp3 sound.

If we did not specify the vid size, framerate and other parameters the input video's ones would be used for output video.

Just to illustrate how to use those command options, let's convert a YouTube video to 3gp format for viewing on a mobile phone. What is this 3gp format? Looking in wikipedia for 3gp we find that this format stores video streams as MPEG-4 Part 2 or H.263 and audio streams as AMR-NB, AMR-WB, AMR-WB+ or AAC-LC.
Also it turned out that H.263 can only take a number of video sizes which are listed on this page.
All we have to do is just to specify the right command line flags - and voila!

> ffmpeg.exe -i youtube_flash_video.flv -r 15 -b 200kbit/s -s 176x144 -vcodec h263 -ar 8000Hz -ab 10.2k -acodec libamr_nb for_mobile.3gp

Yeap, it worked, I uploaded the file to my Nokia N73 and it played perfectly!

To convert it to DivX and leave all the other parameters as is, use the following simple command line:

> ffmpeg.exe -i youtube_flash_video.flv -vcodec mpeg4 divx_youtube_movie.avi

To convert to other formats, just look at the supported encoding formats via '-formats' command line option. For example, to convert a video to .wmv, you would specify '-vcodec wmv2' or '-vocdec wmv1', etc.

Now we know how to download youtube videos and how to convert them. It seems that a perfect time has come to create a free, quick and cool desktop software application which will do this job for us.

Thanks for reading and until next time!