perl pack unpack printf sprintf cheat sheetI decided one day that I want to master Perl's pack() and unpack() functions to be able to manipulate data in Perl efficiently.

Perl's pack and unpack are two functions for transforming data according to a user-defined template, between the guarded way Perl stores values and some well-defined representation as might be required in the environment of a Perl program. Unfortunately, they're also two of the most misunderstood and most often overlooked functions that Perl provides.

As I wrote before, my way of learning these complex functions were to make a cheat sheet first with all the template parameters and then just spend a day reading more about them and experimenting.

As I usually print cheat sheets two pages per side and the pack/unpack cheat sheet consumed just one page, I added Perl's printf/sprintf format and attribute summary.

Here is how I printed this cheat sheet:

perl pack unpack printf sprintf cheat sheet thumbnail
(Sorry for the bad quality, I shot it with my camera phone)

Download Perl's pack/unpack and printf Cheat Sheet

PDF:
Download link: perl's pack/unpack and printf cheat sheet (.pdf)
Downloaded: 126921 times

Microsoft Word 2000 format (.doc):
Download link: perl's pack/unpack and printf cheat sheet (.doc)
Downloaded: 3507 times

javascript rhino and yahoo theatreI decided I wanted to learn JavaScript Programming language better. I had been programming in it now and then but I had never really developed any good skills in it.

If you have read about this blog page then you know that I also run Free Science Online blog which is all about free video lectures online. In my May's post I had found some really good programming video lectures, 13 of them being on JavaScript. Since I run this video lecture blog, I obviously have great interest in video lectures, so why not try to learn better JavaScript from these video lectures?

These lectures are given by Douglas Crockford who is a senior JavaScript Architect at Yahoo!. He is well known for his work in introducing JavaScript Object Notation (JSON).


First four lectures are on the basics of language:

Sometimes Yahoo! Video gives this error: Sorry! This video is no longer available on Yahoo! Video. In this case refresh your browser a couple of times!

The next three lectures are on Advanced JavaScript:

Then there are 6 more lectures on JavaScript which should probably be viewed only after you have viewed the ones just mentioned.

Viewing the YUI Theater I just found another JavaScript lecture which was published just recently:

My approach to watching video lectures

I have been watching various video lectures for almost 4 years now. Mostly mathematics, physics and theory of computer science. My approach to getting most of the lectures is the following. When I watch the video lectures I take notes just as if I were in class.
Even better, when I do not understand any part of the lecture I can always rewind it back and see that fragment again. I can also pause the lecture, think for a while and then continue. But that's physics and maths.
Here is a photo of notes I have taken while watching MIT's 803: Vibrations and Waves (yes! it's available completely for free at MIT's Open Course Ware)

learning 803 vibrations and waves thumbnail pic

I am serious about physics and maths video lectures, as you can see in the image, all the main results are boxed in red, the results are fully derived (even if the professor does not do it on blackboard). Btw, one lecture perfectly fits on both sides of an A4 sheet.

Here is a close-up of Lecture 13: Electromagnetic Waves - Plane Wave Solutions to Maxwell's Equations - Polarization - Malus' Law.

mit'ss 803 - lecture 13 - em waves, plane waves

(Sorry about the bad quality of the photos, I shot them with my Nokia N73 cell phone camera)

This approach probably does not work with programming languages and computer tools. Because mathematics and physics is mostly done on paper, watching these video lectures and taking notes is actually doing them. The process of taking notes develops the skills because you work with the new concepts/operators/theorems/whatnot. Not so in programming languages. Unless you find an online degree for Java programmers, you can take a book on a new programming language, read it, and the next moment you can't even write the hello world program because you have only got familiar with the subject and have not developed the skills. I have experienced this myself.

Here is my approach how I am going to learn JavaScript from these lectures. I might adjust this approach at any moment if I find it not working, I will update this post appropriately then.

I will definitely watch all 11 video lectures. I will start with the first four basic lectures, watch them one by one, take notes as with physics video lectures and experiment as I go.

I will be taking notes lightly to have my mind really think the information I am getting over. But no red boxes around constructs as with physics. That's the experimentation part to learning. I am going to try the new constructs as soon as I see them so they stuck in my mind better.
Update: I dropped the idea of taking any notes on paper, because I am blogging the key points from lectures here.

Also to make this article interesting, I will annotate each lecture if something really interesting catches my eye. As I mentioned, I have programmed JS before so I am not sure how much I will learn from the first four basic lectures.

In my previous blog post I used Windows Script Host to create a program in VBScript. The other language the same scripting host runs is JScript which conforms to the same standard as JavaScript. So I should be safe doing JavaScript experimentation in JScript.

Points that caught my attention in JavaScript Video Lecture Part I

  • (00:45) World's most misunderstood programming language - has "Java" in its name and "Script". It has nothing to do with Java programming language and it's a real programming language not some tiny scripting language
  • (02:38) There are generally no books available to learn JS from - all are bad and full of nasty examples
  • (02:56) The only book recommended is JavaScript: The Definitive Guide, 5th Edition by David Flanagan - the least bad book
  • (03:37) JavaScript is a functional language
  • (08:12) Microsoft reverse engineered original implementation of JavaScript and called it JScript to avoid trademark issues
  • (09:49) During standardization, the original bugs were left as is without fixing to prevent already written programs from breaking (Douglas slips and says "Sun" but he actually means "Microsoft")
  • (12:16) One of the key ideas of the language is prototypal inheritance where objects inherit from objects and there are no classes
  • (12:45) One other key idea is that functions are first-class objects
  • (13:36) There are no integers in the language, everything is represented as 64-bit floating point numbers
  • (14:30) NaN (Not a Number) is not equal to antyhing, including NaN. Which means NaN == NaN is false
  • (15:22) Type of NaN is Number
  • (15:52) + prefix operator does the same thing as Number function
  • (16:08) Generally always specify radix argument in parseInt function because if the first character is '0' and there is no radix argument provided, it will assume that '0' to be an octal constant
  • (17:15) Each character takes 16-bits of memory
  • (17:56) There is no separate character type in JS, characters are represented as strings with a length of 1
  • (19:55) undefined is the default value for uninitialized variables and parameters
  • (20:38) Falsy values are: false, null, undefined, "", 0, NaN. All other values (including all objects are truthy). Everything else in the language are objects
  • (22:15) Object members can be accessed with dot notation Object.member or subscript notation Object["member"]
  • (22:59) The language is loosely typed but not "untyped"
  • (25:19) Reserved words are overused, they can't be used in dot notation as method names
  • (27:25) Operators == and != can do type coercion, so it's better to use === and !=== which do no coercion
  • (28:24) Operator && is also called the 'guard operator', Operator || is also called the 'default operator'
  • (30:16) The bitwise operators convert the operand to a 32-bit signed integer, perform the operation and then turn the result back into 64-bit floating point. Don't use bitwise operators in cases like multiplying by 4 using << 2. It will not.

After having watched the first lecture I decided that there was no point in taking notes on paper because I am blogging the key points here and trying various examples I can come up with with JScript, taking notes just wastes time.

Points that caught my attention in JavaScript Video Lecture Part II

  • (00:20) Break statements can have labels
  • (00:41) Iterating over all the members of an object with for (var name in object) { } syntax, will also iterate over inherited members
  • (06:10) There is only global and function scope in JavaScript, there is no block scope
  • (07:40) If there is no expression in a return statement, the value returned is undefined. Except for constructors, whose default return value is this
  • (08:29) An object is an unordered collection of name/value pairs
  • (21:10) All objects are linked directly or indirectly to Object.prototype
  • (23:42) Array indexes are converted to strings and used as names for retrieving values

Points that caught my attention in JavaScript Video Lecture Part III

  • (00:29) Functions inherit from Object and can store name/value pairs
  • (02:00) The function statement is just a short-hand for a var statement with a function value
  • (02:35) Functions can be defined inside of other functions
  • (03:08) JavaScript has closures
  • (07:29) There are four ways to call a function: function form, method form, constructor form and apply form
  • (10:00) this is an extra parameter. Its value depends on the calling form
  • (10:30) When a function is invoked, in addition to its parameters, it also gets a special parameter called arguments
  • (11:53) Built-in types can be augmented through (Object|Array|Function|Number|String|Boolean).prototype
  • (14:09) The typeof prefix operator returns 'object' for Array and null types
  • (15:23) eval() is the most misused feature of the language
  • (21:57) In web browsers, the global objects is the window object
  • (22:51) Use of the global namespace must be minimized
  • (23:11) Any variable which is not properly declared is assumed to be global by default
  • (23:45) JSLint is a tool which helps identify weaknesses
  • (24:21) Every object is a separate namespace, use an object to organize your variables and functions
  • (27:15) Function scope can create an encapsulation

Points that caught my attention in JavaScript Video Lecture Part IV

  • (03:30) The language definition is neutral on threads
  • (11:20) When the compiler sees an error, it attempts to replace a nearby linefeed with a semicolon and try again
  • (12:51) Do not use extra commas in array literals. Netscape will tell you that length of [1,2,3,] is 3 while IE will tell it's 4
  • (18:48) Key ideas in JavaScript - Load and go delivery, loose typing, objects as general containers, prototypal inheritance, lambda, linkage through global variables

These four lectures gave me much better theoretical understanding of JavaScript but just a little better practical skills. I should do a project entirely in JavaScript to become more skillful.

I can't wait to see the Advanced JavaScript lectures. At the end of the 4th lecture Douglas said that they will continue with theory of DOM which I will follow and only then continue with Advanced JS.

Points that caught my attention in The Theory of the DOM Part I

  • (03:31) A scripted web browser, Netscape Navigator 2, was first introduced in 1995
  • Best thing happened to have standards work was Mozilla abandoning the Netscape layer model in favor of the W3C model
  • (10:03) List of browsers Yahoo! wants their JavaScript library to run on are FireFox 1.5, FireFox 2.0, Safari 2, Internet Explorer 6, IE 7, Opera 9
  • (12:11) The <script> tag first appeared in Netscape Navigator 2
  • (13:05) <!-- --> comment around script was a Netscape 2 hack for Mosaic and Navigator 1.0
  • (14:51) W3C deprecated language=javascript attribute in script tags, don't put it there anymore
  • (18:48) If you call document.write before onload it inserts data into the document, if you call it after, it replaces the document with the new stuff
  • (20:25) name= is used to identify values in form data and to identify a window or frame
  • (20:45) id= is used to uniquely identify an element so that you could get access to it
  • (20:59) Microsoft introduced document.all as a super-collection of all elements with name or id
  • (21:39) W3C instead said use document.getElementById(id) and document.getElementsByName(name)
  • (23:41) Document tree structure is different for IE than for other browsers because Microsoft decided to depart from W3C standard and not to include whitespaces as text nodes in the tree
  • (25:02) document.body gets you to body node of the tree, document.documentElement gets you to the html root tag of the tree

After watching this lecture I decided to add time where each point that caught my attention happened so that if anyone is interested in any of the points he/she could just fast forward to that place in the video. Eventually I will go through the videos up to this one once more and add timestamps.

Points that caught my attention in The Theory of the DOM Part II

  • (04:32) The guy designing CSS chose a not so appealing names to a programmer for CSS style names. JavaScript guys converted those to camel case in JavaScript which is probably the least compatible with CSS style names
  • (08:31) Replacing a child is done with "java oriented, model based, nothing in common with reality sort of api" through old.parentNode.replaceChild(new, old) where you specify old twice
  • (09:03) It is important to remove any event handlers from the object before you delete it
  • (10:10) Microsoft and their Internet Explorer were the first to realize that it is convenient to provide access to HTML parser and provided innerHTML property which can be assigned a string containing HTML directly
  • (10:50) There is no standard describing innerHTML property
  • (12:12) The browser has an event-driven, single-threaded, asynchronous programming model
  • (12:55) There are three ways to adding event handlers - classic mode (node["on" + type] = func), Microsoft mode (node.attachEvent("on" + type, func)) and W3C mode (node.addEventListener(type, func, bool))
  • (14:50) Microsoft does not send an event parameter, they use the global event object instead.
  • (15:58) There are two ways how events are handled - trickling and bubbling
  • (17:23) The extra bool parameter in W3C mode of adding event handlers node.addEventListener(type, func, bool) tells whether the events are processed bottom up (bubbling) or top down (trickling)

Points that caught my attention in The Theory of the DOM Part III

  • (01:26) Hugest memory leaks happen in IE 6
  • (01:33) Because of that you must explicitly remove event handlers from nodes before deleting or replacing them
  • (06:49) self, parent and top are aliases of window object
  • (08:10) A script can access another window if and only if document.domain === otherwindow.document.domain
  • (10:10) There are three ways to get cross browser compatibility - browser detection, feature detection, platform libraries
  • (11:20) Internet Explorer 1.0 identified itself as "Internet Explorer" but many sites refused to serve the contents complaining that it was not "Mozilla" so in version 1.5 IE identifies itself as "Mozilla"
  • (12:16) Browser detection cross compatibility is the least recommended way
  • (15:37) Platform library cross compatibility is the most recommended way
  • (15:35) Platform library cross compatibility is the most recommended way
  • (18:48) No browser completely implements the standards and much of the DOM is not in any standards. If there was a 100% standards compliant browser, it would not work!
  • (19:19) When programming DOM: 1) do what works; 2) do what's common; 3) do what's standard

Okay, I watched the whole Theory of DOM course and have gained good theoretical knowledge but basically no practical skills. To get better with the JavaScript and DOM will require me to do some interesting practical projects with both of these guys.

Now I am off to watch Advanced JavaScript lectures and then the remaining.

Points that caught my attention in Advanced JavaScript Part I

  • (01:20) In prototypal inheritance objects inherit directly from objects, there are no classes
  • (01:30) An objects contains a "secret link" to another object. Mozilla calls it __proto__
  • (03:36) If looking for a member fails, the last object searched is Object.prototype
  • (07:50) When functions are designed to be used with new, they are called constructors
  • (08:13) new Constructor() returns a new object with a link to Constructor.prototype
  • (09:40) Always have your constructors named with a capital letter so you at least develop reflex for putting a new in front of it
  • (09:48) Forgetting the new still makes code work but it does not construct a new object. This is considered one of the language design errors
  • (10:00) When a new function object is created, it is always given a prototype member
  • (10:49) "Differential inheritance" is a special form of inheritance where you specify just the changes from one generation of objects to the next
  • (17:24) JavaScript doesn't have an operator which makes a new object using an existing object as its prototype, so we have to write our own function
  • (18:50) A "public method" is a function that uses this to access its object
  • (21:55) Functions can be used to create module containers
  • (25:01) "Privileged methods" are functions that have access to "secret" information and they are constructed through closures
  • (28:05) "Parasitic inheritance" is a way of creating an object of an augmented version an existing object.

Points that caught my attention in Advanced JavaScript Part II

  • (06:30) Pseudoclassical patterns are less effective than prototypal patterns or parasitic patterns
  • (09:06) Inner functions do not have access to this
  • (09:32) In JavaScript 1.0 there were no arrays
  • (10:31) When arrays got added, arguments object was forgot to be converted to Array object and it continues to be an array like object
  • (15:47) There are debuggers for IE - Microsoft Script Debugger, which is bad and two debuggers built in Visual Studio and Office 2003. Mozilla has Venkman and Firebug. Safari has Drosera
  • (17:30) funny instruction on how to get debugger working with Office 2003 :D
  • (24:29) All implementations of JavaScript have non-standard debugger statement which cause a breakpoint if there is a dubgger present

Points that caught my attention in Advanced JavaScript Part III

  • (04:20) Array join() method is much faster for concatenating large set of strings than using operator +
  • (07:03) Just have the server gzip the JavaScript source file to minimize the load times, avoid tools which can introduce bugs such as minificators and obfuscators
  • (07:19) JSON

The advanced JavaScript lectures provided lots of idioms and patterns used in the language. I did not do much experimentation and have really grasped just the concepts and overall structure of these advanced concepts.

Now I am going to watch "Advancing JavaScript with Libraries" by John Resig, creator of the JQuery JavaScript library and author of Pro JavaScript Techniques, is a Mozilla technologist focused on the relationship between Mozilla and the world of JavaScript libraries.

Interesting points from Advancing JavaScript with Libraries Part I

  • (08:20) In IE7 basically the only change to JavaScript was to XmlHTTPRequest object
  • (25:14) There are two standards for querying the DOM document - XPath and CSS 3 selectors
  • (26:13) IE doesn't have very good CSS selector support, because of that users have been using the very minimum of CSS selectors which almost equates CSS 1
  • (27:30) jQuery allows selecting elements from the DOM by using CSS 3 selectors which is done in one line of jQuery code instead of 20 - 25 lines of just JavaScript DOM code

Interesting points from Advancing JavaScript with Libraries Part II

  • (05:15) Users are expecting the DOM selectors to behave more like CSS, that is like when a new CSS selector is added, it propagates to all the elements affected. The users expect the same to happen when a chunk of HTML is added to the DOM, that the handlers get added to them without re-running any code
  • (12:30) Object Relational Mappings
  • (14:50) Libraries create new patterns on top of existing APIs

There were not that many points that got me interested because it was pretty obvious stuff. It was just interesting to see that the DOM is not perfect and there are many bugs which one or the other browser fails, why they fail and how to solve these DOM related problems. Also typical programming meta-problems were discussed such as JavaScript trying to get elements before browser has loaded the DOM, how the white spaces are treated and what methods to use for navigating the DOM. Later the lecture went on how to query the DOM tree using jQuery and mix of XPath and CSS 3. Then it is discussed how injecting HTML in an existing document is done, what's tricky about it and what problems can arise. Finally the lecture continues with FUEL and object relational mappings.

The other lecture I watched was "Maintainable JavaScript" by Nicholas Zakas. He is an engineer on the team that brings you My Yahoo!, one of the most popular personalized portals on the web. He is also the author of two books on frontend engineering, including "Professional JavaScript for Web Developers," one of the best tomes of its kind.

Interesting points from Maintainable JavaScript

  • (01:15) It is estimated that as much as 80% of time is spent maintaining existing code
  • (04:30) Maintainable code is understandable, intuitive, adaptive, extendable and debuggable
  • (16:20) There are three layers on the client side - JavaScript for behavior, CSS for presentation and HTML for structure
  • (23:42) Programming practices
  • (26:30) Namespace your objects
  • (32:10) Avoid null comparison, use instanceof or typeof operators
  • (37:15) Write code in separate JavaScript files and use a build process to combine them
  • (40:47) Summary of writing maintainable code - code conventions, loose coupling, programming practices and build process

There are not that many interesting points anymore because most of the stuff has been covered in the previous lectures. Apart from that these lecture did not teach me much new because this stuff was pretty obvious. The only point to watch this lecture is to refresh all these obvious suggestions - agree on indentation and naming conventions in your team, comment difficult algorithms and large sections of code, don't write obvious comments, comment hacks, loose coupling, careful use of complex code and design patterns, etc.

What do you think about these lectures?

youtube video downloader in vbscriptWhen I was a small kid I liked RAD (Rapid Application Development) in Microsoft® Visual Basic® programming language. I mastered the language entirely from MSDN's samples and it was such a joy to create GUI applications in a few hours or days and they just worked! Now it has been a few years years since I have not touched Visual Basic and I have become really rusty in this language.

If you have been reading my blog, you might have noticed that I have been creating these YouTube video downloaders in various programming languages. I have no plans to create a downloader in Visual Basic itself, because I am creating a GUI downloader in C and C++ already but I would like to use a derived language (a subset of Visual Basic language), VBScript, to create the downloader.

That would refresh my knowledge of Visual Basic a little and also let me learn a little more about programming Windows Script Host (WSH).

Here is a quote from a really good introduction to what WSH is:

The first time people encounter Windows Script Host, they often express some confusion. What exactly is WSH? Is it a language, like VBScript or JScript? No; although WSH enables you to run programs written in these languages, it is not a language itself. WSH is a script host. A script host is a program that provides an environment in which users can execute scripts in a variety of languages, languages that use a variety of object models to perform tasks.

You are probably already familiar with other script hosts. Microsoft® Internet Explorer, for example, enables users to execute scripts that use the Dynamic HTML object model. Shell programs (such as C Shell, Bourne Shell and Korn Shell) enable you to write scripts that use an object model capable of manipulating the file system. Even the command prompt can be thought of as a scripting environment because it can run scripts written in the "batch file" language.

WSH is an unusual script host in that it was designed to be general-purpose. Unlike most of the scripting tools mentioned above, WSH imposes restrictions on neither the language used to write scripts nor the object models used by scripts.

WSH is ideal for non-interactive scripting needs such as logon scripting and administrative scripting but we will use it to download YouTube videos as well.

The key advantage of this script is that it will run on any Windows operating system newer than Windows 98. If you have Windows 98 or an older Windows, follow this link and install the latest version of WSH (unfortunately version 5.1 (5.6 is the latest))!

Let's define the interface of the script.

The script can either be run in WScript environment or CScript environment. WScript is the system default on Windows 2000. Characteristics of this environment are that output appears in a pop-up window, and script execution waits until the user clears the pop-up window by pressing OK button. The other environment is the CScript environment. The primary difference between the CScript and WScript environments is that the CScript environment directs display information to the command window, which has a side effect of letting a script run to completion without pausing for each message sent to the UI. I want this script to be usable from both of the environments.

Here is what I am thinking. By default I want this script to be accepting the video URL to download as the first command line argument (followed by more videos as next arguments). If the first argument is not provided and it is run in WScript environment I want the script to pop up an InputBox dialog asking for the URL of the video to download. Otherwise, if it is run in CScript environment, I want it to quit.

How do we find in which environment the script is being run? I had no idea, so I turned to Google Groups and searched for "cscript wscript detect" and found this post which explained how to do it with this snippet of code:

If "CSCRIPT.EXE" = UCase(Right(WScript.Fullname, 11)) Then
    WScript.Echo "The program was run by CScript!"
End If 

The next thing we need to figure out is how to get the command line arguments of a running script.

Turns out that each WSH script has a WScript object available without the script needing to bind to the object. Command-line arguments are stored in the WshArguments collection, which you access through the Arguments property of the WScript object.

Here is a diagram of WScript object's methods, properties and WShArguments expanded:

wsh wscript object's hierarcy

The easiest way to loop over all arguments is shown in this snippet:

Set objArgs = WScript.Arguments
For I = 0 to objArgs.Count - 1
   WScript.Echo objArgs(I)
Next

Now just instead of "WScript.Echo objArgs(I)" we call "DownloadVideo objArgs(I)", where DownloadVideo is our procedure for downloading YouTube videos.

In one of the previous articles, downloading youtube videos with awk programming language, I explained how the embedded YouTube flash video player retrieves the video file itself. Please see that article if you are interested in how I figured it out.

Last two things we have to figure out before we have a running script is how to talk over HTTP with VBScript and how to save the incoming binary data to a file.

Both VBScript and WScript object provide CreateObject function (but there is a difference between them) which allows binding to COM objects.

There is a "Microsoft.XmlHttp" COM Object which is shipped with Internet Explorer 5.0 and later. This object is what actually provides the well known AJAX interface used in Web2.0 applications - the XMLHttpRequest interface. Here at MSDN is the documentation of this interface with all the methods and properties it provides.

When we get the video, we will be dealing with binary data. Microsoft Scripting Runtime provides us with FileSystemObject (FSO) but unfortunately it is not suitable for writing binary files.

There is a way to write binary files with FSO but it is so slow that I dropped this solution. It took more than 10 minutes to write 1MB of data!

' Given a FileName and Data, saves Data to file named FileName
Sub SaveData(FileName, Data)
    Dim Fso: Set Fso = CreateObject("Scripting.FileSystemObject")
    Dim TextStream: Set TextStream = Fso.CreateTextFile(FileName, True)

    WScript.Echo LenB(Data)
    TextStream.Write BinaryToString(Data)
End Sub

' Given Binary data, converts it to a string
Function BinaryToString(Binary)
  Dim I, S
  For I = 1 To LenB(Binary)
    S = S & Chr(AscB(MidB(Binary, I, 1)))
  Next
  BinaryToString = S
End Function

A much better way to write binary data to a file with VBScript is using ADODO.Stream Object which deals with binary data out of the box. Look at SaveVideo function in my final program to see how it is used to write binary data to file.

Script Usage

Before the script can be used, you have to tell your computer to trust youtube.com domain. If you do not do this, executing the script will lead you to the following error:

ytdown.vbs(73, 5) msxml3.dll: Access is denied.

Youtube.com domain can be trusted by adding it to trusted sites security zone in Internet Explorer.
Launch your Internet Explorer browser and head to Tools -> Internet Options, then follow the steps illustrated in this image:

internet explorer trusted sites zone security

Once you have trusted youtube.com domain, you can start downloading the videos.

One of the ways is to double click the ytdown.vbs icon which will launch the script in WScript environment and an input dialog will appear asking for a video to download:

launch ytdown.vbs by double clicking the icon

After you press "OK", the downloader will save the video to a file with the title of the video and .flv extension in the same directory!

The other way to download a video is to call it via command line in CScript environment:

C:\ytdown>cscript ytdown.vbs "http://www.youtube.com/watch?v=h9MN2mKGZoo"
Microsoft (R) Windows Script Host Version 5.6
Copyright (C) Microsoft Corporation 1996-2001. All rights reserved.

Downloading video '30 fps, 30 frames (1 second)'
Done!

Happy downloading!

The downloaded .flv file can be converted to a better format like DivX or .avi.
Read about converting a video to a better format in this article.

Here is the final script:

'
' Peteris Krumins (peter@catonmat.net)
' http://www.catonmat.net  -  good coders code, great reuse
'
' 2007.08.03 v1.0 - initial release
' 2007.10.21 v1.1 - youtube changed the way it displays vids
' 2008.03.01 v1.2 - youtube changed the way it displays vids
' 2009.12.02 v1.3 - youtube changed the way it displays vids
'
Option Explicit

Dim WscriptMode

' Detect if we are running in WScript or CScript
If UCase(Right(WScript.Fullname, 11)) = "WSCRIPT.EXE" Then
    WScriptMode = True
Else 
    WScriptMode = False
End If 

Dim Args: Set Args = WScript.Arguments

If Args.Count = 0 And WScriptMode Then
    ' If running in WScript and no command line args are provided
    ' ask the user for a URL to the YouTube video
    Dim Url: Url = InputBox("Enter a YouTube video URL to download" & vbCrLf & _
                   "For example, http://youtube.com/watch?v=G1ynTV_E-5s", _
                   "YouTube Downloader, http://www.catonmat.net")
    If Len(Url) = 0 Then: WScript.Quit 1
    DownloadVideo Url
ElseIf Args.Count = 0 And Not WScriptMode Then
    ' If running in CScript and no command line args are provided
    ' show the usage and quit
    WScript.Echo "Usage: " & WScript.ScriptName & " <video url 1> [video url 2] ..."
    WScript.Quit 1
Else 
    ' Download all videos
    Dim I

    For I = 0 to args.Count - 1
        DownloadVideo args(I)
    Next
End If

' Downloads a YouTube video and saves it to a file
Sub DownloadVideo(Url)
    Dim Http, VideoTitle, VideoName, Req

    Set Http = CreateObject("Microsoft.XmlHttp")
    Http.open "GET", Url, False
    Http.send

    If Http.status <> 200 Then
        WScript.Echo "Failed getting video page at: " & Url & vbCrLf & _
                     "Error: " & Http.statusText
        Exit Sub
    End If

    Dim VideoId: VideoId = ExtractMatch(Url, "v=([A-Za-z0-9-_]+)")
    If Len(VideoID) = 0 Then
        WScript.Echo "Could not extract video ID from " & Url
        Exit Sub
    End If

    VideoTitle = GetVideoTitle(Http.responseText)
    If Len(VideoTitle) = 0 Then
        WScript.Echo "Failed extracting video title from video at URL: " & Url & vbCrLf & _
                     "Will use the video ID '" & VideoID & "' for the filename."
        VideoName = VideoID
    Else
        VideoName = VideoTitle
    End If

    Dim FmtMap: FmtMap = GetFmtMap(Http.responseText)
    If Len(FmtMap) = 0 Then
        WScript.Echo "Could not extract fmt_url_map from the video page."
        Exit Sub
    End If

    Dim VideoURL: VideoURL = Find_Video_5(FmtMap)
    If Len(VideoURL) = 0 Then
        WScript.Echo "Could not extract fmt_url_map from the video page."
        Exit Sub
    End If

    If WScriptMode = False Then: WScript.Echo "Downloading video '" & VideoName & "'"
    Http.open "GET", VideoURL, False
    Http.send

    If Http.status <> 200 Then
        WScript.Echo "Failed getting the flv video: " & Url & vbCrLf & _
                     "Error: " & Http.statusText
        Exit Sub
    End If

    Dim SaneFilename
    SaneFilename = MkFileName(VideoName)

    SaveVideo SaneFilename, Http.ResponseBody
    WScript.Echo "Done downloading video. Saved to " & SaneFilename & "."
End Sub

' Given fmt_url_map, url-escapes it, and finds the video url for video
' with id 5, which is the regular quality flv video.
Function Find_Video_5(FmtMap)
    FmtMap = Unescape(FmtMap)
    Find_Video_5 = ExtractMatch(FmtMap, ",?5\|([^,]+)")
End Function

' Given YouTube Html page, extract the fmt_url_map parameter that contains
' the URL to the .flv video
Function GetFmtMap(Html)
    GetFmtMap = ExtractMatch(Html, """fmt_url_map"": ""([^""]+)""")
End Function

' Given YouTube Html page, the function extracts the title from <title> tag
Function GetVideoTitle(Html)
    ' get rid of all tabs
    Html = Replace(Html, Chr(9), "")

    ' get rid of all newlines (vbscript regex engine doesn't like them)
    Html = Replace(Html, vbCrLf, "")
    Html = Replace(Html, vbLf, "")
    Html = Replace(Html, vbCr, "")

    GetVideoTitle = ExtractMatch(Html, "<title>YouTube ?- ?([^<]+)<")
End Function

' Given the Title of a video, function creates a usable filename for a video by
' sanitizing it - stripping parenthesis, changing non alphanumeric characters
' to _ and adding .flv extension
Function MkFileName(Title)
    Title = Replace(Title, "(", "")
    Title = Replace(Title, ")", "")

    Dim Regex
    Set Regex = New RegExp
    With Regex
        .Pattern = "[^A-Za-z0-9-_]"
        .Global = True
    End With

    Title = Regex.Replace(Title, "_")
    MkFileName = Title & ".flv"
End Function

' Given Text and a regular expression Pattern, the function extracts
' the first submatch
Function ExtractMatch(Text, Pattern)
    Dim Regex, Matches

    Set Regex = New RegExp
    Regex.Pattern = Pattern

    Set Matches = Regex.Execute(Text)
    If Matches.Count = 0 Then
        ExtractMatch = ""
        Exit Function
    End If

    ExtractMatch = Matches(0).SubMatches(0)
End Function

' Function saves Data to FileName
Function SaveVideo(FileName, Data)
  Const adTypeBinary = 1
  Const adSaveCreateOverWrite = 2
  
  Dim Stream: Set Stream = CreateObject("ADODB.Stream")
  
  Stream.Type = adTypeBinary
  Stream.Open
  Stream.Write Data
  Stream.SaveToFile FileName, adSaveCreateOverWrite
End Function

'
' ==========================================================================
' The following code saves binary data to file using FileSystemObject
' It is so slow that even on a 3.2Ghz computer saving 1 MB takes 10 minutes!
' Don't use it! I put it here just to illustrate the wrong solution!
' ==========================================================================
'

' Given a Filename and Data, the function saves Data to File
'Sub SaveVideo(File, Data)
'    Dim Fso: Set Fso = CreateObject("Scripting.FileSystemObject")
'    Dim TextStream: Set TextStream = Fso.CreateTextFile(File, True)
'
'    WScript.Echo LenB(Data)
'    TextStream.Write BinaryToString(Data)
'End Sub

' Given Binary data, converts it to a string
'Function BinaryToString(Binary)
'  Dim I, S
'  For I = 1 To LenB(Binary)
'    S = S & Chr(AscB(MidB(Binary, I, 1)))
'  Next
'  BinaryToString = S
'End Function


'
' ==========================================================================
' The following is an implementation of UrlUnescape. It turned out VBScript
' has Unescape() function built in already, that does it!
'
'Function UrlUnescape(Str)
'    Dim Regex, Match, Matches
'
'    Set Regex = New RegExp
'    With Regex
'        .Pattern = "%([0-9a-f][0-9a-f])"
'        .IgnoreCase = True
'        .Global = True
'    End With
'    ' Wanted to do this, but it wasn't quite possible
'    ' UrlUnescape = Regex.Replace(Str, Chr(CInt("&H" & $0)))
'
'    Set Matches = Regex.Execute(Str)
'    For Each Match in Matches
'        Str = Replace(Str, Match, Chr(CInt("&H" & Match.SubMatches(0))))
'    Next
'
'    UrlUnescape = Str
'End Function

Download Visual Basic Script YouTube Video Downloader

Download link: vbscript youtube video downloader
Total downloads: 21633 times

If you notice any bugs in this script, have any recommendations, want to criticize my code, want to thank me, then just post a comment on this article!

ps. remember the ILOVEYOU virus? It was also written in VBScript, that alone indicates that this language is worth knowing.

This article is part of the article series "GUI Video Downloader."
<- previous article next article ->

video downloader software, vim, mingw, wxwidgetsThis is a continuation of multi-part series on how to create a cross-platform desktop video downloading and converting application. In part I we prepared our coding environment with vim, various vim plugins, mingw and gdb. In this part we will start creating a C library for downloading just the YouTube videos with modularity in mind, so that we could create a nice desktop application capable of downloading videos from other video sharing sites like break, metacafe, dailymotion and others. Complementary to this library we will be also be creating a command line application which will use this C library. The command line application will serve us as a test tool for the library, so we knew the library worked ok. Once we have these two components, we will build GUI around the library and create the desktop app.

Writing the Library

I have chosen a name libvd for the library which stands for video download library.

The library will be written in C programming language because:

  • it is the number one cross platform language,
  • I haven't programming in C for a while, so it will be a great refresher,
  • the GUI framework which I will be using is written in C++ which will have no problems with using C code
  • the application will be small and will require no unnecessary library/toolkit dependencies

Downloading a video means working with HTTP protocol. Let's use "good coders code, great reuse" approach and seek for a C library which would ease the usage of HTTP protocol.

Having worked on a few projects in C programming language involving HTTP protocol, I know an excellent library. It's called libcurl. I even implemented cookie interface for this library which was missing two years ago.

What is libcurl? Quoting the author of libcurl: "libcurl is a free and easy-to-use client-side URL transfer library, supporting FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, FILE and LDAP. libcurl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, Kerberos4), file transfer resume, http proxy tunneling and more!

libcurl is highly portable, it builds and works identically on numerous platforms, including Solaris, NetBSD, FreeBSD, OpenBSD, Darwin, HPUX, IRIX, AIX, Tru64, Linux, UnixWare, HURD, Windows, Amiga, OS/2, BeOs, Mac OS X, Ultrix, QNX, OpenVMS, RISC OS, Novell NetWare, DOS and more..."

Woah! Much more than we need! And notice how cross platform it is! Exactly what we are looking for!

Let's download the libcurl library. Since I am developing the program on a fresh Windows XP virtual machine, I chose to download curl-7.16.4.zip version of the library. It really does not matter what operating system I am developing this program on, I could have as well chosen Linux, then I would have downloaded the library ending with extension .gz or .bz2.

download libcurl for windows

Compiling libcurl

Once we have downloaded libcurl, let's unpack it and look for instructions on how to compile it with MinGW on a Windows machine. I chose c:\vd as the root directory for the project on the windows machine because I am afraid of those whitespace characters windows has in path. I unpacked the libcurl library to c:\vd\curl.

At this point we have no idea how to build the libcurl on Windows, right? So let's look in the docs subdirectory (C:\vd\curl\docs) for a clue. Usually the instructions on how to build a library on a particular platform can be found in files like README.platform or INSTALL.platform or just INSTALL. Indeed, libcurl has a file README.win32. Let's read it.

libcurl's win32 readme file

Oops, it says to read README first, we look in the same docs subdirectory but there is no README! We look in the parent directory, whew, yeah there is a README. But it is just 1KB in size and says nothing...

The only reasonable file left is INSTALL, let's look at it. Oh yeah, it's 29K in size! Must be it!

Indeed, scrolling a few screens down by pressing vim's Ctrl_D, we find instructions on how to build libcurl on win32 with MinGW compiler!

libcurl's win32 install file with mingw

Libcurl also provides support for compressed HTTP content with zlib, and secure connections over HTTPS with OpenSSL. The INSTALL file mentions what to do to get support for these thingies. We do not need them, so we ignore them.

Don't rush yet compiling the library! We also want our application to be small, not just portable. Libcurl has support for so many protocols but our project need just the HTTP protocol!

Looking a little further in the INSTALL file, we find the following instruction:

   Disabling Specific Protocols in Win32 builds
   --------------------------------------------

   The configure utility, unfortunately, is not available for the Windows
   environment, therefore, you cannot use the various disable-protocol
   options of the configure utility on this platform.

   However, you can use the following defines to disable specific
   protocols:

   HTTP_ONLY             disables all protocols except HTTP
   [...]

   If you want to set any of these defines you have the following
   possibilities:

   - Modify lib/setup.h
   [...]

We really want just the HTTP protocol, so let's follow this instruction and modify setup.h file in lib directory (file C:\vd\curl\lib\setup.h) and add the following line right after the comments:

#define HTTP_ONLY

The file should now look like this:

modifying libcurl to make it support HTTP only!

Okay, now we are ready to compile libcurl!

The INSTALL file says that to compile the libcurl library we first need to run 'mingw32.bat' and then 'make mingw32'.

   MingW32
   -------

   Run the 'mingw32.bat' file to get the proper environment variables set,
   then run 'make mingw32' in the root dir. Use 'make mingw32-ssl' to build
   curl SSL enabled.

mingw32.bat? I don't have such file in my mingw32 distribution! Now what? Let's be real developers and compile the library ourselves then. Let's make sure the MinGW's bin directory (in my case C:\MinGW\bin) is in our PATH environment variable and let's run the second command - 'make mingw32'.

Oops:

C:\vd\curl>make mingw32
'make' is not recognized as an internal or external command,
operable program or batch file.

There is no program named 'make' in MinGW's distribution, there is only mingw32-make.exe, let's run it:

C:\vd\curl>c:\MinGW\bin\mingw32-make.exe mingw32
c:/MinGW/bin/mingw32-make -C lib -f Makefile.m32 ZLIB=1
mingw32-make[1]: Entering directory `C:/vd/curl/lib'
gcc -I. -I../include -I"../../zlib-1.2.3" -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -DHAVE_LIBZ -DHAVE_ZLIB_H -c file.c
gcc -I. -I../include -I"../../zlib-1.2.3" -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -DHAVE_LIBZ -DHAVE_ZLIB_H -c timeval.c
gcc -I. -I../include -I"../../zlib-1.2.3" -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -DHAVE_LIBZ -DHAVE_ZLIB_H -c base64.c
In file included from base64.c:43:
urldata.h:92:59: zlib.h: No such file or directory
In file included from base64.c:43:

Huh? I never wanted my libcurl to be built with zlib support. Let's get rid of it by editing the Makefile (C:\vd\curl\Makefile) directly. Search for mingw32 target (string "mingw32:") and you'll find:

mingw32:
	$(MAKE) -C lib -f Makefile.m32 ZLIB=1
	$(MAKE) -C src -f Makefile.m32 ZLIB=1

We really don't want to build the curl executable, so let's get rid of the last line and also get rid of "ZLIB=1" variable definition.
The lines should look now like this:

mingw32:
	$(MAKE) -C lib -f Makefile.m32

Now let's run the mingw32-make again:

C:\vd\curl>c:\MinGW\bin\mingw32-make.exe mingw32
c:/MinGW/bin/mingw32-make -C lib -f Makefile.m32
mingw32-make[1]: Entering directory `C:/vd/curl/lib'
gcc -I. -I../include -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -c base64.c
gcc -I. -I../include -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -c hostip.c
[...]
rm -f libcurl.a
process_begin: CreateProcess(NULL, rm -f libcurl.a, ...) failed.
make (e=2): The system cannot find the file specified.
mingw32-make[1]: *** [libcurl.a] Error 2
mingw32-make[1]: Leaving directory `C:/vd/curl/lib'
mingw32-make: *** [mingw32] Error 2

Argh! There is no 'rm' on Windows, at least not with MinGW development tools package. Let's modify the Makefile.m32 file (C:\vd\curl\lib\Makefile.m32) and change 'rm' to Windows 'del' command.

Find this line in Makefile.m32:

RM = rm -f

and change it to

RM = del /F

Perfect! Running the mingw32-make.exe again finishes with no errors! Lookin in the C:\vd\curl\lib directory we find the static library libcurl.a and the dynamic library libcurl.dll. Had we compiled it on Linux, the dynamic library would have been named libcurl.so.

Testing libcurl

I wrote the following test program which outputs the website of google.com to the standard output if libcurl works correctly:

#include <stdio.h>
#include <stdlib.h>
#include "curl/curl.h"

int main(void) {
    CURLcode ret;
    CURL *curl = curl_easy_init();
    if (curl == NULL) {
        fprintf(stderr, "Failed creating CURL easy handle!\n");
        exit(EXIT_FAILURE);
    }

    /* let's get google.com because I love google */
    ret = curl_easy_setopt(curl, CURLOPT_URL, "http://www.google.com");
    if (ret != CURLE_OK) {
        fprintf(stderr, "Failed getting http://www.google.com: %s\n",
                curl_easy_strerror(ret));
        exit(EXIT_FAILURE);
    }

    ret = curl_easy_perform(curl);
    if (ret != 0) {
        fprintf(stderr, "Failed getting http://www.google.com: %s\n",
                curl_easy_strerror(ret));
        exit(EXIT_FAILURE);
    }

    return 0;
}

I placed this program in C:\vd\tests\curltest directory and named the source file testcurl.c. Our curl library is located in C:\vd\curl\lib and the curl header files in C:\vd\curl\include. Assuming out currenly working directory is C:\vd\tests\curltest, the following command line will compile the C source file to an executable:

gcc testcurl.c -o testcurl -DCURL_STATICLIB -I..\..\curl\include -L..\..\curl\lib -lcurl -lwsock32 -lwinmm -Wall

It will create testcurl.exe executable which can be run. Let's try running it:

C:\vd\tests\libcurl>testcurl.exe
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>Google</title>
[...]

Yeah! It works! Now we are ready to write the libvd (video download) library which will use the libcurl for communicating over HTTP protocol.

I'll do this in Part III of this series. Until then, have fun compiling libcurl!

ed text editor and mr. edI have always had historic interest of various UNIX tools and the UNIX itself. One day when I was learning sed, the stream editor, (sed cheat sheet) I got interested how much the most powerful text editing tools on UNIX - 'ed', 'ex', 'sed' and 'vim' (or just 'vi') had in common and what they borrowed from one another.

I was not perfectly sure if 'sed' has originated from 'ed', so I Google searched for 'sed history' which turned up some interesting results, one of them being "The original users manual for sed, by Lee E. McMahon, from the 7th edition UNIX Manual (1978)".

Quoting the manual "Sed is a lineal descendant of the UNIX editor, ed." Great! I knew that 'sed' was non-interactive text editor whereas 'ed' was an interactive one. Also I knew that 'ex' was improved 'ed' and that 'sed' was a non-interactive 'ed' and that 'ex' had become 'vi' and was used in 'vi' command line mode. To boost my vi/vim skills and get a touch on how text editing was done with just 'ed', and satisfy my historical curiosity I set myself a goal to learn it.

As I mentioned in awk and sed cheat sheet posts, the best method for me to learn a new tool is to have all commands in front of me, so when I am doing experimentation I can quickly find the command and also see all the other commands at the same time, blowing them in subconsciousness.

Since 'ed' is line-oriented text editor, it is important to understand the line addressing, this cheat sheet summarizes the line addressing.

Most of 'ed' command are single letter in length. This cheat sheet summarizes all the commands, showing the supported address ranges and a detailed description of each command.

From my experience, once I had completed this cheat sheet and had it in front of me, I picked 'ed' up in 30 minutes. And then spent a few more hours experimenting and trying various constructs.

Good luck with learning ed! :)

And don't forget: ED IS THE TRUE PATH TO NIRVANA!

ps. if you notice any inaccuracies, spelling or grammar errors, please leave a comment here, so I can fix it. Thanks!
pss. logo for this article from mr. ed comedy ;)

Download Ed UNIX Text Editor's Cheat-Sheet

Plain Text (.txt):
Download link: ed text editor cheat sheet (.txt)
Downloaded: 20084 times

PDF:
Download link: ed text editor cheat sheet (.pdf)
Downloaded: 32981 times

Microsoft Word 2000 format (.doc):
Download link: ed text editor cheat sheet (.doc)
Downloaded: 3247 times