reddit media website post iconDuring my usage of reddit, I have observed that many titles have "(Pic)" or "[Picture]", "(Video)", etc. after them. It means that the contents the link points to has a picture or video in it. Sometimes I want to have fun with my friends and go through all the pics or vids. Unfortunately reddit's search is broken and there is really no good way to see the best pics and videos voted on reddit in the past.

I decided to create reddit media site which will monitor reddit's front page, collect picture & video links, and build an archive of them over time.

The site has been launched:
visit reddit media now

In (read more about it on about this blog page) page I wrote about one of the methods I like to use when developing software (and this project requires writing a few tools quickly, more about them below). It is call "the hacker's approach". The hacker's approach method is basically writing software as fast as possible using everything available and not thinking much about the best development practices, and not worrying what others will think about your code. If you are a good programmer the code quality produced is just a bit worse than writing it carefully but the time saved is enormous.

I will release full source code of website with all the programs generating the website. Also I will blog how the tools work and what ideas I used.

Update: Done! The site is up at reddit media: intelligent fun online.

Reddit Media Website's Technical Design Sketch

I use DreamHost shared hosting to run this website. Overall it is great hosting company and I have been with them for more than a year now! Unfortunately since it is a shared hosting, sometimes the server gets overloaded and serving of dynamic pages can become slow (a few seconds to load).

I want the new website to be as fast as possible even when the server is a bit loaded. I do not want any dynamic parsing to be involved when accessing the website. Because of this I will go with generating static HTML pages.

A Perl script will run every 30 mins from crontab, get reddit.com website, extract titles and URLs. Another script will add the titles to the lightweight sqlite on-disk database in case I ever want to make the website dynamic. And the third script will use the entries in the database and generate HTML pages.

Technical Design

A knowledgeable user might ask if this design does not have a race-condition at the moment the new static page is generated and user requesting the same page. The answer is no. The way new pages will be generated is that they will be written to temporary files, then moved in place of the existing ones. The website runs on Linux operating system and by looking up `man 2 rename' we find that

If newpath already exists it will be atomically replaced (subject to a
few conditions - see ERRORS below), so that there is no point at which
another process attempting to access(2,5) newpath will find it missing.

rename system call is atomic which means we have no trouble with race conditions!

Reddit provides RSS feed to the front page news. It has 25 latest news and maybe 5 are media links. That is not enough links to launch the website. People visiting the site will get bored with just 5 links and a few new added daily. I need more content right at the moment I launch the site. Or I could to launch the site later when articles have piled up. Unfortunately, I do not want to wait and I want to launch it ASAP! The hacker's approach!

First, I will create a script which will go through all the pages on reddit looking for picture and video links, and insert the found items in the database. It will match patterns in link titles and will match domains which exclusively contain media.
Here is the list of patterns I could come up with which describe pictures and videos:

  • picture
  • pic
  • image
  • photo
  • comic
  • chart
  • video
  • vid
  • clip
  • film
  • movie

And here are the domains found on youtube which exclusively contain media:

  • youtube.com
  • video.google.com
  • liveleak.com
  • break.com
  • metacafe.com
  • brightcove.com
  • dailymotion.com
  • flicklife.com
  • flurl.com
  • gofish.com
  • ifilm.com
  • livevideo.com
  • video.yahoo.com
  • photobucket.com
  • flickr.com
  • xkcd.com

To write this script I will use LWP::UserAgent to get HTML contents and HTML::TreeBuilder to extract titles and links.

This script will output the found items in human readable format, ready for input to another script which will absorb this information and put it in the SQLite database.

This script is called 'reddit_extractor.pl'. It takes one optional argument which is number of reddit pages to extract links from. If no argument is specified, it goes through all reddit pages until it hits the last one. For example, specifying 1 as the first argument makes it parse just the front page. I can now run this script periodically to find links on the front page. No need for parsing RSS.

There is one constant in this script which can be changed. This constant, VOTE_THRESHOLD, sets the threshold of how many votes a post on reddit should have received to be collected by our program. I had to add it because when digging in older reddit's posts, media with 1 or 2 votes can be found which means it really wasn't that good.

The script outputs each media post matching a pattern or domain in the following format:

title (type, user, reddit id, url)
  • title is the title of the article
  • type is the media type. It can be one of 'video', 'videos', 'picture', 'pictures'. It's plural if the title contains "pics" or "videos" (plural) form of media.
  • user is the reddit user who posted the link
  • reddit id is the unique identifier reddit uses to identify its links
  • url is the url to the media

Script 'reddit_extractor.pl' can be viewed here:
reddit extractor (perl script, reddit media generator)

Then I will create a script which takes this input and puts it into SQLite database. It is so trivial that there is nothing much to write about it.

This script will also be written in Perl programming langauge and will use just DBI and DBD::SQLite modules for accessing the SQLite database.

The script will create an empty database on the first invocation, read the data from stdin and insert the data in the database.

The database design is dead simple. It contains just two tables:

  • reddit which stores the links found on reddit, and
  • reddit_status which contains some info about how the page generator script used the reddit table

Going into more details, reddit table contains the following colums:

  • id - the primary key of the table
  • title - title of the media link found on reddit
  • url - url to the media
  • reddit_id - id reddit uses to identify it's posts (used by my scripts to link to comments)
  • user - username of the person who posted the link on reddit
  • type - type of the media, can be: 'video', 'videos', 'picture', 'pictures'. It's plural if the title contains "pics" or "videos" (plural) form of media.
  • date_added - the date the entry was added to the database

The other table, reddit_status contains just two colums:

  • last_id - the last id in the reddit table which the generator script used for generating the site
  • last_run - date the of last successful run of the generator script

This script is called 'db_inserter.pl'. It does not take any arguments but has one constant which has to be changed before using. This constant, DATABASE_PATH, defined the path to SQLite database. As I mentioned, it is allowed for the database not to exist, this script will create one on the first invocation.

These two scripts used together can now be periodically run from crontab to monitor the reddit's front page and insert the links in the database. It can be done with as simple command as:

reddit_extractor.pl 1 | db_inserter.pl

Script 'db_inserter.pl' ca be viewed here:
db inserter (perl script, reddit media generator)

Now that we have our data, we just need to display it in a nice manner. That's the job of generator script.

The generator script will be run after the previous two scripts have been run together and it will use information in the database to build static HTML pages.

Since generating static pages is computationally expensive, the generator has to be smart enough to minimize regeneration of already generated pages. I commented the algorithm (pretty simple algorithm) that minimizes regeneration script carefully, you can take a look at 'generate_pages' function in the source.

The script generates three kinds of pages at the moment - pages containing all pictures and videos, pages containing just pictures and pages containing just videos.

There is a lot of media featured on reddit and as the script keeps things cached, the directory sizes can grow pretty quickly. If a file system which performs badly with thousands of files in a single directory is used, the runtime of the script can degrade. To avoid this, the generator stores cached reddit posts in subdirectories based on the first char of their file name. For example, if a filename of a cached file is 'foo.bar', then it stores the file in /f/foo.bar directory.

The other thing this script does is locate thumbnail images for media. For example, for YouTube videos, it would construct URL to their static thumbnails. For Google Video I could not find a public service for easily getting the thumbnail. The only way I found to get a thumbnail of Google Video is to get the contents of the actual video page and extract it from there. The same applies to many other video sites which do not tell developers how to get the thumbnail of the video. Because of this I had to write a Perl module 'ThumbExtractor.pm', which given a link to a video or picture, extracts the thumbnail.

'ThumbExtractor.pm' module can be viewed here:
thumbnail extractor (perl module, reddit media generator)

Some of the links on reddit contain the link to actual image. I wouldn't want the reddit media site to take long to load, that's why I set out to seek a solution for caching small thumbnails on the server the website is generated.

I had to write another module 'ThumbMaker.pm' which goes and downloads the image, makes a thumbnail image of it and saves to a known path accessible from web server.

'ThumbMaker.pm' module can be viewed here:
thumbnail maker (perl module, reddit media generator)

To manipulate the images (create thumbnails), the ThumbMaker package uses Netpbm open source software.

Netpbm is a toolkit for manipulation of graphic images, including conversion of images between a variety of different formats. There are over 300 separate tools in the package including converters for about 100 graphics formats. Examples of the sort of image manipulation we're talking about are: Shrinking an image by 10%; Cutting the top half off of an image; Making a mirror image; Creating a sequence of images that fade from one image to another.

You will need this software (either compile yourself, or get the precompiled packages) if you want to run the the reddit media website generator scripts!

To use the most common image operations easily, I wrote a package 'Netpbm.pl', which provides operations like resize, cut, add border and others.
'Netpbm.pm' package can be viewed here:
netpbm image manipulation (perl module, reddit media generator)

I hit an interesting problem while developing the ThumbExtractor.pm and ThumbMaker.pm packages - what should they do if the link is to a regular website with just images? There is no simple way to download the right image which the website wanted to show to users.
I thought for a moment and came up with an interesting but simple algorithm which finds "the best" image on the site.
It retrieve ALL the images from the site and find the one with biggest dimensions and make a thumbnail out of it. It is pretty obvious, pictures posted on reddit are big and nice, so the biggest picture on the site must be the one that was meant to be shown.
A more advanced algorithm would analyze it's location on the page and add weigh to the score of how good the image is, depending on where it is located. The more in the center of the screen, the higher score.

For this reason I developed yet another Perl module called 'ImageFinder.pm'. See the 'find_best_image' subroutine to see how it works!

'ImageFinder.pm' module can be viewed here:
best image finder (perl module, reddit media generator)

The generator script also uses CPAN's Template::Toolkit package for generating HTML pages from templates.

The name of the generator script is 'page_gen.pl'. It takes one optional argument 'regenerate' which if specified clears the cache and regenerates all the pages anew. It is useful when templates are updated or changes are made to thumbnail generator.

Program 'page_gen.pl' can be viewed here:
reddit media page generator (perl script)

While developing any piece of software I like solving various problems on paper. For example, with this site I had to solve problem how to regenerate existing pages minimally and how to resize thumbnails so they looked nice.
Here is how the sheet on which I took small notes looked like after the site got published:

reddit media website quick design notes
(sorry for the quality again, i took the picture with camera phone with two shots and stitched it together with image editor)

The final website is at redditmedia.com address (now moved to http://reddit.picurls.com). Click http://reddit.picurls.com to visit it!

Here are all the scripts packed together with basic documentation:

Download Reddit's Media Site Generator Scripts

All the scripts in a single .zip:
Download link: reddit media website generator suite (.zip)
Downloaded: 1685 times

Individual scripts:

reddit_extractor.pl
Download link: reddit extractor (perl script, reddit media generator)
Downloaded: 4310 times

db_inserter.pl
Download link: db inserter (perl script, reddit media generator)
Downloaded: 3079 times

page_gen.pl
Download link: reddit media page generator (perl script)
Downloaded: 2903 times

ThumbExtractor.pm
Download link: thumbnail extractor (perl module, reddit media generator)
Downloaded: 3668 times

ThumbMaker.pm
Download link: thumbnail maker (perl module, reddit media generator)
Downloaded: 3013 times

ImageFinder.pm
Download link: best image finder (perl module, reddit media generator)
Downloaded: 3135 times

NetPbm.pm
Download link: netpbm image manipulation (perl module, reddit media generator)
Downloaded: 3152 times

For newcomers - What is reddit?

For newcomers, reddit is a social news website where users decide its contents.

From their faq:

What is reddit?

A source for what's new and popular on the web -- personalized for you. We want to democratize the traditional model by giving editorial control to the people who use the site, not those who run it. Your votes train a filter, so let reddit know what you liked and disliked, because you'll begin to be recommended links filtered to your tastes. All of the content on reddit is from users who are rewarded for good submissions (and punished for bad ones) by their peers; you decide what appears on your front page and which submissions rise to fame or fall into obscurity.

Have fun with the website and please tell me what do you think about it in the comments! Thanks :)

perl pack unpack printf sprintf cheat sheetI decided one day that I want to master Perl's pack() and unpack() functions to be able to manipulate data in Perl efficiently.

Perl's pack and unpack are two functions for transforming data according to a user-defined template, between the guarded way Perl stores values and some well-defined representation as might be required in the environment of a Perl program. Unfortunately, they're also two of the most misunderstood and most often overlooked functions that Perl provides.

As I wrote before, my way of learning these complex functions were to make a cheat sheet first with all the template parameters and then just spend a day reading more about them and experimenting.

As I usually print cheat sheets two pages per side and the pack/unpack cheat sheet consumed just one page, I added Perl's printf/sprintf format and attribute summary.

Here is how I printed this cheat sheet:

perl pack unpack printf sprintf cheat sheet thumbnail
(Sorry for the bad quality, I shot it with my camera phone)

Download Perl's pack/unpack and printf Cheat Sheet

PDF:
Download link: perl's pack/unpack and printf cheat sheet (.pdf)
Downloaded: 123494 times

Microsoft Word 2000 format (.doc):
Download link: perl's pack/unpack and printf cheat sheet (.doc)
Downloaded: 3335 times

javascript rhino and yahoo theatreI decided I wanted to learn JavaScript Programming language better. I had been programming in it now and then but I had never really developed any good skills in it.

If you have read about this blog page then you know that I also run Free Science Online blog which is all about free video lectures online. In my May's post I had found some really good programming video lectures, 13 of them being on JavaScript. Since I run this video lecture blog, I obviously have great interest in video lectures, so why not try to learn better JavaScript from these video lectures?

These lectures are given by Douglas Crockford who is a senior JavaScript Architect at Yahoo!. He is well known for his work in introducing JavaScript Object Notation (JSON).


First four lectures are on the basics of language:

Sometimes Yahoo! Video gives this error: Sorry! This video is no longer available on Yahoo! Video. In this case refresh your browser a couple of times!

The next three lectures are on Advanced JavaScript:

Then there are 6 more lectures on JavaScript which should probably be viewed only after you have viewed the ones just mentioned.

Viewing the YUI Theater I just found another JavaScript lecture which was published just recently:

My approach to watching video lectures

I have been watching various video lectures for almost 4 years now. Mostly mathematics, physics and theory of computer science. My approach to getting most of the lectures is the following. When I watch the video lectures I take notes just as if I were in class.
Even better, when I do not understand any part of the lecture I can always rewind it back and see that fragment again. I can also pause the lecture, think for a while and then continue. But that's physics and maths.
Here is a photo of notes I have taken while watching MIT's 803: Vibrations and Waves (yes! it's available completely for free at MIT's Open Course Ware)

learning 803 vibrations and waves thumbnail pic

I am serious about physics and maths video lectures, as you can see in the image, all the main results are boxed in red, the results are fully derived (even if the professor does not do it on blackboard). Btw, one lecture perfectly fits on both sides of an A4 sheet.

Here is a close-up of Lecture 13: Electromagnetic Waves - Plane Wave Solutions to Maxwell's Equations - Polarization - Malus' Law.

mit'ss 803 - lecture 13 - em waves, plane waves

(Sorry about the bad quality of the photos, I shot them with my Nokia N73 cell phone camera)

This approach probably does not work with programming languages and computer tools. Because mathematics and physics is mostly done on paper, watching these video lectures and taking notes is actually doing them. The process of taking notes develops the skills because you work with the new concepts/operators/theorems/whatnot. Not so in programming languages. Unless you find an online degree for Java programmers, you can take a book on a new programming language, read it, and the next moment you can't even write the hello world program because you have only got familiar with the subject and have not developed the skills. I have experienced this myself.

Here is my approach how I am going to learn JavaScript from these lectures. I might adjust this approach at any moment if I find it not working, I will update this post appropriately then.

I will definitely watch all 11 video lectures. I will start with the first four basic lectures, watch them one by one, take notes as with physics video lectures and experiment as I go.

I will be taking notes lightly to have my mind really think the information I am getting over. But no red boxes around constructs as with physics. That's the experimentation part to learning. I am going to try the new constructs as soon as I see them so they stuck in my mind better.
Update: I dropped the idea of taking any notes on paper, because I am blogging the key points from lectures here.

Also to make this article interesting, I will annotate each lecture if something really interesting catches my eye. As I mentioned, I have programmed JS before so I am not sure how much I will learn from the first four basic lectures.

In my previous blog post I used Windows Script Host to create a program in VBScript. The other language the same scripting host runs is JScript which conforms to the same standard as JavaScript. So I should be safe doing JavaScript experimentation in JScript.

Points that caught my attention in JavaScript Video Lecture Part I

  • (00:45) World's most misunderstood programming language - has "Java" in its name and "Script". It has nothing to do with Java programming language and it's a real programming language not some tiny scripting language
  • (02:38) There are generally no books available to learn JS from - all are bad and full of nasty examples
  • (02:56) The only book recommended is JavaScript: The Definitive Guide, 5th Edition by David Flanagan - the least bad book
  • (03:37) JavaScript is a functional language
  • (08:12) Microsoft reverse engineered original implementation of JavaScript and called it JScript to avoid trademark issues
  • (09:49) During standardization, the original bugs were left as is without fixing to prevent already written programs from breaking (Douglas slips and says "Sun" but he actually means "Microsoft")
  • (12:16) One of the key ideas of the language is prototypal inheritance where objects inherit from objects and there are no classes
  • (12:45) One other key idea is that functions are first-class objects
  • (13:36) There are no integers in the language, everything is represented as 64-bit floating point numbers
  • (14:30) NaN (Not a Number) is not equal to antyhing, including NaN. Which means NaN == NaN is false
  • (15:22) Type of NaN is Number
  • (15:52) + prefix operator does the same thing as Number function
  • (16:08) Generally always specify radix argument in parseInt function because if the first character is '0' and there is no radix argument provided, it will assume that '0' to be an octal constant
  • (17:15) Each character takes 16-bits of memory
  • (17:56) There is no separate character type in JS, characters are represented as strings with a length of 1
  • (19:55) undefined is the default value for uninitialized variables and parameters
  • (20:38) Falsy values are: false, null, undefined, "", 0, NaN. All other values (including all objects are truthy). Everything else in the language are objects
  • (22:15) Object members can be accessed with dot notation Object.member or subscript notation Object["member"]
  • (22:59) The language is loosely typed but not "untyped"
  • (25:19) Reserved words are overused, they can't be used in dot notation as method names
  • (27:25) Operators == and != can do type coercion, so it's better to use === and !=== which do no coercion
  • (28:24) Operator && is also called the 'guard operator', Operator || is also called the 'default operator'
  • (30:16) The bitwise operators convert the operand to a 32-bit signed integer, perform the operation and then turn the result back into 64-bit floating point. Don't use bitwise operators in cases like multiplying by 4 using << 2. It will not.

After having watched the first lecture I decided that there was no point in taking notes on paper because I am blogging the key points here and trying various examples I can come up with with JScript, taking notes just wastes time.

Points that caught my attention in JavaScript Video Lecture Part II

  • (00:20) Break statements can have labels
  • (00:41) Iterating over all the members of an object with for (var name in object) { } syntax, will also iterate over inherited members
  • (06:10) There is only global and function scope in JavaScript, there is no block scope
  • (07:40) If there is no expression in a return statement, the value returned is undefined. Except for constructors, whose default return value is this
  • (08:29) An object is an unordered collection of name/value pairs
  • (21:10) All objects are linked directly or indirectly to Object.prototype
  • (23:42) Array indexes are converted to strings and used as names for retrieving values

Points that caught my attention in JavaScript Video Lecture Part III

  • (00:29) Functions inherit from Object and can store name/value pairs
  • (02:00) The function statement is just a short-hand for a var statement with a function value
  • (02:35) Functions can be defined inside of other functions
  • (03:08) JavaScript has closures
  • (07:29) There are four ways to call a function: function form, method form, constructor form and apply form
  • (10:00) this is an extra parameter. Its value depends on the calling form
  • (10:30) When a function is invoked, in addition to its parameters, it also gets a special parameter called arguments
  • (11:53) Built-in types can be augmented through (Object|Array|Function|Number|String|Boolean).prototype
  • (14:09) The typeof prefix operator returns 'object' for Array and null types
  • (15:23) eval() is the most misused feature of the language
  • (21:57) In web browsers, the global objects is the window object
  • (22:51) Use of the global namespace must be minimized
  • (23:11) Any variable which is not properly declared is assumed to be global by default
  • (23:45) JSLint is a tool which helps identify weaknesses
  • (24:21) Every object is a separate namespace, use an object to organize your variables and functions
  • (27:15) Function scope can create an encapsulation

Points that caught my attention in JavaScript Video Lecture Part IV

  • (03:30) The language definition is neutral on threads
  • (11:20) When the compiler sees an error, it attempts to replace a nearby linefeed with a semicolon and try again
  • (12:51) Do not use extra commas in array literals. Netscape will tell you that length of [1,2,3,] is 3 while IE will tell it's 4
  • (18:48) Key ideas in JavaScript - Load and go delivery, loose typing, objects as general containers, prototypal inheritance, lambda, linkage through global variables

These four lectures gave me much better theoretical understanding of JavaScript but just a little better practical skills. I should do a project entirely in JavaScript to become more skillful.

I can't wait to see the Advanced JavaScript lectures. At the end of the 4th lecture Douglas said that they will continue with theory of DOM which I will follow and only then continue with Advanced JS.

Points that caught my attention in The Theory of the DOM Part I

  • (03:31) A scripted web browser, Netscape Navigator 2, was first introduced in 1995
  • Best thing happened to have standards work was Mozilla abandoning the Netscape layer model in favor of the W3C model
  • (10:03) List of browsers Yahoo! wants their JavaScript library to run on are FireFox 1.5, FireFox 2.0, Safari 2, Internet Explorer 6, IE 7, Opera 9
  • (12:11) The <script> tag first appeared in Netscape Navigator 2
  • (13:05) <!-- --> comment around script was a Netscape 2 hack for Mosaic and Navigator 1.0
  • (14:51) W3C deprecated language=javascript attribute in script tags, don't put it there anymore
  • (18:48) If you call document.write before onload it inserts data into the document, if you call it after, it replaces the document with the new stuff
  • (20:25) name= is used to identify values in form data and to identify a window or frame
  • (20:45) id= is used to uniquely identify an element so that you could get access to it
  • (20:59) Microsoft introduced document.all as a super-collection of all elements with name or id
  • (21:39) W3C instead said use document.getElementById(id) and document.getElementsByName(name)
  • (23:41) Document tree structure is different for IE than for other browsers because Microsoft decided to depart from W3C standard and not to include whitespaces as text nodes in the tree
  • (25:02) document.body gets you to body node of the tree, document.documentElement gets you to the html root tag of the tree

After watching this lecture I decided to add time where each point that caught my attention happened so that if anyone is interested in any of the points he/she could just fast forward to that place in the video. Eventually I will go through the videos up to this one once more and add timestamps.

Points that caught my attention in The Theory of the DOM Part II

  • (04:32) The guy designing CSS chose a not so appealing names to a programmer for CSS style names. JavaScript guys converted those to camel case in JavaScript which is probably the least compatible with CSS style names
  • (08:31) Replacing a child is done with "java oriented, model based, nothing in common with reality sort of api" through old.parentNode.replaceChild(new, old) where you specify old twice
  • (09:03) It is important to remove any event handlers from the object before you delete it
  • (10:10) Microsoft and their Internet Explorer were the first to realize that it is convenient to provide access to HTML parser and provided innerHTML property which can be assigned a string containing HTML directly
  • (10:50) There is no standard describing innerHTML property
  • (12:12) The browser has an event-driven, single-threaded, asynchronous programming model
  • (12:55) There are three ways to adding event handlers - classic mode (node["on" + type] = func), Microsoft mode (node.attachEvent("on" + type, func)) and W3C mode (node.addEventListener(type, func, bool))
  • (14:50) Microsoft does not send an event parameter, they use the global event object instead.
  • (15:58) There are two ways how events are handled - trickling and bubbling
  • (17:23) The extra bool parameter in W3C mode of adding event handlers node.addEventListener(type, func, bool) tells whether the events are processed bottom up (bubbling) or top down (trickling)

Points that caught my attention in The Theory of the DOM Part III

  • (01:26) Hugest memory leaks happen in IE 6
  • (01:33) Because of that you must explicitly remove event handlers from nodes before deleting or replacing them
  • (06:49) self, parent and top are aliases of window object
  • (08:10) A script can access another window if and only if document.domain === otherwindow.document.domain
  • (10:10) There are three ways to get cross browser compatibility - browser detection, feature detection, platform libraries
  • (11:20) Internet Explorer 1.0 identified itself as "Internet Explorer" but many sites refused to serve the contents complaining that it was not "Mozilla" so in version 1.5 IE identifies itself as "Mozilla"
  • (12:16) Browser detection cross compatibility is the least recommended way
  • (15:37) Platform library cross compatibility is the most recommended way
  • (15:35) Platform library cross compatibility is the most recommended way
  • (18:48) No browser completely implements the standards and much of the DOM is not in any standards. If there was a 100% standards compliant browser, it would not work!
  • (19:19) When programming DOM: 1) do what works; 2) do what's common; 3) do what's standard

Okay, I watched the whole Theory of DOM course and have gained good theoretical knowledge but basically no practical skills. To get better with the JavaScript and DOM will require me to do some interesting practical projects with both of these guys.

Now I am off to watch Advanced JavaScript lectures and then the remaining.

Points that caught my attention in Advanced JavaScript Part I

  • (01:20) In prototypal inheritance objects inherit directly from objects, there are no classes
  • (01:30) An objects contains a "secret link" to another object. Mozilla calls it __proto__
  • (03:36) If looking for a member fails, the last object searched is Object.prototype
  • (07:50) When functions are designed to be used with new, they are called constructors
  • (08:13) new Constructor() returns a new object with a link to Constructor.prototype
  • (09:40) Always have your constructors named with a capital letter so you at least develop reflex for putting a new in front of it
  • (09:48) Forgetting the new still makes code work but it does not construct a new object. This is considered one of the language design errors
  • (10:00) When a new function object is created, it is always given a prototype member
  • (10:49) "Differential inheritance" is a special form of inheritance where you specify just the changes from one generation of objects to the next
  • (17:24) JavaScript doesn't have an operator which makes a new object using an existing object as its prototype, so we have to write our own function
  • (18:50) A "public method" is a function that uses this to access its object
  • (21:55) Functions can be used to create module containers
  • (25:01) "Privileged methods" are functions that have access to "secret" information and they are constructed through closures
  • (28:05) "Parasitic inheritance" is a way of creating an object of an augmented version an existing object.

Points that caught my attention in Advanced JavaScript Part II

  • (06:30) Pseudoclassical patterns are less effective than prototypal patterns or parasitic patterns
  • (09:06) Inner functions do not have access to this
  • (09:32) In JavaScript 1.0 there were no arrays
  • (10:31) When arrays got added, arguments object was forgot to be converted to Array object and it continues to be an array like object
  • (15:47) There are debuggers for IE - Microsoft Script Debugger, which is bad and two debuggers built in Visual Studio and Office 2003. Mozilla has Venkman and Firebug. Safari has Drosera
  • (17:30) funny instruction on how to get debugger working with Office 2003 :D
  • (24:29) All implementations of JavaScript have non-standard debugger statement which cause a breakpoint if there is a dubgger present

Points that caught my attention in Advanced JavaScript Part III

  • (04:20) Array join() method is much faster for concatenating large set of strings than using operator +
  • (07:03) Just have the server gzip the JavaScript source file to minimize the load times, avoid tools which can introduce bugs such as minificators and obfuscators
  • (07:19) JSON

The advanced JavaScript lectures provided lots of idioms and patterns used in the language. I did not do much experimentation and have really grasped just the concepts and overall structure of these advanced concepts.

Now I am going to watch "Advancing JavaScript with Libraries" by John Resig, creator of the JQuery JavaScript library and author of Pro JavaScript Techniques, is a Mozilla technologist focused on the relationship between Mozilla and the world of JavaScript libraries.

Interesting points from Advancing JavaScript with Libraries Part I

  • (08:20) In IE7 basically the only change to JavaScript was to XmlHTTPRequest object
  • (25:14) There are two standards for querying the DOM document - XPath and CSS 3 selectors
  • (26:13) IE doesn't have very good CSS selector support, because of that users have been using the very minimum of CSS selectors which almost equates CSS 1
  • (27:30) jQuery allows selecting elements from the DOM by using CSS 3 selectors which is done in one line of jQuery code instead of 20 - 25 lines of just JavaScript DOM code

Interesting points from Advancing JavaScript with Libraries Part II

  • (05:15) Users are expecting the DOM selectors to behave more like CSS, that is like when a new CSS selector is added, it propagates to all the elements affected. The users expect the same to happen when a chunk of HTML is added to the DOM, that the handlers get added to them without re-running any code
  • (12:30) Object Relational Mappings
  • (14:50) Libraries create new patterns on top of existing APIs

There were not that many points that got me interested because it was pretty obvious stuff. It was just interesting to see that the DOM is not perfect and there are many bugs which one or the other browser fails, why they fail and how to solve these DOM related problems. Also typical programming meta-problems were discussed such as JavaScript trying to get elements before browser has loaded the DOM, how the white spaces are treated and what methods to use for navigating the DOM. Later the lecture went on how to query the DOM tree using jQuery and mix of XPath and CSS 3. Then it is discussed how injecting HTML in an existing document is done, what's tricky about it and what problems can arise. Finally the lecture continues with FUEL and object relational mappings.

The other lecture I watched was "Maintainable JavaScript" by Nicholas Zakas. He is an engineer on the team that brings you My Yahoo!, one of the most popular personalized portals on the web. He is also the author of two books on frontend engineering, including "Professional JavaScript for Web Developers," one of the best tomes of its kind.

Interesting points from Maintainable JavaScript

  • (01:15) It is estimated that as much as 80% of time is spent maintaining existing code
  • (04:30) Maintainable code is understandable, intuitive, adaptive, extendable and debuggable
  • (16:20) There are three layers on the client side - JavaScript for behavior, CSS for presentation and HTML for structure
  • (23:42) Programming practices
  • (26:30) Namespace your objects
  • (32:10) Avoid null comparison, use instanceof or typeof operators
  • (37:15) Write code in separate JavaScript files and use a build process to combine them
  • (40:47) Summary of writing maintainable code - code conventions, loose coupling, programming practices and build process

There are not that many interesting points anymore because most of the stuff has been covered in the previous lectures. Apart from that these lecture did not teach me much new because this stuff was pretty obvious. The only point to watch this lecture is to refresh all these obvious suggestions - agree on indentation and naming conventions in your team, comment difficult algorithms and large sections of code, don't write obvious comments, comment hacks, loose coupling, careful use of complex code and design patterns, etc.

What do you think about these lectures?

youtube video downloader in vbscriptWhen I was a small kid I liked RAD (Rapid Application Development) in Microsoft® Visual Basic® programming language. I mastered the language entirely from MSDN's samples and it was such a joy to create GUI applications in a few hours or days and they just worked! Now it has been a few years years since I have not touched Visual Basic and I have become really rusty in this language.

If you have been reading my blog, you might have noticed that I have been creating these YouTube video downloaders in various programming languages. I have no plans to create a downloader in Visual Basic itself, because I am creating a GUI downloader in C and C++ already but I would like to use a derived language (a subset of Visual Basic language), VBScript, to create the downloader.

That would refresh my knowledge of Visual Basic a little and also let me learn a little more about programming Windows Script Host (WSH).

Here is a quote from a really good introduction to what WSH is:

The first time people encounter Windows Script Host, they often express some confusion. What exactly is WSH? Is it a language, like VBScript or JScript? No; although WSH enables you to run programs written in these languages, it is not a language itself. WSH is a script host. A script host is a program that provides an environment in which users can execute scripts in a variety of languages, languages that use a variety of object models to perform tasks.

You are probably already familiar with other script hosts. Microsoft® Internet Explorer, for example, enables users to execute scripts that use the Dynamic HTML object model. Shell programs (such as C Shell, Bourne Shell and Korn Shell) enable you to write scripts that use an object model capable of manipulating the file system. Even the command prompt can be thought of as a scripting environment because it can run scripts written in the "batch file" language.

WSH is an unusual script host in that it was designed to be general-purpose. Unlike most of the scripting tools mentioned above, WSH imposes restrictions on neither the language used to write scripts nor the object models used by scripts.

WSH is ideal for non-interactive scripting needs such as logon scripting and administrative scripting but we will use it to download YouTube videos as well.

The key advantage of this script is that it will run on any Windows operating system newer than Windows 98. If you have Windows 98 or an older Windows, follow this link and install the latest version of WSH (unfortunately version 5.1 (5.6 is the latest))!

Let's define the interface of the script.

The script can either be run in WScript environment or CScript environment. WScript is the system default on Windows 2000. Characteristics of this environment are that output appears in a pop-up window, and script execution waits until the user clears the pop-up window by pressing OK button. The other environment is the CScript environment. The primary difference between the CScript and WScript environments is that the CScript environment directs display information to the command window, which has a side effect of letting a script run to completion without pausing for each message sent to the UI. I want this script to be usable from both of the environments.

Here is what I am thinking. By default I want this script to be accepting the video URL to download as the first command line argument (followed by more videos as next arguments). If the first argument is not provided and it is run in WScript environment I want the script to pop up an InputBox dialog asking for the URL of the video to download. Otherwise, if it is run in CScript environment, I want it to quit.

How do we find in which environment the script is being run? I had no idea, so I turned to Google Groups and searched for "cscript wscript detect" and found this post which explained how to do it with this snippet of code:

If "CSCRIPT.EXE" = UCase(Right(WScript.Fullname, 11)) Then
    WScript.Echo "The program was run by CScript!"
End If 

The next thing we need to figure out is how to get the command line arguments of a running script.

Turns out that each WSH script has a WScript object available without the script needing to bind to the object. Command-line arguments are stored in the WshArguments collection, which you access through the Arguments property of the WScript object.

Here is a diagram of WScript object's methods, properties and WShArguments expanded:

wsh wscript object's hierarcy

The easiest way to loop over all arguments is shown in this snippet:

Set objArgs = WScript.Arguments
For I = 0 to objArgs.Count - 1
   WScript.Echo objArgs(I)
Next

Now just instead of "WScript.Echo objArgs(I)" we call "DownloadVideo objArgs(I)", where DownloadVideo is our procedure for downloading YouTube videos.

In one of the previous articles, downloading youtube videos with awk programming language, I explained how the embedded YouTube flash video player retrieves the video file itself. Please see that article if you are interested in how I figured it out.

Last two things we have to figure out before we have a running script is how to talk over HTTP with VBScript and how to save the incoming binary data to a file.

Both VBScript and WScript object provide CreateObject function (but there is a difference between them) which allows binding to COM objects.

There is a "Microsoft.XmlHttp" COM Object which is shipped with Internet Explorer 5.0 and later. This object is what actually provides the well known AJAX interface used in Web2.0 applications - the XMLHttpRequest interface. Here at MSDN is the documentation of this interface with all the methods and properties it provides.

When we get the video, we will be dealing with binary data. Microsoft Scripting Runtime provides us with FileSystemObject (FSO) but unfortunately it is not suitable for writing binary files.

There is a way to write binary files with FSO but it is so slow that I dropped this solution. It took more than 10 minutes to write 1MB of data!

' Given a FileName and Data, saves Data to file named FileName
Sub SaveData(FileName, Data)
    Dim Fso: Set Fso = CreateObject("Scripting.FileSystemObject")
    Dim TextStream: Set TextStream = Fso.CreateTextFile(FileName, True)

    WScript.Echo LenB(Data)
    TextStream.Write BinaryToString(Data)
End Sub

' Given Binary data, converts it to a string
Function BinaryToString(Binary)
  Dim I, S
  For I = 1 To LenB(Binary)
    S = S & Chr(AscB(MidB(Binary, I, 1)))
  Next
  BinaryToString = S
End Function

A much better way to write binary data to a file with VBScript is using ADODO.Stream Object which deals with binary data out of the box. Look at SaveVideo function in my final program to see how it is used to write binary data to file.

Script Usage

Before the script can be used, you have to tell your computer to trust youtube.com domain. If you do not do this, executing the script will lead you to the following error:

ytdown.vbs(73, 5) msxml3.dll: Access is denied.

Youtube.com domain can be trusted by adding it to trusted sites security zone in Internet Explorer.
Launch your Internet Explorer browser and head to Tools -> Internet Options, then follow the steps illustrated in this image:

internet explorer trusted sites zone security

Once you have trusted youtube.com domain, you can start downloading the videos.

One of the ways is to double click the ytdown.vbs icon which will launch the script in WScript environment and an input dialog will appear asking for a video to download:

launch ytdown.vbs by double clicking the icon

After you press "OK", the downloader will save the video to a file with the title of the video and .flv extension in the same directory!

The other way to download a video is to call it via command line in CScript environment:

C:\ytdown>cscript ytdown.vbs "http://www.youtube.com/watch?v=h9MN2mKGZoo"
Microsoft (R) Windows Script Host Version 5.6
Copyright (C) Microsoft Corporation 1996-2001. All rights reserved.

Downloading video '30 fps, 30 frames (1 second)'
Done!

Happy downloading!

The downloaded .flv file can be converted to a better format like DivX or .avi.
Read about converting a video to a better format in this article.

Here is the final script:

'
' Peteris Krumins (peter@catonmat.net)
' http://www.catonmat.net  -  good coders code, great reuse
'
' 2007.08.03 v1.0 - initial release
' 2007.10.21 v1.1 - youtube changed the way it displays vids
' 2008.03.01 v1.2 - youtube changed the way it displays vids
' 2009.12.02 v1.3 - youtube changed the way it displays vids
'
Option Explicit

Dim WscriptMode

' Detect if we are running in WScript or CScript
If UCase(Right(WScript.Fullname, 11)) = "WSCRIPT.EXE" Then
    WScriptMode = True
Else 
    WScriptMode = False
End If 

Dim Args: Set Args = WScript.Arguments

If Args.Count = 0 And WScriptMode Then
    ' If running in WScript and no command line args are provided
    ' ask the user for a URL to the YouTube video
    Dim Url: Url = InputBox("Enter a YouTube video URL to download" & vbCrLf & _
                   "For example, http://youtube.com/watch?v=G1ynTV_E-5s", _
                   "YouTube Downloader, http://www.catonmat.net")
    If Len(Url) = 0 Then: WScript.Quit 1
    DownloadVideo Url
ElseIf Args.Count = 0 And Not WScriptMode Then
    ' If running in CScript and no command line args are provided
    ' show the usage and quit
    WScript.Echo "Usage: " & WScript.ScriptName & " <video url 1> [video url 2] ..."
    WScript.Quit 1
Else 
    ' Download all videos
    Dim I

    For I = 0 to args.Count - 1
        DownloadVideo args(I)
    Next
End If

' Downloads a YouTube video and saves it to a file
Sub DownloadVideo(Url)
    Dim Http, VideoTitle, VideoName, Req

    Set Http = CreateObject("Microsoft.XmlHttp")
    Http.open "GET", Url, False
    Http.send

    If Http.status <> 200 Then
        WScript.Echo "Failed getting video page at: " & Url & vbCrLf & _
                     "Error: " & Http.statusText
        Exit Sub
    End If

    Dim VideoId: VideoId = ExtractMatch(Url, "v=([A-Za-z0-9-_]+)")
    If Len(VideoID) = 0 Then
        WScript.Echo "Could not extract video ID from " & Url
        Exit Sub
    End If

    VideoTitle = GetVideoTitle(Http.responseText)
    If Len(VideoTitle) = 0 Then
        WScript.Echo "Failed extracting video title from video at URL: " & Url & vbCrLf & _
                     "Will use the video ID '" & VideoID & "' for the filename."
        VideoName = VideoID
    Else
        VideoName = VideoTitle
    End If

    Dim FmtMap: FmtMap = GetFmtMap(Http.responseText)
    If Len(FmtMap) = 0 Then
        WScript.Echo "Could not extract fmt_url_map from the video page."
        Exit Sub
    End If

    Dim VideoURL: VideoURL = Find_Video_5(FmtMap)
    If Len(VideoURL) = 0 Then
        WScript.Echo "Could not extract fmt_url_map from the video page."
        Exit Sub
    End If

    If WScriptMode = False Then: WScript.Echo "Downloading video '" & VideoName & "'"
    Http.open "GET", VideoURL, False
    Http.send

    If Http.status <> 200 Then
        WScript.Echo "Failed getting the flv video: " & Url & vbCrLf & _
                     "Error: " & Http.statusText
        Exit Sub
    End If

    Dim SaneFilename
    SaneFilename = MkFileName(VideoName)

    SaveVideo SaneFilename, Http.ResponseBody
    WScript.Echo "Done downloading video. Saved to " & SaneFilename & "."
End Sub

' Given fmt_url_map, url-escapes it, and finds the video url for video
' with id 5, which is the regular quality flv video.
Function Find_Video_5(FmtMap)
    FmtMap = Unescape(FmtMap)
    Find_Video_5 = ExtractMatch(FmtMap, ",?5\|([^,]+)")
End Function

' Given YouTube Html page, extract the fmt_url_map parameter that contains
' the URL to the .flv video
Function GetFmtMap(Html)
    GetFmtMap = ExtractMatch(Html, """fmt_url_map"": ""([^""]+)""")
End Function

' Given YouTube Html page, the function extracts the title from <title> tag
Function GetVideoTitle(Html)
    ' get rid of all tabs
    Html = Replace(Html, Chr(9), "")

    ' get rid of all newlines (vbscript regex engine doesn't like them)
    Html = Replace(Html, vbCrLf, "")
    Html = Replace(Html, vbLf, "")
    Html = Replace(Html, vbCr, "")

    GetVideoTitle = ExtractMatch(Html, "<title>YouTube ?- ?([^<]+)<")
End Function

' Given the Title of a video, function creates a usable filename for a video by
' sanitizing it - stripping parenthesis, changing non alphanumeric characters
' to _ and adding .flv extension
Function MkFileName(Title)
    Title = Replace(Title, "(", "")
    Title = Replace(Title, ")", "")

    Dim Regex
    Set Regex = New RegExp
    With Regex
        .Pattern = "[^A-Za-z0-9-_]"
        .Global = True
    End With

    Title = Regex.Replace(Title, "_")
    MkFileName = Title & ".flv"
End Function

' Given Text and a regular expression Pattern, the function extracts
' the first submatch
Function ExtractMatch(Text, Pattern)
    Dim Regex, Matches

    Set Regex = New RegExp
    Regex.Pattern = Pattern

    Set Matches = Regex.Execute(Text)
    If Matches.Count = 0 Then
        ExtractMatch = ""
        Exit Function
    End If

    ExtractMatch = Matches(0).SubMatches(0)
End Function

' Function saves Data to FileName
Function SaveVideo(FileName, Data)
  Const adTypeBinary = 1
  Const adSaveCreateOverWrite = 2
  
  Dim Stream: Set Stream = CreateObject("ADODB.Stream")
  
  Stream.Type = adTypeBinary
  Stream.Open
  Stream.Write Data
  Stream.SaveToFile FileName, adSaveCreateOverWrite
End Function

'
' ==========================================================================
' The following code saves binary data to file using FileSystemObject
' It is so slow that even on a 3.2Ghz computer saving 1 MB takes 10 minutes!
' Don't use it! I put it here just to illustrate the wrong solution!
' ==========================================================================
'

' Given a Filename and Data, the function saves Data to File
'Sub SaveVideo(File, Data)
'    Dim Fso: Set Fso = CreateObject("Scripting.FileSystemObject")
'    Dim TextStream: Set TextStream = Fso.CreateTextFile(File, True)
'
'    WScript.Echo LenB(Data)
'    TextStream.Write BinaryToString(Data)
'End Sub

' Given Binary data, converts it to a string
'Function BinaryToString(Binary)
'  Dim I, S
'  For I = 1 To LenB(Binary)
'    S = S & Chr(AscB(MidB(Binary, I, 1)))
'  Next
'  BinaryToString = S
'End Function


'
' ==========================================================================
' The following is an implementation of UrlUnescape. It turned out VBScript
' has Unescape() function built in already, that does it!
'
'Function UrlUnescape(Str)
'    Dim Regex, Match, Matches
'
'    Set Regex = New RegExp
'    With Regex
'        .Pattern = "%([0-9a-f][0-9a-f])"
'        .IgnoreCase = True
'        .Global = True
'    End With
'    ' Wanted to do this, but it wasn't quite possible
'    ' UrlUnescape = Regex.Replace(Str, Chr(CInt("&H" & $0)))
'
'    Set Matches = Regex.Execute(Str)
'    For Each Match in Matches
'        Str = Replace(Str, Match, Chr(CInt("&H" & Match.SubMatches(0))))
'    Next
'
'    UrlUnescape = Str
'End Function

Download Visual Basic Script YouTube Video Downloader

Download link: vbscript youtube video downloader
Total downloads: 21102 times

If you notice any bugs in this script, have any recommendations, want to criticize my code, want to thank me, then just post a comment on this article!

ps. remember the ILOVEYOU virus? It was also written in VBScript, that alone indicates that this language is worth knowing.

This article is part of the article series "GUI Video Downloader."
<- previous article next article ->

video downloader software, vim, mingw, wxwidgetsThis is a continuation of multi-part series on how to create a cross-platform desktop video downloading and converting application. In part I we prepared our coding environment with vim, various vim plugins, mingw and gdb. In this part we will start creating a C library for downloading just the YouTube videos with modularity in mind, so that we could create a nice desktop application capable of downloading videos from other video sharing sites like break, metacafe, dailymotion and others. Complementary to this library we will be also be creating a command line application which will use this C library. The command line application will serve us as a test tool for the library, so we knew the library worked ok. Once we have these two components, we will build GUI around the library and create the desktop app.

Writing the Library

I have chosen a name libvd for the library which stands for video download library.

The library will be written in C programming language because:

  • it is the number one cross platform language,
  • I haven't programming in C for a while, so it will be a great refresher,
  • the GUI framework which I will be using is written in C++ which will have no problems with using C code
  • the application will be small and will require no unnecessary library/toolkit dependencies

Downloading a video means working with HTTP protocol. Let's use "good coders code, great reuse" approach and seek for a C library which would ease the usage of HTTP protocol.

Having worked on a few projects in C programming language involving HTTP protocol, I know an excellent library. It's called libcurl. I even implemented cookie interface for this library which was missing two years ago.

What is libcurl? Quoting the author of libcurl: "libcurl is a free and easy-to-use client-side URL transfer library, supporting FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, FILE and LDAP. libcurl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, Kerberos4), file transfer resume, http proxy tunneling and more!

libcurl is highly portable, it builds and works identically on numerous platforms, including Solaris, NetBSD, FreeBSD, OpenBSD, Darwin, HPUX, IRIX, AIX, Tru64, Linux, UnixWare, HURD, Windows, Amiga, OS/2, BeOs, Mac OS X, Ultrix, QNX, OpenVMS, RISC OS, Novell NetWare, DOS and more..."

Woah! Much more than we need! And notice how cross platform it is! Exactly what we are looking for!

Let's download the libcurl library. Since I am developing the program on a fresh Windows XP virtual machine, I chose to download curl-7.16.4.zip version of the library. It really does not matter what operating system I am developing this program on, I could have as well chosen Linux, then I would have downloaded the library ending with extension .gz or .bz2.

download libcurl for windows

Compiling libcurl

Once we have downloaded libcurl, let's unpack it and look for instructions on how to compile it with MinGW on a Windows machine. I chose c:\vd as the root directory for the project on the windows machine because I am afraid of those whitespace characters windows has in path. I unpacked the libcurl library to c:\vd\curl.

At this point we have no idea how to build the libcurl on Windows, right? So let's look in the docs subdirectory (C:\vd\curl\docs) for a clue. Usually the instructions on how to build a library on a particular platform can be found in files like README.platform or INSTALL.platform or just INSTALL. Indeed, libcurl has a file README.win32. Let's read it.

libcurl's win32 readme file

Oops, it says to read README first, we look in the same docs subdirectory but there is no README! We look in the parent directory, whew, yeah there is a README. But it is just 1KB in size and says nothing...

The only reasonable file left is INSTALL, let's look at it. Oh yeah, it's 29K in size! Must be it!

Indeed, scrolling a few screens down by pressing vim's Ctrl_D, we find instructions on how to build libcurl on win32 with MinGW compiler!

libcurl's win32 install file with mingw

Libcurl also provides support for compressed HTTP content with zlib, and secure connections over HTTPS with OpenSSL. The INSTALL file mentions what to do to get support for these thingies. We do not need them, so we ignore them.

Don't rush yet compiling the library! We also want our application to be small, not just portable. Libcurl has support for so many protocols but our project need just the HTTP protocol!

Looking a little further in the INSTALL file, we find the following instruction:

   Disabling Specific Protocols in Win32 builds
   --------------------------------------------

   The configure utility, unfortunately, is not available for the Windows
   environment, therefore, you cannot use the various disable-protocol
   options of the configure utility on this platform.

   However, you can use the following defines to disable specific
   protocols:

   HTTP_ONLY             disables all protocols except HTTP
   [...]

   If you want to set any of these defines you have the following
   possibilities:

   - Modify lib/setup.h
   [...]

We really want just the HTTP protocol, so let's follow this instruction and modify setup.h file in lib directory (file C:\vd\curl\lib\setup.h) and add the following line right after the comments:

#define HTTP_ONLY

The file should now look like this:

modifying libcurl to make it support HTTP only!

Okay, now we are ready to compile libcurl!

The INSTALL file says that to compile the libcurl library we first need to run 'mingw32.bat' and then 'make mingw32'.

   MingW32
   -------

   Run the 'mingw32.bat' file to get the proper environment variables set,
   then run 'make mingw32' in the root dir. Use 'make mingw32-ssl' to build
   curl SSL enabled.

mingw32.bat? I don't have such file in my mingw32 distribution! Now what? Let's be real developers and compile the library ourselves then. Let's make sure the MinGW's bin directory (in my case C:\MinGW\bin) is in our PATH environment variable and let's run the second command - 'make mingw32'.

Oops:

C:\vd\curl>make mingw32
'make' is not recognized as an internal or external command,
operable program or batch file.

There is no program named 'make' in MinGW's distribution, there is only mingw32-make.exe, let's run it:

C:\vd\curl>c:\MinGW\bin\mingw32-make.exe mingw32
c:/MinGW/bin/mingw32-make -C lib -f Makefile.m32 ZLIB=1
mingw32-make[1]: Entering directory `C:/vd/curl/lib'
gcc -I. -I../include -I"../../zlib-1.2.3" -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -DHAVE_LIBZ -DHAVE_ZLIB_H -c file.c
gcc -I. -I../include -I"../../zlib-1.2.3" -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -DHAVE_LIBZ -DHAVE_ZLIB_H -c timeval.c
gcc -I. -I../include -I"../../zlib-1.2.3" -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -DHAVE_LIBZ -DHAVE_ZLIB_H -c base64.c
In file included from base64.c:43:
urldata.h:92:59: zlib.h: No such file or directory
In file included from base64.c:43:

Huh? I never wanted my libcurl to be built with zlib support. Let's get rid of it by editing the Makefile (C:\vd\curl\Makefile) directly. Search for mingw32 target (string "mingw32:") and you'll find:

mingw32:
	$(MAKE) -C lib -f Makefile.m32 ZLIB=1
	$(MAKE) -C src -f Makefile.m32 ZLIB=1

We really don't want to build the curl executable, so let's get rid of the last line and also get rid of "ZLIB=1" variable definition.
The lines should look now like this:

mingw32:
	$(MAKE) -C lib -f Makefile.m32

Now let's run the mingw32-make again:

C:\vd\curl>c:\MinGW\bin\mingw32-make.exe mingw32
c:/MinGW/bin/mingw32-make -C lib -f Makefile.m32
mingw32-make[1]: Entering directory `C:/vd/curl/lib'
gcc -I. -I../include -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -c base64.c
gcc -I. -I../include -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -c hostip.c
[...]
rm -f libcurl.a
process_begin: CreateProcess(NULL, rm -f libcurl.a, ...) failed.
make (e=2): The system cannot find the file specified.
mingw32-make[1]: *** [libcurl.a] Error 2
mingw32-make[1]: Leaving directory `C:/vd/curl/lib'
mingw32-make: *** [mingw32] Error 2

Argh! There is no 'rm' on Windows, at least not with MinGW development tools package. Let's modify the Makefile.m32 file (C:\vd\curl\lib\Makefile.m32) and change 'rm' to Windows 'del' command.

Find this line in Makefile.m32:

RM = rm -f

and change it to

RM = del /F

Perfect! Running the mingw32-make.exe again finishes with no errors! Lookin in the C:\vd\curl\lib directory we find the static library libcurl.a and the dynamic library libcurl.dll. Had we compiled it on Linux, the dynamic library would have been named libcurl.so.

Testing libcurl

I wrote the following test program which outputs the website of google.com to the standard output if libcurl works correctly:

#include <stdio.h>
#include <stdlib.h>
#include "curl/curl.h"

int main(void) {
    CURLcode ret;
    CURL *curl = curl_easy_init();
    if (curl == NULL) {
        fprintf(stderr, "Failed creating CURL easy handle!\n");
        exit(EXIT_FAILURE);
    }

    /* let's get google.com because I love google */
    ret = curl_easy_setopt(curl, CURLOPT_URL, "http://www.google.com");
    if (ret != CURLE_OK) {
        fprintf(stderr, "Failed getting http://www.google.com: %s\n",
                curl_easy_strerror(ret));
        exit(EXIT_FAILURE);
    }

    ret = curl_easy_perform(curl);
    if (ret != 0) {
        fprintf(stderr, "Failed getting http://www.google.com: %s\n",
                curl_easy_strerror(ret));
        exit(EXIT_FAILURE);
    }

    return 0;
}

I placed this program in C:\vd\tests\curltest directory and named the source file testcurl.c. Our curl library is located in C:\vd\curl\lib and the curl header files in C:\vd\curl\include. Assuming out currenly working directory is C:\vd\tests\curltest, the following command line will compile the C source file to an executable:

gcc testcurl.c -o testcurl -DCURL_STATICLIB -I..\..\curl\include -L..\..\curl\lib -lcurl -lwsock32 -lwinmm -Wall

It will create testcurl.exe executable which can be run. Let's try running it:

C:\vd\tests\libcurl>testcurl.exe
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>Google</title>
[...]

Yeah! It works! Now we are ready to write the libvd (video download) library which will use the libcurl for communicating over HTTP protocol.

I'll do this in Part III of this series. Until then, have fun compiling libcurl!