This article is part of the article series "GUI Video Downloader."
<- previous article next article ->
video downloader software, vim, mingw, wxwidgets

This is a continuation of multi-part series on how to create a cross-platform desktop video downloading and converting application. In part I we prepared our coding environment with vim, various vim plugins, mingw and gdb. In this part we will start creating a C library for downloading just the YouTube videos with modularity in mind, so that we could create a nice desktop application capable of downloading videos from other video sharing sites like break, metacafe, dailymotion and others. Complementary to this library we will be also be creating a command line application which will use this C library. The command line application will serve us as a test tool for the library, so we knew the library worked ok. Once we have these two components, we will build GUI around the library and create the desktop app.

Writing the Library

I have chosen a name libvd for the library which stands for video download library.

The library will be written in C programming language because:

  • it is the number one cross platform language,
  • I haven't programming in C for a while, so it will be a great refresher,
  • the GUI framework which I will be using is written in C++ which will have no problems with using C code
  • the application will be small and will require no unnecessary library/toolkit dependencies

Downloading a video means working with HTTP protocol. Let's use "good coders code, great reuse" approach and seek for a C library which would ease the usage of HTTP protocol.

Having worked on a few projects in C programming language involving HTTP protocol, I know an excellent library. It's called libcurl. I even implemented cookie interface for this library which was missing two years ago.

What is libcurl? Quoting the author of libcurl: "libcurl is a free and easy-to-use client-side URL transfer library, supporting FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, FILE and LDAP. libcurl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, Kerberos4), file transfer resume, http proxy tunneling and more!

libcurl is highly portable, it builds and works identically on numerous platforms, including Solaris, NetBSD, FreeBSD, OpenBSD, Darwin, HPUX, IRIX, AIX, Tru64, Linux, UnixWare, HURD, Windows, Amiga, OS/2, BeOs, Mac OS X, Ultrix, QNX, OpenVMS, RISC OS, Novell NetWare, DOS and more..."

Woah! Much more than we need! And notice how cross platform it is! Exactly what we are looking for!

Let's download the libcurl library. Since I am developing the program on a fresh Windows XP virtual machine, I chose to download curl-7.16.4.zip version of the library. It really does not matter what operating system I am developing this program on, I could have as well chosen Linux, then I would have downloaded the library ending with extension .gz or .bz2.

download libcurl for windows

Compiling libcurl

Once we have downloaded libcurl, let's unpack it and look for instructions on how to compile it with MinGW on a Windows machine. I chose c:\vd as the root directory for the project on the windows machine because I am afraid of those whitespace characters windows has in path. I unpacked the libcurl library to c:\vd\curl.

At this point we have no idea how to build the libcurl on Windows, right? So let's look in the docs subdirectory (C:\vd\curl\docs) for a clue. Usually the instructions on how to build a library on a particular platform can be found in files like README.platform or INSTALL.platform or just INSTALL. Indeed, libcurl has a file README.win32. Let's read it.

libcurl's win32 readme file

Oops, it says to read README first, we look in the same docs subdirectory but there is no README! We look in the parent directory, whew, yeah there is a README. But it is just 1KB in size and says nothing...

The only reasonable file left is INSTALL, let's look at it. Oh yeah, it's 29K in size! Must be it!

Indeed, scrolling a few screens down by pressing vim's Ctrl_D, we find instructions on how to build libcurl on win32 with MinGW compiler!

libcurl's win32 install file with mingw

Libcurl also provides support for compressed HTTP content with zlib, and secure connections over HTTPS with OpenSSL. The INSTALL file mentions what to do to get support for these thingies. We do not need them, so we ignore them.

Don't rush yet compiling the library! We also want our application to be small, not just portable. Libcurl has support for so many protocols but our project need just the HTTP protocol!

Looking a little further in the INSTALL file, we find the following instruction:

   Disabling Specific Protocols in Win32 builds
   --------------------------------------------

   The configure utility, unfortunately, is not available for the Windows
   environment, therefore, you cannot use the various disable-protocol
   options of the configure utility on this platform.

   However, you can use the following defines to disable specific
   protocols:

   HTTP_ONLY             disables all protocols except HTTP
   [...]

   If you want to set any of these defines you have the following
   possibilities:

   - Modify lib/setup.h
   [...]

We really want just the HTTP protocol, so let's follow this instruction and modify setup.h file in lib directory (file C:\vd\curl\lib\setup.h) and add the following line right after the comments:

#define HTTP_ONLY

The file should now look like this:

modifying libcurl to make it support HTTP only!

Okay, now we are ready to compile libcurl!

The INSTALL file says that to compile the libcurl library we first need to run 'mingw32.bat' and then 'make mingw32'.

   MingW32
   -------

   Run the 'mingw32.bat' file to get the proper environment variables set,
   then run 'make mingw32' in the root dir. Use 'make mingw32-ssl' to build
   curl SSL enabled.

mingw32.bat? I don't have such file in my mingw32 distribution! Now what? Let's be real developers and compile the library ourselves then. Let's make sure the MinGW's bin directory (in my case C:\MinGW\bin) is in our PATH environment variable and let's run the second command - 'make mingw32'.

Oops:

C:\vd\curl>make mingw32
'make' is not recognized as an internal or external command,
operable program or batch file.

There is no program named 'make' in MinGW's distribution, there is only mingw32-make.exe, let's run it:

C:\vd\curl>c:\MinGW\bin\mingw32-make.exe mingw32
c:/MinGW/bin/mingw32-make -C lib -f Makefile.m32 ZLIB=1
mingw32-make[1]: Entering directory `C:/vd/curl/lib'
gcc -I. -I../include -I"../../zlib-1.2.3" -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -DHAVE_LIBZ -DHAVE_ZLIB_H -c file.c
gcc -I. -I../include -I"../../zlib-1.2.3" -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -DHAVE_LIBZ -DHAVE_ZLIB_H -c timeval.c
gcc -I. -I../include -I"../../zlib-1.2.3" -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -DHAVE_LIBZ -DHAVE_ZLIB_H -c base64.c
In file included from base64.c:43:
urldata.h:92:59: zlib.h: No such file or directory
In file included from base64.c:43:

Huh? I never wanted my libcurl to be built with zlib support. Let's get rid of it by editing the Makefile (C:\vd\curl\Makefile) directly. Search for mingw32 target (string "mingw32:") and you'll find:

mingw32:
	$(MAKE) -C lib -f Makefile.m32 ZLIB=1
	$(MAKE) -C src -f Makefile.m32 ZLIB=1

We really don't want to build the curl executable, so let's get rid of the last line and also get rid of "ZLIB=1" variable definition.
The lines should look now like this:

mingw32:
	$(MAKE) -C lib -f Makefile.m32

Now let's run the mingw32-make again:

C:\vd\curl>c:\MinGW\bin\mingw32-make.exe mingw32
c:/MinGW/bin/mingw32-make -C lib -f Makefile.m32
mingw32-make[1]: Entering directory `C:/vd/curl/lib'
gcc -I. -I../include -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -c base64.c
gcc -I. -I../include -g -O2 -DBUILDING_LIBCURL -DHAVE_LONGLONG -c hostip.c
[...]
rm -f libcurl.a
process_begin: CreateProcess(NULL, rm -f libcurl.a, ...) failed.
make (e=2): The system cannot find the file specified.
mingw32-make[1]: *** [libcurl.a] Error 2
mingw32-make[1]: Leaving directory `C:/vd/curl/lib'
mingw32-make: *** [mingw32] Error 2

Argh! There is no 'rm' on Windows, at least not with MinGW development tools package. Let's modify the Makefile.m32 file (C:\vd\curl\lib\Makefile.m32) and change 'rm' to Windows 'del' command.

Find this line in Makefile.m32:

RM = rm -f

and change it to

RM = del /F

Perfect! Running the mingw32-make.exe again finishes with no errors! Lookin in the C:\vd\curl\lib directory we find the static library libcurl.a and the dynamic library libcurl.dll. Had we compiled it on Linux, the dynamic library would have been named libcurl.so.

Testing libcurl

I wrote the following test program which outputs the website of google.com to the standard output if libcurl works correctly:

#include <stdio.h>
#include <stdlib.h>
#include "curl/curl.h"

int main(void) {
    CURLcode ret;
    CURL *curl = curl_easy_init();
    if (curl == NULL) {
        fprintf(stderr, "Failed creating CURL easy handle!\n");
        exit(EXIT_FAILURE);
    }

    /* let's get google.com because I love google */
    ret = curl_easy_setopt(curl, CURLOPT_URL, "http://www.google.com");
    if (ret != CURLE_OK) {
        fprintf(stderr, "Failed getting http://www.google.com: %s\n",
                curl_easy_strerror(ret));
        exit(EXIT_FAILURE);
    }

    ret = curl_easy_perform(curl);
    if (ret != 0) {
        fprintf(stderr, "Failed getting http://www.google.com: %s\n",
                curl_easy_strerror(ret));
        exit(EXIT_FAILURE);
    }

    return 0;
}

I placed this program in C:\vd\tests\curltest directory and named the source file testcurl.c. Our curl library is located in C:\vd\curl\lib and the curl header files in C:\vd\curl\include. Assuming out currenly working directory is C:\vd\tests\curltest, the following command line will compile the C source file to an executable:

gcc testcurl.c -o testcurl -DCURL_STATICLIB -I..\..\curl\include -L..\..\curl\lib -lcurl -lwsock32 -lwinmm -Wall

It will create testcurl.exe executable which can be run. Let's try running it:

C:\vd\tests\libcurl>testcurl.exe
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>Google</title>
[...]

Yeah! It works! Now we are ready to write the libvd (video download) library which will use the libcurl for communicating over HTTP protocol.

I'll do this in Part III of this series. Until then, have fun compiling libcurl!

This article is part of the article series "GUI Video Downloader."
<- previous article next article ->

Comments

Daniel Permalink
January 07, 2008, 12:25

waw!!!... i can't wait for the part III !!!

January 21, 2008, 15:03

WHEN WILL BE THE THIRD PART AVAILABLE !!!

Emre Permalink
February 08, 2008, 01:15

Waiting Part III too.

Luis Permalink
March 07, 2008, 19:10

I'm following along....excellent series...can't wait for the next part.

Alan Permalink
August 24, 2008, 01:07

Very interesting article indeed Thanks!

Zhijin Liu Permalink
November 01, 2008, 10:36

Waiting for the following parts!
Very interesting!

December 08, 2008, 22:33

Yes, PLEASE post part 3!! I'm in the process of learning C programming (I've been learning entirely on Linux so far) and these two posts have been an incredibly useful insight into programming something in C on a Windows system.

Bath Squal Permalink
December 20, 2008, 08:56

Please post part III. I am really interested in knowing it. Plea.....se

bob Permalink
July 23, 2009, 09:43

Sorry, i can't find PART III at
http://www.catonmat.net/blog/creating-a-video-downloader-application-using-opensource-tools-part-three/

Milan Permalink
February 25, 2010, 09:20

Thank you very much for your description of building libcurl with Mingw, it helped me a lot!

Linn Permalink
August 17, 2010, 13:14

Thank you so much for this! I was struggling with getting libcurl to compile correctly so that I could link the library static-ly. This article was EXACTLY what I was looking for.

Leave a new comment

(why do I need your e-mail?)

(Your twitter name, if you have one. (I'm @pkrumins, btw.))

Type the first letter of your name: (just to make sure you're a human)

Please preview the comment before submitting to make sure it's OK.

Advertisements