+++ /dev/null
-$Id$
- _ _ ____ _
- ___| | | | _ \| |
- / __| | | | |_) | |
- | (__| |_| | _ <| |___
- \___|\___/|_| \_\_____|
-
-PROGRAMMING WITH LIBCURL
-
-About this Document
-
- This document attempts to describe the general principles and some basic
- approaches to consider when programming with libcurl. The text will focus
- mainly on the C interface but might apply fairly well on other interfaces as
- well as they usually follow the C one pretty closely.
-
- This document will refer to 'the user' as the person writing the source code
- that uses libcurl. That would probably be you or someone in your position.
- What will be generally referred to as 'the program' will be the collected
- source code that you write that is using libcurl for transfers. The program
- is outside libcurl and libcurl is outside of the program.
-
- To get the more details on all options and functions described herein, please
- refer to their respective man pages.
-
-Building
-
- There are many different ways to build C programs. This chapter will assume a
- unix-style build process. If you use a different build system, you can still
- read this to get general information that may apply to your environment as
- well.
-
- Compiling the Program
-
- Your compiler needs to know where the libcurl headers are
- located. Therefore you must set your compiler's include path to point to
- the directory where you installed them. The 'curl-config'[3] tool can be
- used to get this information:
-
- $ curl-config --cflags
-
- Linking the Program with libcurl
-
- When having compiled the program, you need to link your object files to
- create a single executable. For that to succeed, you need to link with
- libcurl and possibly also with other libraries that libcurl itself depends
- on. Like OpenSSL libraries, but even some standard OS libraries may be
- needed on the command line. To figure out which flags to use, once again
- the 'curl-config' tool comes to the rescue:
-
- $ curl-config --libs
-
- SSL or Not
-
- libcurl can be built and customized in many ways. One of the things that
- varies from different libraries and builds is the support for SSL-based
- transfers, like HTTPS and FTPS. If OpenSSL was detected properly at
- build-time, libcurl will be built with SSL support. To figure out if an
- installed libcurl has been built with SSL support enabled, use
- 'curl-config' like this:
-
- $ curl-config --feature
-
- And if SSL is supported, the keyword 'SSL' will be written to stdout,
- possibly together with a few other features that can be on and off on
- different libcurls.
-
- See also the "Features libcurl Provides" further down.
-
-
-Portable Code in a Portable World
-
- The people behind libcurl have put a considerable effort to make libcurl work
- on a large amount of different operating systems and environments.
-
- You program libcurl the same way on all platforms that libcurl runs on. There
- are only very few minor considerations that differs. If you just make sure to
- write your code portable enough, you may very well create yourself a very
- portable program. libcurl shouldn't stop you from that.
-
-
-Global Preparation
-
- The program must initialize some of the libcurl functionality globally. That
- means it should be done exactly once, no matter how many times you intend to
- use the library. Once for your program's entire life time. This is done using
-
- curl_global_init()
-
- and it takes one parameter which is a bit pattern that tells libcurl what to
- initialize. Using CURL_GLOBAL_ALL will make it initialize all known internal
- sub modules, and might be a good default option. The current two bits that
- are specified are:
-
- CURL_GLOBAL_WIN32 which only does anything on Windows machines. When used on
- a Windows machine, it'll make libcurl initialize the win32 socket
- stuff. Without having that initialized properly, your program cannot use
- sockets properly. You should only do this once for each application, so if
- your program already does this or of another library in use does it, you
- should not tell libcurl to do this as well.
-
- CURL_GLOBAL_SSL which only does anything on libcurls compiled and built
- SSL-enabled. On these systems, this will make libcurl initialize OpenSSL
- properly for this application. This is only needed to do once for each
- application so if your program or another library already does this, this
- bit should not be needed.
-
- libcurl has a default protection mechanism that detects if curl_global_init()
- hasn't been called by the time curl_easy_perform() is called and if that is
- the case, libcurl runs the function itself with a guessed bit pattern. Please
- note that depending solely on this is not considered nice nor very good.
-
- When the program no longer uses libcurl, it should call
- curl_global_cleanup(), which is the opposite of the init call. It will then
- do the reversed operations to cleanup the resources the curl_global_init()
- call initialized.
-
- Repeated calls to curl_global_init() and curl_global_cleanup() should be
- avoided. They should only be called once each.
-
-
-Features libcurl Provides
-
- It is considered best-practice to determine libcurl features run-time rather
- than build-time (if possible of course). By calling curl_version_info() and
- checking tout he details of the returned struct, your program can figure out
- exactly what the currently running libcurl supports.
-
-
-Handle the Easy libcurl
-
- libcurl first introduced the so called easy interface. All operations in the
- easy interface are prefixed with 'curl_easy'.
-
- Recent libcurl versions also offer the multi interface. More about that
- interface, what it is targeted for and how to use it is detailed in a
- separate chapter further down. You still need to understand the easy
- interface first, so please continue reading for better understanding.
-
- To use the easy interface, you must first create yourself an easy handle. You
- need one handle for each easy session you want to perform. Basically, you
- should use one handle for every thread you plan to use for transferring. You
- must never share the same handle in multiple threads.
-
- Get an easy handle with
-
- easyhandle = curl_easy_init();
-
- It returns an easy handle. Using that you proceed to the next step: setting
- up your preferred actions. A handle is just a logic entity for the upcoming
- transfer or series of transfers.
-
- You set properties and options for this handle using curl_easy_setopt(). They
- control how the subsequent transfer or transfers will be made. Options remain
- set in the handle until set again to something different. Alas, multiple
- requests using the same handle will use the same options.
-
- Many of the options you set in libcurl are "strings", pointers to data
- terminated with a zero byte. Keep in mind that when you set strings with
- curl_easy_setopt(), libcurl will not copy the data. It will merely point to
- the data. You MUST make sure that the data remains available for libcurl to
- use until finished or until you use the same option again to point to
- something else.
-
- One of the most basic properties to set in the handle is the URL. You set
- your preferred URL to transfer with CURLOPT_URL in a manner similar to:
-
- curl_easy_setopt(easyhandle, CURLOPT_URL, "http://curl.haxx.se/");
-
- Let's assume for a while that you want to receive data as the URL identifies
- a remote resource you want to get here. Since you write a sort of application
- that needs this transfer, I assume that you would like to get the data passed
- to you directly instead of simply getting it passed to stdout. So, you write
- your own function that matches this prototype:
-
- size_t write_data(void *buffer, size_t size, size_t nmemb, void *userp);
-
- You tell libcurl to pass all data to this function by issuing a function
- similar to this:
-
- curl_easy_setopt(easyhandle, CURLOPT_WRITEFUNCTION, write_data);
-
- You can control what data your function get in the forth argument by setting
- another property:
-
- curl_easy_setopt(easyhandle, CURLOPT_FILE, &internal_struct);
-
- Using that property, you can easily pass local data between your application
- and the function that gets invoked by libcurl. libcurl itself won't touch the
- data you pass with CURLOPT_FILE.
-
- libcurl offers its own default internal callback that'll take care of the
- data if you don't set the callback with CURLOPT_WRITEFUNCTION. It will then
- simply output the received data to stdout. You can have the default callback
- write the data to a different file handle by passing a 'FILE *' to a file
- opened for writing with the CURLOPT_FILE option.
-
- Now, we need to take a step back and have a deep breath. Here's one of those
- rare platform-dependent nitpicks. Did you spot it? On some platforms[2],
- libcurl won't be able to operate on files opened by the program. Thus, if you
- use the default callback and pass in a an open file with CURLOPT_FILE, it
- will crash. You should therefore avoid this to make your program run fine
- virtually everywhere.
-
- There are of course many more options you can set, and we'll get back to a
- few of them later. Let's instead continue to the actual transfer:
-
- success = curl_easy_perform(easyhandle);
-
- The curl_easy_perform() will connect to the remote site, do the necessary
- commands and receive the transfer. Whenever it receives data, it calls the
- callback function we previously set. The function may get one byte at a time,
- or it may get many kilobytes at once. libcurl delivers as much as possible as
- often as possible. Your callback function should return the number of bytes
- it "took care of". If that is not the exact same amount of bytes that was
- passed to it, libcurl will abort the operation and return with an error code.
-
- When the transfer is complete, the function returns a return code that
- informs you if it succeeded in its mission or not. If a return code isn't
- enough for you, you can use the CURLOPT_ERRORBUFFER to point libcurl to a
- buffer of yours where it'll store a human readable error message as well.
-
- If you then want to transfer another file, the handle is ready to be used
- again. Mind you, it is even preferred that you re-use an existing handle if
- you intend to make another transfer. libcurl will then attempt to re-use the
- previous
-
-
-Multi-threading issues
-
- libcurl is completely thread safe, except for two issues: signals and alarm
- handlers. Signals are needed for a SIGPIPE handler, and the alarm() Bacall
- is used to catch timeouts (mostly during ENS lookup).
-
- If you are accessing HTTPS or FTPS URLs in a multi-threaded manner, you are
- then of course using OpenSSL multi-threaded and it has itself a few
- requirements on this. Basilio, you need to provide one or two functions to
- allow it to function properly. For all details, see this:
-
- http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION
-
- When using multiple threads you should set the CURLOPT_NOSIGNAL option to
- TRUE for all handles. Everything will work fine except that timeouts are not
- honored during the DNS lookup - which you can work around by building libcurl
- with c-ares support. c-ares is a library that provides asynchronous name
- resolves. Unfortunately, c-ares does not yet support IPv6.
-
- Also, note that CURLOPT_DNS_USE_GLOBAL_CACHE is not thread-safe.
-
-When It Doesn't Work
-
- There will always be times when the transfer fails for some reason. You might
- have set the wrong libcurl option or misunderstood what the libcurl option
- actually does, or the remote server might return non-standard replies that
- confuse the library which then confuses your program.
-
- There's one golden rule when these things occur: set the CURLOPT_VERBOSE
- option to TRUE. It'll cause the library to spew out the entire protocol
- details it sends, some internal info and some received protocol data as well
- (especially when using FTP). If you're using HTTP, adding the headers in the
- received output to study is also a clever way to get a better understanding
- why the server behaves the way it does. Include headers in the normal body
- output with CURLOPT_HEADER set TRUE.
-
- Of course there are bugs left. We need to get to know about them to be able
- to fix them, so we're quite dependent on your bug reports! When you do report
- suspected bugs in libcurl, please include as much details you possibly can: a
- protocol dump that CURLOPT_VERBOSE produces, library version, as much as
- possible of your code that uses libcurl, operating system name and version,
- compiler name and version etc.
-
- If CURLOPT_VERBOSE is not enough, you increase the level of debug data your
- application receive by using the CURLOPT_DEBUGFUNCTION.
-
- Getting some in-depth knowledge about the protocols involved is never wrong,
- and if you're trying to do funny things, you might very well understand
- libcurl and how to use it better if you study the appropriate RFC documents
- at least briefly.
-
-
-Upload Data to a Remote Site
-
- libcurl tries to keep a protocol independent approach to most transfers, thus
- uploading to a remote FTP site is very similar to uploading data to a HTTP
- server with a PUT request.
-
- Of course, first you either create an easy handle or you re-use one existing
- one. Then you set the URL to operate on just like before. This is the remote
- URL, that we now will upload.
-
- Since we write an application, we most likely want libcurl to get the upload
- data by asking us for it. To make it do that, we set the read callback and
- the custom pointer libcurl will pass to our read callback. The read callback
- should have a prototype similar to:
-
- size_t function(char *bufptr, size_t size, size_t nitems, void *userp);
-
- Where bufptr is the pointer to a buffer we fill in with data to upload and
- size*nitems is the size of the buffer and therefore also the maximum amount
- of data we can return to libcurl in this call. The 'userp' pointer is the
- custom pointer we set to point to a struct of ours to pass private data
- between the application and the callback.
-
- curl_easy_setopt(easyhandle, CURLOPT_READFUNCTION, read_function);
-
- curl_easy_setopt(easyhandle, CURLOPT_INFILE, &filedata);
-
- Tell libcurl that we want to upload:
-
- curl_easy_setopt(easyhandle, CURLOPT_UPLOAD, TRUE);
-
- A few protocols won't behave properly when uploads are done without any prior
- knowledge of the expected file size. So, set the upload file size using the
- CURLOPT_INFILESIZE_LARGE for all known file sizes like this[1]:
-
- /* in this example, file_size must be an off_t variable */
- curl_easy_setopt(easyhandle, CURLOPT_INFILESIZE_LARGE, file_size);
-
- When you call curl_easy_perform() this time, it'll perform all the necessary
- operations and when it has invoked the upload it'll call your supplied
- callback to get the data to upload. The program should return as much data as
- possible in every invoke, as that is likely to make the upload perform as
- fast as possible. The callback should return the number of bytes it wrote in
- the buffer. Returning 0 will signal the end of the upload.
-
-
-Passwords
-
- Many protocols use or even require that user name and password are provided
- to be able to download or upload the data of your choice. libcurl offers
- several ways to specify them.
-
- Most protocols support that you specify the name and password in the URL
- itself. libcurl will detect this and use them accordingly. This is written
- like this:
-
- protocol://user:password@example.com/path/
-
- If you need any odd letters in your user name or password, you should enter
- them URL encoded, as %XX where XX is a two-digit hexadecimal number.
-
- libcurl also provides options to set various passwords. The user name and
- password as shown embedded in the URL can instead get set with the
- CURLOPT_USERPWD option. The argument passed to libcurl should be a char * to
- a string in the format "user:password:". In a manner like this:
-
- curl_easy_setopt(easyhandle, CURLOPT_USERPWD, "myname:thesecret");
-
- Another case where name and password might be needed at times, is for those
- users who need to authenticate themselves to a proxy they use. libcurl offers
- another option for this, the CURLOPT_PROXYUSERPWD. It is used quite similar
- to the CURLOPT_USERPWD option like this:
-
- curl_easy_setopt(easyhandle, CURLOPT_PROXYUSERPWD, "myname:thesecret");
-
- There's a long time unix "standard" way of storing ftp user names and
- passwords, namely in the $HOME/.netrc file. The file should be made private
- so that only the user may read it (see also the "Security Considerations"
- chapter), as it might contain the password in plain text. libcurl has the
- ability to use this file to figure out what set of user name and password to
- use for a particular host. As an extension to the normal functionality,
- libcurl also supports this file for non-FTP protocols such as HTTP. To make
- curl use this file, use the CURLOPT_NETRC option:
-
- curl_easy_setopt(easyhandle, CURLOPT_NETRC, TRUE);
-
- And a very basic example of how such a .netrc file may look like:
-
- machine myhost.mydomain.com
- login userlogin
- password secretword
-
- All these examples have been cases where the password has been optional, or
- at least you could leave it out and have libcurl attempt to do its job
- without it. There are times when the password isn't optional, like when
- you're using an SSL private key for secure transfers.
-
- To pass the known private key password to libcurl:
-
- curl_easy_setopt(easyhandle, CURLOPT_SSLKEYPASSWD, "keypassword");
-
-
-HTTP Authentication
-
- The previous chapter showed how to set user name and password for getting
- URLs that require authentication. When using the HTTP protocol, there are
- many different ways a client can provide those credentials to the server and
- you can control what way libcurl will (attempt to) use. The default HTTP
- authentication method is called 'Basic', which is sending the name and
- password in clear-text in the HTTP request, base64-encoded. This is insecure.
-
- At the time of this writing libcurl can be built to use: Basic, Digest, NTLM,
- Negotiate, GSS-Negotiate and SPNEGO. You can tell libcurl which one to use
- with CURLOPT_HTTPAUTH as in:
-
- curl_easy_setopt(easyhandle, CURLOPT_HTTPAUTH, CURLAUTH_DIGEST);
-
- And when you send authentication to a proxy, you can also set authentication
- type the same way but instead with CURLOPT_PROXYAUTH:
-
- curl_easy_setopt(easyhandle, CURLOPT_PROXYAUTH, CURLAUTH_NTLM);
-
- Both these options allow you to set multiple types (by ORing them together),
- to make libcurl pick the most secure one out of the types the server/proxy
- claims to support. This method does however add a round-trip since libcurl
- must first ask the server what it supports:
-
- curl_easy_setopt(easyhandle, CURLOPT_HTTPAUTH,
- CURLAUTH_DIGEST|CURLAUTH_BASIC);
-
- For convenience, you can use the 'CURLAUTH_ANY' define (instead of a list
- with specific types) which allows libcurl to use whatever method it wants.
-
- When asking for multiple types, libcurl will pick the available one it
- considers "best" in its own internal order of preference.
-
-
-HTTP POSTing
-
- We get many questions regarding how to issue HTTP POSTs with libcurl the
- proper way. This chapter will thus include examples using both different
- versions of HTTP POST that libcurl supports.
-
- The first version is the simple POST, the most common version, that most HTML
- pages using the <form> tag uses. We provide a pointer to the data and tell
- libcurl to post it all to the remote site:
-
- char *data="name=daniel&project=curl";
- curl_easy_setopt(easyhandle, CURLOPT_POSTFIELDS, data);
- curl_easy_setopt(easyhandle, CURLOPT_URL, "http://posthere.com/");
-
- curl_easy_perform(easyhandle); /* post away! */
-
- Simple enough, huh? Since you set the POST options with the
- CURLOPT_POSTFIELDS, this automatically switches the handle to use POST in the
- upcoming request.
-
- Ok, so what if you want to post binary data that also requires you to set the
- Content-Type: header of the post? Well, binary posts prevents libcurl from
- being able to do strlen() on the data to figure out the size, so therefore we
- must tell libcurl the size of the post data. Setting headers in libcurl
- requests are done in a generic way, by building a list of our own headers and
- then passing that list to libcurl.
-
- struct curl_slist *headers=NULL;
- headers = curl_slist_append(headers, "Content-Type: text/xml");
-
- /* post binary data */
- curl_easy_setopt(easyhandle, CURLOPT_POSTFIELDS, binaryptr);
-
- /* set the size of the postfields data */
- curl_easy_setopt(easyhandle, CURLOPT_POSTFIELDSIZE, 23);
-
- /* pass our list of custom made headers */
- curl_easy_setopt(easyhandle, CURLOPT_HTTPHEADER, headers);
-
- curl_easy_perform(easyhandle); /* post away! */
-
- curl_slist_free_all(headers); /* free the header list */
-
- While the simple examples above cover the majority of all cases where HTTP
- POST operations are required, they don't do multi-part formposts. Multi-part
- formposts were introduced as a better way to post (possibly large) binary
- data and was first documented in the RFC1867. They're called multi-part
- because they're built by a chain of parts, each being a single unit. Each
- part has its own name and contents. You can in fact create and post a
- multi-part formpost with the regular libcurl POST support described above, but
- that would require that you build a formpost yourself and provide to
- libcurl. To make that easier, libcurl provides curl_formadd(). Using this
- function, you add parts to the form. When you're done adding parts, you post
- the whole form.
-
- The following example sets two simple text parts with plain textual contents,
- and then a file with binary contents and upload the whole thing.
-
- struct curl_httppost *post=NULL;
- struct curl_httppost *last=NULL;
- curl_formadd(&post, &last,
- CURLFORM_COPYNAME, "name",
- CURLFORM_COPYCONTENTS, "daniel", CURLFORM_END);
- curl_formadd(&post, &last,
- CURLFORM_COPYNAME, "project",
- CURLFORM_COPYCONTENTS, "curl", CURLFORM_END);
- curl_formadd(&post, &last,
- CURLFORM_COPYNAME, "logotype-image",
- CURLFORM_FILECONTENT, "curl.png", CURLFORM_END);
-
- /* Set the form info */
- curl_easy_setopt(easyhandle, CURLOPT_HTTPPOST, post);
-
- curl_easy_perform(easyhandle); /* post away! */
-
- /* free the post data again */
- curl_formfree(post);
-
- Multipart formposts are chains of parts using MIME-style separators and
- headers. It means that each one of these separate parts get a few headers set
- that describe the individual content-type, size etc. To enable your
- application to handicraft this formpost even more, libcurl allows you to
- supply your own set of custom headers to such an individual form part. You
- can of course supply headers to as many parts you like, but this little
- example will show how you set headers to one specific part when you add that
- to the post handle:
-
- struct curl_slist *headers=NULL;
- headers = curl_slist_append(headers, "Content-Type: text/xml");
-
- curl_formadd(&post, &last,
- CURLFORM_COPYNAME, "logotype-image",
- CURLFORM_FILECONTENT, "curl.xml",
- CURLFORM_CONTENTHEADER, headers,
- CURLFORM_END);
-
- curl_easy_perform(easyhandle); /* post away! */
-
- curl_formfree(post); /* free post */
- curl_slist_free_all(post); /* free custom header list */
-
- Since all options on an easyhandle are "sticky", they remain the same until
- changed even if you do call curl_easy_perform(), you may need to tell curl to
- go back to a plain GET request if you intend to do such a one as your next
- request. You force an easyhandle to back to GET by using the CURLOPT_HTTPGET
- option:
-
- curl_easy_setopt(easyhandle, CURLOPT_HTTPGET, TRUE);
-
- Just setting CURLOPT_POSTFIELDS to "" or NULL will *not* stop libcurl from
- doing a POST. It will just make it POST without any data to send!
-
-
-Showing Progress
-
- For historical and traditional reasons, libcurl has a built-in progress meter
- that can be switched on and then makes it presents a progress meter in your
- terminal.
-
- Switch on the progress meter by, oddly enough, set CURLOPT_NOPROGRESS to
- FALSE. This option is set to TRUE by default.
-
- For most applications however, the built-in progress meter is useless and
- what instead is interesting is the ability to specify a progress
- callback. The function pointer you pass to libcurl will then be called on
- irregular intervals with information about the current transfer.
-
- Set the progress callback by using CURLOPT_PROGRESSFUNCTION. And pass a
- pointer to a function that matches this prototype:
-
- int progress_callback(void *clientp,
- double dltotal,
- double dlnow,
- double ultotal,
- double ulnow);
-
- If any of the input arguments is unknown, a 0 will be passed. The first
- argument, the 'clientp' is the pointer you pass to libcurl with
- CURLOPT_PROGRESSDATA. libcurl won't touch it.
-
-
-libcurl with C++
-
- There's basically only one thing to keep in mind when using C++ instead of C
- when interfacing libcurl:
-
- "The Callbacks Must Be Plain C"
-
- So if you want a write callback set in libcurl, you should put it within
- 'extern'. Similar to this:
-
- extern "C" {
- size_t write_data(void *ptr, size_t size, size_t nmemb,
- void *ourpointer)
- {
- /* do what you want with the data */
- }
- }
-
- This will of course effectively turn the callback code into C. There won't be
- any "this" pointer available etc.
-
-
-Proxies
-
- What "proxy" means according to Merriam-Webster: "a person authorized to act
- for another" but also "the agency, function, or office of a deputy who acts
- as a substitute for another".
-
- Proxies are exceedingly common these days. Companies often only offer
- Internet access to employees through their HTTP proxies. Network clients or
- user-agents ask the proxy for documents, the proxy does the actual request
- and then it returns them.
-
- libcurl has full support for HTTP proxies, so when a given URL is wanted,
- libcurl will ask the proxy for it instead of trying to connect to the actual
- host identified in the URL.
-
- The fact that the proxy is a HTTP proxy puts certain restrictions on what can
- actually happen. A requested URL that might not be a HTTP URL will be still
- be passed to the HTTP proxy to deliver back to libcurl. This happens
- transparently, and an application may not need to know. I say "may", because
- at times it is very important to understand that all operations over a HTTP
- proxy is using the HTTP protocol. For example, you can't invoke your own
- custom FTP commands or even proper FTP directory listings.
-
- Proxy Options
-
- To tell libcurl to use a proxy at a given port number:
-
- curl_easy_setopt(easyhandle, CURLOPT_PROXY, "proxy-host.com:8080");
-
- Some proxies require user authentication before allowing a request, and
- you pass that information similar to this:
-
- curl_easy_setopt(easyhandle, CURLOPT_PROXYUSERPWD, "user:password");
-
- If you want to, you can specify the host name only in the CURLOPT_PROXY
- option, and set the port number separately with CURLOPT_PROXYPORT.
-
- Environment Variables
-
- libcurl automatically checks and uses a set of environment variables to
- know what proxies to use for certain protocols. The names of the variables
- are following an ancient de facto standard and are built up as
- "[protocol]_proxy" (note the lower casing). Which makes the variable
- 'http_proxy' checked for a name of a proxy to use when the input URL is
- HTTP. Following the same rule, the variable named 'ftp_proxy' is checked
- for FTP URLs. Again, the proxies are always HTTP proxies, the different
- names of the variables simply allows different HTTP proxies to be used.
-
- The proxy environment variable contents should be in the format
- "[protocol://][user:password]machine[:port]". Where the protocol:// part
- is simply ignored if present (so http://proxy and bluerk://proxy will do
- the same) and the optional port number specifies on which port the proxy
- operates on the host. If not specified, the internal default port number
- will be used and that is most likely *not* the one you would like it to
- be.
-
- There are two special environment variables. 'all_proxy' is what sets
- proxy for any URL in case the protocol specific variable wasn't set, and
- 'no_proxy' defines a list of hosts that should not use a proxy even though
- a variable may say so. If 'no_proxy' is a plain asterisk ("*") it matches
- all hosts.
-
- SSL and Proxies
-
- SSL is for secure point-to-point connections. This involves strong
- encryption and similar things, which effectively makes it impossible for a
- proxy to operate as a "man in between" which the proxy's task is, as
- previously discussed. Instead, the only way to have SSL work over a HTTP
- proxy is to ask the proxy to tunnel trough everything without being able
- to check or fiddle with the traffic.
-
- Opening an SSL connection over a HTTP proxy is therefor a matter of asking
- the proxy for a straight connection to the target host on a specified
- port. This is made with the HTTP request CONNECT. ("please mr proxy,
- connect me to that remote host").
-
- Because of the nature of this operation, where the proxy has no idea what
- kind of data that is passed in and out through this tunnel, this breaks
- some of the very few advantages that come from using a proxy, such as
- caching. Many organizations prevent this kind of tunneling to other
- destination port numbers than 443 (which is the default HTTPS port
- number).
-
- Tunneling Through Proxy
-
- As explained above, tunneling is required for SSL to work and often even
- restricted to the operation intended for SSL; HTTPS.
-
- This is however not the only time proxy-tunneling might offer benefits to
- you or your application.
-
- As tunneling opens a direct connection from your application to the remote
- machine, it suddenly also re-introduces the ability to do non-HTTP
- operations over a HTTP proxy. You can in fact use things such as FTP
- upload or FTP custom commands this way.
-
- Again, this is often prevented by the administrators of proxies and is
- rarely allowed.
-
- Tell libcurl to use proxy tunneling like this:
-
- curl_easy_setopt(easyhandle, CURLOPT_HTTPPROXYTUNNEL, TRUE);
-
- In fact, there might even be times when you want to do plain HTTP
- operations using a tunnel like this, as it then enables you to operate on
- the remote server instead of asking the proxy to do so. libcurl will not
- stand in the way for such innovative actions either!
-
- Proxy Auto-Config
-
- Netscape first came up with this. It is basically a web page (usually
- using a .pac extension) with a javascript that when executed by the
- browser with the requested URL as input, returns information to the
- browser on how to connect to the URL. The returned information might be
- "DIRECT" (which means no proxy should be used), "PROXY host:port" (to tell
- the browser where the proxy for this particular URL is) or "SOCKS
- host:port" (to direct the browser to a SOCKS proxy).
-
- libcurl has no means to interpret or evaluate javascript and thus it
- doesn't support this. If you get yourself in a position where you face
- this nasty invention, the following advice have been mentioned and used in
- the past:
-
- - Depending on the javascript complexity, write up a script that
- translates it to another language and execute that.
-
- - Read the javascript code and rewrite the same logic in another language.
-
- - Implement a javascript interpreted, people have successfully used the
- Mozilla javascript engine in the past.
-
- - Ask your admins to stop this, for a static proxy setup or similar.
-
-
-Persistence Is The Way to Happiness
-
- Re-cycling the same easy handle several times when doing multiple requests is
- the way to go.
-
- After each single curl_easy_perform() operation, libcurl will keep the
- connection alive and open. A subsequent request using the same easy handle to
- the same host might just be able to use the already open connection! This
- reduces network impact a lot.
-
- Even if the connection is dropped, all connections involving SSL to the same
- host again, will benefit from libcurl's session ID cache that drastically
- reduces re-connection time.
-
- FTP connections that are kept alive saves a lot of time, as the command-
- response round-trips are skipped, and also you don't risk getting blocked
- without permission to login again like on many FTP servers only allowing N
- persons to be logged in at the same time.
-
- libcurl caches DNS name resolving results, to make lookups of a previously
- looked up name a lot faster.
-
- Other interesting details that improve performance for subsequent requests
- may also be added in the future.
-
- Each easy handle will attempt to keep the last few connections alive for a
- while in case they are to be used again. You can set the size of this "cache"
- with the CURLOPT_MAXCONNECTS option. Default is 5. It is very seldom any
- point in changing this value, and if you think of changing this it is often
- just a matter of thinking again.
-
- When the connection cache gets filled, libcurl must close an existing
- connection in order to get room for the new one. To know which connection to
- close, libcurl uses a "close policy" that you can affect with the
- CURLOPT_CLOSEPOLICY option. There's only two polices implemented as of this
- writing (libcurl 7.9.4) and they are:
-
- CURLCLOSEPOLICY_LEAST_RECENTLY_USED simply close the one that hasn't been
- used for the longest time. This is the default behavior.
-
- CURLCLOSEPOLICY_OLDEST closes the oldest connection, the one that was
- created the longest time ago.
-
- There are, or at least were, plans to support a close policy that would call
- a user-specified callback to let the user be able to decide which connection
- to dump when this is necessary and therefor is the CURLOPT_CLOSEFUNCTION an
- existing option still today. Nothing ever uses this though and this will not
- be used within the foreseeable future either.
-
- To force your upcoming request to not use an already existing connection (it
- will even close one first if there happens to be one alive to the same host
- you're about to operate on), you can do that by setting CURLOPT_FRESH_CONNECT
- to TRUE. In a similar spirit, you can also forbid the upcoming request to be
- "lying" around and possibly get re-used after the request by setting
- CURLOPT_FORBID_REUSE to TRUE.
-
-
-HTTP Headers Used by libcurl
-
- When you use libcurl to do HTTP requests, it'll pass along a series of
- headers automatically. It might be good for you to know and understand these
- ones.
-
- Host
-
- This header is required by HTTP 1.1 and even many 1.0 servers and should
- be the name of the server we want to talk to. This includes the port
- number if anything but default.
-
- Pragma
-
- "no-cache". Tells a possible proxy to not grab a copy from the cache but
- to fetch a fresh one.
-
- Accept:
-
- "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*". Cloned from a
- browser once a hundred years ago.
-
- Expect:
-
- When doing multi-part formposts, libcurl will set this header to
- "100-continue" to ask the server for an "OK" message before it proceeds
- with sending the data part of the post.
-
-
-Customizing Operations
-
- There is an ongoing development today where more and more protocols are built
- upon HTTP for transport. This has obvious benefits as HTTP is a tested and
- reliable protocol that is widely deployed and have excellent proxy-support.
-
- When you use one of these protocols, and even when doing other kinds of
- programming you may need to change the traditional HTTP (or FTP or...)
- manners. You may need to change words, headers or various data.
-
- libcurl is your friend here too.
-
- CUSTOMREQUEST
-
- If just changing the actual HTTP request keyword is what you want, like
- when GET, HEAD or POST is not good enough for you, CURLOPT_CUSTOMREQUEST
- is there for you. It is very simple to use:
-
- curl_easy_setopt(easyhandle, CURLOPT_CUSTOMREQUEST, "MYOWNRUQUEST");
-
- When using the custom request, you change the request keyword of the
- actual request you are performing. Thus, by default you make GET request
- but you can also make a POST operation (as described before) and then
- replace the POST keyword if you want to. You're the boss.
-
- Modify Headers
-
- HTTP-like protocols pass a series of headers to the server when doing the
- request, and you're free to pass any amount of extra headers that you
- think fit. Adding headers are this easy:
-
- struct curl_slist *headers=NULL; /* init to NULL is important */
-
- headers = curl_slist_append(headers, "Hey-server-hey: how are you?");
- headers = curl_slist_append(headers, "X-silly-content: yes");
-
- /* pass our list of custom made headers */
- curl_easy_setopt(easyhandle, CURLOPT_HTTPHEADER, headers);
-
- curl_easy_perform(easyhandle); /* transfer http */
-
- curl_slist_free_all(headers); /* free the header list */
-
- ... and if you think some of the internally generated headers, such as
- Accept: or Host: don't contain the data you want them to contain, you can
- replace them by simply setting them too:
-
- headers = curl_slist_append(headers, "Accept: Agent-007");
- headers = curl_slist_append(headers, "Host: munged.host.line");
-
- Delete Headers
-
- If you replace an existing header with one with no contents, you will
- prevent the header from being sent. Like if you want to completely prevent
- the "Accept:" header to be sent, you can disable it with code similar to
- this:
-
- headers = curl_slist_append(headers, "Accept:");
-
- Both replacing and canceling internal headers should be done with careful
- consideration and you should be aware that you may violate the HTTP
- protocol when doing so.
-
- Enforcing chunked transfer-encoding
-
- By making sure a request uses the custom header "Transfer-Encoding:
- chunked" when doing a non-GET HTTP operation, libcurl will switch over to
- "chunked" upload, even though the size of the data to upload might be
- known. By default, libcurl usually switches over to chunked upload
- automatically if the upload data size is unknown.
-
- HTTP Version
-
- There's only one aspect left in the HTTP requests that we haven't yet
- mentioned how to modify: the version field. All HTTP requests includes the
- version number to tell the server which version we support. libcurl speak
- HTTP 1.1 by default. Some very old servers don't like getting 1.1-requests
- and when dealing with stubborn old things like that, you can tell libcurl
- to use 1.0 instead by doing something like this:
-
- curl_easy_setopt(easyhandle, CURLOPT_HTTP_VERSION,
- CURLHTTP_VERSION_1_0);
-
- FTP Custom Commands
-
- Not all protocols are HTTP-like, and thus the above may not help you when
- you want to make for example your FTP transfers to behave differently.
-
- Sending custom commands to a FTP server means that you need to send the
- commands exactly as the FTP server expects them (RFC959 is a good guide
- here), and you can only use commands that work on the control-connection
- alone. All kinds of commands that requires data interchange and thus needs
- a data-connection must be left to libcurl's own judgment. Also be aware
- that libcurl will do its very best to change directory to the target
- directory before doing any transfer, so if you change directory (with CWD
- or similar) you might confuse libcurl and then it might not attempt to
- transfer the file in the correct remote directory.
-
- A little example that deletes a given file before an operation:
-
- headers = curl_slist_append(headers, "DELE file-to-remove");
-
- /* pass the list of custom commands to the handle */
- curl_easy_setopt(easyhandle, CURLOPT_QUOTE, headers);
-
- curl_easy_perform(easyhandle); /* transfer ftp data! */
-
- curl_slist_free_all(headers); /* free the header list */
-
- If you would instead want this operation (or chain of operations) to
- happen _after_ the data transfer took place the option to
- curl_easy_setopt() would instead be called CURLOPT_POSTQUOTE and used the
- exact same way.
-
- The custom FTP command will be issued to the server in the same order they
- are added to the list, and if a command gets an error code returned back
- from the server, no more commands will be issued and libcurl will bail out
- with an error code (CURLE_FTP_QUOTE_ERROR). Note that if you use
- CURLOPT_QUOTE to send commands before a transfer, no transfer will
- actually take place when a quote command has failed.
-
- If you set the CURLOPT_HEADER to true, you will tell libcurl to get
- information about the target file and output "headers" about it. The
- headers will be in "HTTP-style", looking like they do in HTTP.
-
- The option to enable headers or to run custom FTP commands may be useful
- to combine with CURLOPT_NOBODY. If this option is set, no actual file
- content transfer will be performed.
-
- FTP Custom CUSTOMREQUEST
-
- If you do what list the contents of a FTP directory using your own defined
- FTP command, CURLOPT_CUSTOMREQUEST will do just that. "NLST" is the
- default one for listing directories but you're free to pass in your idea
- of a good alternative.
-
-
-Cookies Without Chocolate Chips
-
- In the HTTP sense, a cookie is a name with an associated value. A server
- sends the name and value to the client, and expects it to get sent back on
- every subsequent request to the server that matches the particular conditions
- set. The conditions include that the domain name and path match and that the
- cookie hasn't become too old.
-
- In real-world cases, servers send new cookies to replace existing one to
- update them. Server use cookies to "track" users and to keep "sessions".
-
- Cookies are sent from server to clients with the header Set-Cookie: and
- they're sent from clients to servers with the Cookie: header.
-
- To just send whatever cookie you want to a server, you can use CURLOPT_COOKIE
- to set a cookie string like this:
-
- curl_easy_setopt(easyhandle, CURLOPT_COOKIE, "name1=var1; name2=var2;");
-
- In many cases, that is not enough. You might want to dynamically save
- whatever cookies the remote server passes to you, and make sure those cookies
- are then use accordingly on later requests.
-
- One way to do this, is to save all headers you receive in a plain file and
- when you make a request, you tell libcurl to read the previous headers to
- figure out which cookies to use. Set header file to read cookies from with
- CURLOPT_COOKIEFILE.
-
- The CURLOPT_COOKIEFILE option also automatically enables the cookie parser in
- libcurl. Until the cookie parser is enabled, libcurl will not parse or
- understand incoming cookies and they will just be ignored. However, when the
- parser is enabled the cookies will be understood and the cookies will be kept
- in memory and used properly in subsequent requests when the same handle is
- used. Many times this is enough, and you may not have to save the cookies to
- disk at all. Note that the file you specify to CURLOPT_COOKIEFILE doesn't
- have to exist to enable the parser, so a common way to just enable the parser
- and not read able might be to use a file name you know doesn't exist.
-
- If you rather use existing cookies that you've previously received with your
- Netscape or Mozilla browsers, you can make libcurl use that cookie file as
- input. The CURLOPT_COOKIEFILE is used for that too, as libcurl will
- automatically find out what kind of file it is and act accordingly.
-
- The perhaps most advanced cookie operation libcurl offers, is saving the
- entire internal cookie state back into a Netscape/Mozilla formatted cookie
- file. We call that the cookie-jar. When you set a file name with
- CURLOPT_COOKIEJAR, that file name will be created and all received cookies
- will be stored in it when curl_easy_cleanup() is called. This enabled cookies
- to get passed on properly between multiple handles without any information
- getting lost.
-
-
-FTP Peculiarities We Need
-
- FTP transfers use a second TCP/IP connection for the data transfer. This is
- usually a fact you can forget and ignore but at times this fact will come
- back to haunt you. libcurl offers several different ways to custom how the
- second connection is being made.
-
- libcurl can either connect to the server a second time or tell the server to
- connect back to it. The first option is the default and it is also what works
- best for all the people behind firewalls, NATs or IP-masquerading setups.
- libcurl then tells the server to open up a new port and wait for a second
- connection. This is by default attempted with EPSV first, and if that doesn't
- work it tries PASV instead. (EPSV is an extension to the original FTP spec
- and does not exist nor work on all FTP servers.)
-
- You can prevent libcurl from first trying the EPSV command by setting
- CURLOPT_FTP_USE_EPSV to FALSE.
-
- In some cases, you will prefer to have the server connect back to you for the
- second connection. This might be when the server is perhaps behind a firewall
- or something and only allows connections on a single port. libcurl then
- informs the remote server which IP address and port number to connect to.
- This is made with the CURLOPT_FTPPORT option. If you set it to "-", libcurl
- will use your system's "default IP address". If you want to use a particular
- IP, you can set the full IP address, a host name to resolve to an IP address
- or even a local network interface name that libcurl will get the IP address
- from.
-
- When doing the "PORT" approach, libcurl will attempt to use the EPRT and the
- LPRT before trying PORT, as they work with more protocols. You can disable
- this behavior by setting CURLOPT_FTP_USE_EPRT to FALSE.
-
-
-Headers Equal Fun
-
- Some protocols provide "headers", meta-data separated from the normal
- data. These headers are by default not included in the normal data stream,
- but you can make them appear in the data stream by setting CURLOPT_HEADER to
- TRUE.
-
- What might be even more useful, is libcurl's ability to separate the headers
- from the data and thus make the callbacks differ. You can for example set a
- different pointer to pass to the ordinary write callback by setting
- CURLOPT_WRITEHEADER.
-
- Or, you can set an entirely separate function to receive the headers, by
- using CURLOPT_HEADERFUNCTION.
-
- The headers are passed to the callback function one by one, and you can
- depend on that fact. It makes it easier for you to add custom header parsers
- etc.
-
- "Headers" for FTP transfers equal all the FTP server responses. They aren't
- actually true headers, but in this case we pretend they are! ;-)
-
-
-Post Transfer Information
-
- [ curl_easy_getinfo ]
-
-
-Security Considerations
-
- libcurl is in itself not insecure. If used the right way, you can use libcurl
- to transfer data pretty safely.
-
- There are of course many things to consider that may loosen up this
- situation:
-
- Command Lines
-
- If you use a command line tool (such as curl) that uses libcurl, and you
- give option to the tool on the command line those options can very likely
- get read by other users of your system when they use 'ps' or other tools
- to list currently running processes.
-
- To avoid this problem, never feed sensitive things to programs using
- command line options.
-
- .netrc
-
- .netrc is a pretty handy file/feature that allows you to login quickly and
- automatically to frequently visited sites. The file contains passwords in
- clear text and is a real security risk. In some cases, your .netrc is also
- stored in a home directory that is NFS mounted or used on another network
- based file system, so the clear text password will fly through your
- network every time anyone reads that file!
-
- To avoid this problem, don't use .netrc files and never store passwords in
- plain text anywhere.
-
- Clear Text Passwords
-
- Many of the protocols libcurl supports send name and password unencrypted
- as clear text (HTTP Basic authentication, FTP, TELNET etc). It is very
- easy for anyone on your network or a network nearby yours, to just fire up
- a network analyzer tool and eavesdrop on your passwords. Don't let the
- fact that HTTP uses base64 encoded passwords fool you. They may not look
- readable at a first glance, but they very easily "deciphered" by anyone
- within seconds.
-
- To avoid this problem, use protocols that don't let snoopers see your
- password: HTTPS, FTPS and FTP-kerberos are a few examples. HTTP Digest
- authentication allows this too, but isn't supported by libcurl as of this
- writing.
-
- Showing What You Do
-
- On a related issue, be aware that even in situations like when you have
- problems with libcurl and ask someone for help, everything you reveal in
- order to get best possible help might also impose certain security related
- risks. Host names, user names, paths, operating system specifics etc (not
- to mention passwords of course) may in fact be used by intruders to gain
- additional information of a potential target.
-
- To avoid this problem, you must of course use your common sense. Often,
- you can just edit out the sensitive data or just search/replace your true
- information with faked data.
-
-
-Multiple Transfers Using the multi Interface
-
- The easy interface as described in detail in this document is a synchronous
- interface that transfers one file at a time and doesn't return until its
- done.
-
- The multi interface on the other hand, allows your program to transfer
- multiple files in both directions at the same time, without forcing you to
- use multiple threads.
-
- To use this interface, you are better off if you first understand the basics
- of how to use the easy interface. The multi interface is simply a way to make
- multiple transfers at the same time, by adding up multiple easy handles in to
- a "multi stack".
-
- You create the easy handles you want and you set all the options just like
- you have been told above, and then you create a multi handle with
- curl_multi_init() and add all those easy handles to that multi handle with
- curl_multi_add_handle().
-
- When you've added the handles you have for the moment (you can still add new
- ones at any time), you start the transfers by call curl_multi_perform().
-
- curl_multi_perform() is asynchronous. It will only execute as little as
- possible and then return back control to your program. It is designed to
- never block. If it returns CURLM_CALL_MULTI_PERFORM you better call it again
- soon, as that is a signal that it still has local data to send or remote data
- to receive.
-
- The best usage of this interface is when you do a select() on all possible
- file descriptors or sockets to know when to call libcurl again. This also
- makes it easy for you to wait and respond to actions on your own
- application's sockets/handles. You figure out what to select() for by using
- curl_multi_fdset(), that fills in a set of fd_set variables for you with the
- particular file descriptors libcurl uses for the moment.
-
- When you then call select(), it'll return when one of the file handles signal
- action and you then call curl_multi_perform() to allow libcurl to do what it
- wants to do. Take note that libcurl does also feature some time-out code so
- we advice you to never use very long timeouts on select() before you call
- curl_multi_perform(), which thus should be called unconditionally every now
- and then even if none of its file descriptors have signaled ready. Another
- precaution you should use: always call curl_multi_fdset() immediately before
- the select() call since the current set of file descriptors may change when
- calling a curl function.
-
- If you want to stop the transfer of one of the easy handles in the stack, you
- can use curl_multi_remove_handle() to remove individual easy
- handles. Remember that easy handles should be curl_easy_cleanup()ed.
-
- When a transfer within the multi stack has finished, the counter of running
- transfers (as filled in by curl_multi_perform()) will decrease. When the
- number reaches zero, all transfers are done.
-
- curl_multi_info_read() can be used to get information about completed
- transfers. It then returns the CURLcode for each easy transfer, to allow you
- to figure out success on each individual transfer.
-
-
-SSL, Certificates and Other Tricks
-
- [ seeding, passwords, keys, certificates, ENGINE, ca certs ]
-
-
-Sharing Data Between Easy Handles
-
- [ fill in ]
-
------
-Footnotes:
-
-[1] = libcurl 7.10.3 and later have the ability to switch over to chunked
- Transfer-Encoding in cases were HTTP uploads are done with data of an
- unknown size.
-
-[2] = This happens on Windows machines when libcurl is built and used as a
- DLL. However, you can still do this on Windows if you link with a static
- library.
-
-[3] = The curl-config tool is generated at build-time (on unix-like systems)
- and should be installed with the 'make install' or similar instruction
- that installs the library, header files, man pages etc.
projects / curl / commitdiff