From: Graham Leggett Date: Wed, 7 Nov 2007 23:31:03 +0000 (+0000) Subject: core: Add the option to keep aside a request body up to a certain X-Git-Tag: 2.3.0~1282 X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=a6d30e317ae333026fa4c7bf260bc21d0fa26495;p=apache core: Add the option to keep aside a request body up to a certain size that would otherwise be discarded, to be consumed by filters such as mod_include. When enabled for a directory, POST requests to shtml files can be passed through to embedded scripts as POST requests, rather being downgraded to GET requests. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@592951 13f79535-47bb-0310-9956-ffa450edef68 --- diff --git a/CHANGES b/CHANGES index 7caa227e58..e32d1fdea3 100644 --- a/CHANGES +++ b/CHANGES @@ -2,6 +2,12 @@ Changes with Apache 2.3.0 [ When backported to 2.2.x, remove entry from this file ] + *) core: Add the option to keep aside a request body up to a certain + size that would otherwise be discarded, to be consumed by filters + such as mod_include. When enabled for a directory, POST requests + to shtml files can be passed through to embedded scripts as POST + requests, rather being downgraded to GET requests. [Graham Leggett] + *) mod_charset_lite: Don't crash when the request has no associated filename. [Jeff Trawick] diff --git a/docs/manual/mod/core.xml b/docs/manual/mod/core.xml index bf4f6d5b9d..bf5a62d833 100644 --- a/docs/manual/mod/core.xml +++ b/docs/manual/mod/core.xml @@ -1503,6 +1503,63 @@ the server configuration files MaxKeepAliveRequests + +KeptBodySize +Keep the request body instead of discarding it up to +the specified maximum size, for potential use by filters such as +mod_include. +KeptBodySize maximum size in bytes +KeptBodySize 0 +directory + + + +

Under normal circumstances, request handlers such as the + default handler for static files will discard the request body + when it is not needed by the request handler. As a result, + filters such as mod_include are limited to making GET requests + only when including other URLs as subrequests, even if the + original request was a POST request, as the discarded + request body is no longer available once filter processing is + taking place.

+ +

When this directive has a value greater than zero, request + handlers that would otherwise discard request bodies will + instead set the request body aside for use by filters up to + the maximum size specified. In the case of the mod_include + filter, an attempt to POST a request to the static + shtml file will cause any subrequests to be POST + requests, instead of GET requests as before.

+ +

This feature makes it possible to break up complex web pages and + web applications into small individual components, and combine + the components and the surrounding web page structure together + using mod_include. The components can take the + form of CGI programs, scripted languages, or URLs reverse proxied + into the URL space from another server using + mod_proxy.

+ +

Note: Each request set aside has to be set + aside in temporary RAM until the request is complete. As a result, + care should be taken to ensure sufficient RAM is available on the + server to support the intended load. Use of this directive + should be limited to where needed on targeted parts of your + URL space, and with the lowest possible value that is still big + enough to hold a request body.

+ +

If the request size sent by the client exceeds the maximum + size allocated by this directive, the server will return + 413 Request Entity Too Large.

+ +

Handlers such as mod_cgi that consume request + bodies for their own purposes rather than discard them do not take + this directive into account.

+ + + +mod_include documentation + + KeepAliveTimeout Amount of time the server will wait for subsequent diff --git a/docs/manual/mod/mod_include.xml b/docs/manual/mod/mod_include.xml index 851db2eb40..c526416e3c 100644 --- a/docs/manual/mod/mod_include.xml +++ b/docs/manual/mod/mod_include.xml @@ -343,14 +343,22 @@

If the specified URL is a CGI program, the program will be executed and its output inserted in place of the directive in the parsed file. You may include a query string in a CGI url:

- + <!--#include virtual="/cgi-bin/example.cgi?argument=value" --> - +

include virtual should be used in preference to exec cgi to include the output of CGI programs into an HTML document.

+ +

If the KeptBodySize + directive is correctly configured and valid for this included + file, attempts to POST requests to the enclosing HTML document + will be passed through to subrequests as POST requests as well. + Without the directive, all subrequests are processed as GET + requests.

+ diff --git a/include/ap_mmn.h b/include/ap_mmn.h index 3c8cc3354e..6ddaff5b39 100644 --- a/include/ap_mmn.h +++ b/include/ap_mmn.h @@ -139,14 +139,15 @@ * 20071023.2 (2.3.0-dev) Add ap_mod_status_reqtail * 20071023.3 (2.3.0-dev) Declare ap_time_process_request() as part of the * public scoreboard API. + * 20071108.1 (2.3.0-dev) Add the optional kept_body brigade to request_rec */ #define MODULE_MAGIC_COOKIE 0x41503234UL /* "AP24" */ #ifndef MODULE_MAGIC_NUMBER_MAJOR -#define MODULE_MAGIC_NUMBER_MAJOR 20071023 +#define MODULE_MAGIC_NUMBER_MAJOR 20071108 #endif -#define MODULE_MAGIC_NUMBER_MINOR 3 /* 0...n */ +#define MODULE_MAGIC_NUMBER_MINOR 1 /* 0...n */ /** * Determine if the server's current MODULE_MAGIC_NUMBER is at least a diff --git a/include/httpd.h b/include/httpd.h index 454e8b0681..332521446f 100644 --- a/include/httpd.h +++ b/include/httpd.h @@ -988,6 +988,9 @@ struct request_rec { /** A flag to determine if the eos bucket has been sent yet */ int eos_sent; + /** The optional kept body of the request. */ + apr_bucket_brigade *kept_body; + /* Things placed at the end of the record to avoid breaking binary * compatibility. It would be nice to remember to reorder the entire * record to improve 64bit alignment the next time we need to break diff --git a/modules/filters/mod_include.c b/modules/filters/mod_include.c index db8b5faa94..2003e20b20 100644 --- a/modules/filters/mod_include.c +++ b/modules/filters/mod_include.c @@ -1712,7 +1712,12 @@ static apr_status_t handle_include(include_ctx_t *ctx, ap_filter_t *f, } } else { - rr = ap_sub_req_lookup_uri(parsed_string, r, f->next); + if (r->kept_body) { + rr = ap_sub_req_method_uri(r->method, parsed_string, r, f->next); + } + else { + rr = ap_sub_req_lookup_uri(parsed_string, r, f->next); + } } if (!error_fmt && rr->status != HTTP_OK) { diff --git a/modules/http/http_core.c b/modules/http/http_core.c index e1a2efd3e9..c2607809d2 100644 --- a/modules/http/http_core.c +++ b/modules/http/http_core.c @@ -37,6 +37,7 @@ /* Handles for core filters */ AP_DECLARE_DATA ap_filter_rec_t *ap_http_input_filter_handle; +AP_DECLARE_DATA ap_filter_rec_t *ap_kept_body_input_filter_handle; AP_DECLARE_DATA ap_filter_rec_t *ap_http_header_filter_handle; AP_DECLARE_DATA ap_filter_rec_t *ap_chunk_filter_handle; AP_DECLARE_DATA ap_filter_rec_t *ap_http_outerror_filter_handle; @@ -89,6 +90,20 @@ static const char *set_keep_alive_max(cmd_parms *cmd, void *dummy, return NULL; } +static const char *set_kept_body_size(cmd_parms *cmd, void *dconf, + const char *arg) +{ + core_dir_conf *conf = dconf; + + if (APR_SUCCESS != apr_strtoff(&(conf->keep_body), arg, NULL, 0) + || conf->keep_body < 0) { + return "KeptBodySize must be a size in bytes, or zero."; + } + conf->keep_body_set = 1; + + return NULL; +} + static const command_rec http_cmds[] = { AP_INIT_TAKE1("KeepAliveTimeout", set_keep_alive_timeout, NULL, RSRC_CONF, "Keep-Alive timeout duration (sec)"), @@ -97,6 +112,8 @@ static const command_rec http_cmds[] = { "or 0 for infinite"), AP_INIT_TAKE1("KeepAlive", set_keep_alive, NULL, RSRC_CONF, "Whether persistent connections should be On or Off"), + AP_INIT_TAKE1("KeptBodySize", set_kept_body_size, NULL, ACCESS_CONF, + "Maximum size of request bodies kept aside for use by filters"), { NULL } }; @@ -268,6 +285,9 @@ static void register_hooks(apr_pool_t *p) ap_http_input_filter_handle = ap_register_input_filter("HTTP_IN", ap_http_filter, NULL, AP_FTYPE_PROTOCOL); + ap_kept_body_input_filter_handle = + ap_register_input_filter("KEPT_BODY", ap_kept_body_filter, + ap_kept_body_filter_init, AP_FTYPE_RESOURCE); ap_http_header_filter_handle = ap_register_output_filter("HTTP_HEADER", ap_http_header_filter, NULL, AP_FTYPE_PROTOCOL); @@ -283,12 +303,35 @@ static void register_hooks(apr_pool_t *p) ap_method_registry_init(p); } +static void *create_core_dir_config(apr_pool_t *p, char *dummy) +{ + core_dir_conf *new = + (core_dir_conf *) apr_pcalloc(p, sizeof(core_dir_conf)); + + new->keep_body_set = 0; /* unset */ + new->keep_body = 0; /* don't by default */ + + return (void *) new; +} + +static void *merge_core_dir_config(apr_pool_t *p, void *basev, void *addv) +{ + core_dir_conf *new = (core_dir_conf *) apr_pcalloc(p, sizeof(core_dir_conf)); + core_dir_conf *add = (core_dir_conf *) addv; + core_dir_conf *base = (core_dir_conf *) basev; + + new->keep_body = (add->keep_body_set == 0) ? base->keep_body : add->keep_body; + new->keep_body_set = add->keep_body_set || base->keep_body_set; + + return new; +} + module AP_MODULE_DECLARE_DATA http_module = { STANDARD20_MODULE_STUFF, - NULL, /* create per-directory config structure */ - NULL, /* merge per-directory config structures */ - NULL, /* create per-server config structure */ - NULL, /* merge per-server config structures */ - http_cmds, /* command apr_table_t */ - register_hooks /* register hooks */ + create_core_dir_config, /* create per-directory config structure */ + merge_core_dir_config, /* merge per-directory config structures */ + NULL, /* create per-server config structure */ + NULL, /* merge per-server config structures */ + http_cmds, /* command apr_table_t */ + register_hooks /* register hooks */ }; diff --git a/modules/http/http_filters.c b/modules/http/http_filters.c index bf17d8fedd..db18d6f40b 100644 --- a/modules/http/http_filters.c +++ b/modules/http/http_filters.c @@ -55,6 +55,8 @@ #include #endif +extern module AP_MODULE_DECLARE_DATA http_module; + static long get_chunk_size(char *); typedef struct http_filter_ctx { @@ -1145,8 +1147,11 @@ AP_CORE_DECLARE_NONSTD(apr_status_t) ap_http_header_filter(ap_filter_t *f, */ AP_DECLARE(int) ap_discard_request_body(request_rec *r) { - apr_bucket_brigade *bb; + apr_bucket_brigade *bb, *kept_body = NULL; + apr_bucket *e; int rv, seen_eos; + core_dir_conf *dconf; + apr_size_t left = 0; /* Sometimes we'll get in a state where the input handling has * detected an error where we want to drop the connection, so if @@ -1160,6 +1165,20 @@ AP_DECLARE(int) ap_discard_request_body(request_rec *r) return OK; } + /* We may want to save this body away if the administrator has + * asked us to do so for this directory. This allows the body + * to be reexamined by filters such as mod_include, even though + * the main request has no need for this body. + */ + if (!r->kept_body) { + dconf = ap_get_module_config(r->per_dir_config, + &http_module); + if (dconf->keep_body > 0) { + left = dconf->keep_body; + kept_body = apr_brigade_create(r->pool, r->connection->bucket_alloc); + } + } + bb = apr_brigade_create(r->pool, r->connection->bucket_alloc); seen_eos = 0; do { @@ -1211,10 +1230,33 @@ AP_DECLARE(int) ap_discard_request_body(request_rec *r) apr_brigade_destroy(bb); return HTTP_BAD_REQUEST; } + + /* If we have been asked to, keep the data up until the + * configured limit. If the limit is exceeded, we return an + * HTTP_REQUEST_ENTITY_TOO_LARGE response so the caller is + * clear the server couldn't handle their request. + */ + if (kept_body) { + if (len <= left) { + apr_bucket_copy(bucket, &e); + APR_BRIGADE_INSERT_TAIL(kept_body, e); + left -= len; + } + else { + apr_brigade_destroy(bb); + apr_brigade_destroy(kept_body); + return HTTP_REQUEST_ENTITY_TOO_LARGE; + } + } + } apr_brigade_cleanup(bb); } while (!seen_eos); + if (kept_body) { + r->kept_body = kept_body; + } + return OK; } @@ -1421,3 +1463,101 @@ apr_status_t ap_http_outerror_filter(ap_filter_t *f, return ap_pass_brigade(f->next, b); } +typedef struct kept_body_filter_ctx { + apr_off_t offset; + apr_off_t remaining; +} kept_body_ctx_t; + +/** + * Initialisation of filter to handle a kept body on subrequests. + * + * If a body is to be reinserted into a subrequest, any chunking will have + * been removed from the body during storage. We need to change the request + * from Transfer-Encoding: chunked to an explicit Content-Length. + */ +int ap_kept_body_filter_init(ap_filter_t *f) { + apr_off_t length = 0; + request_rec *r = f->r; + apr_bucket_brigade *kept_body = r->kept_body; + + if (kept_body) { + apr_table_unset(r->headers_in, "Transfer-Encoding"); + apr_brigade_length(kept_body, 1, &length); + apr_table_set(r->headers_in, "Content-Length", apr_off_t_toa(r->pool, length)); + } + + return OK; +} + +/** + * Filter to handle a kept body on subrequests. + * + * If a body has been previously kept by the request, and if a subrequest wants + * to re-insert the body into the request, this input filter makes it happen. + */ +apr_status_t ap_kept_body_filter(ap_filter_t *f, apr_bucket_brigade *b, + ap_input_mode_t mode, apr_read_type_e block, + apr_off_t readbytes) { + request_rec *r = f->r; + apr_bucket_brigade *kept_body = r->kept_body; + kept_body_ctx_t *ctx = f->ctx; + apr_bucket *ec, *e2; + apr_status_t rv; + + /* just get out of the way of things we don't want. */ + if (!kept_body || (mode != AP_MODE_READBYTES && mode != AP_MODE_GETLINE)) { + return ap_get_brigade(f->next, b, mode, block, readbytes); + } + + /* set up the context if it does not already exist */ + if (!ctx) { + f->ctx = ctx = apr_palloc(f->r->pool, sizeof(*ctx)); + ctx->offset = 0; + apr_brigade_length(kept_body, 1, &ctx->remaining); + } + + /* kept_body is finished, send next filter */ + if (ctx->remaining <= 0) { + return ap_get_brigade(f->next, b, mode, block, readbytes); + } + + /* send all of the kept_body, but no more */ + if (readbytes > ctx->remaining) { + readbytes = ctx->remaining; + } + + /* send part of the kept_body */ + if ((rv = apr_brigade_partition(kept_body, ctx->offset, &ec)) != APR_SUCCESS) { + ap_log_rerror(APLOG_MARK, APLOG_ERR, rv, r, + "apr_brigade_partition() failed on kept_body at %" APR_OFF_T_FMT, ctx->offset); + return rv; + } + if ((rv = apr_brigade_partition(kept_body, ctx->offset + readbytes, &e2)) != APR_SUCCESS) { + ap_log_rerror(APLOG_MARK, APLOG_ERR, rv, r, + "apr_brigade_partition() failed on kept_body at %" APR_OFF_T_FMT, ctx->offset + readbytes); + return rv; + } + + do { + apr_bucket *foo; + const char *str; + apr_size_t len; + + if (apr_bucket_copy(ec, &foo) != APR_SUCCESS) { + /* As above; this should not fail since the bucket has + * a known length, but just to be sure, this takes + * care of uncopyable buckets that do somehow manage + * to slip through. */ + /* XXX: check for failure? */ + apr_bucket_read(ec, &str, &len, APR_BLOCK_READ); + apr_bucket_copy(ec, &foo); + } + APR_BRIGADE_INSERT_TAIL(b, foo); + ec = APR_BUCKET_NEXT(ec); + } while (ec != e2); + + ctx->remaining -= readbytes; + ctx->offset += readbytes; + return APR_SUCCESS; + +} diff --git a/modules/http/mod_core.h b/modules/http/mod_core.h index 79810d003d..47b8a704ae 100644 --- a/modules/http/mod_core.h +++ b/modules/http/mod_core.h @@ -38,8 +38,17 @@ extern "C" { #endif +/** + * Core per-directory configuration. + */ +typedef struct { + apr_off_t keep_body; + int keep_body_set; +} core_dir_conf; + /* Handles for core filters */ extern AP_DECLARE_DATA ap_filter_rec_t *ap_http_input_filter_handle; +extern AP_DECLARE_DATA ap_filter_rec_t *ap_kept_body_input_filter_handle; extern AP_DECLARE_DATA ap_filter_rec_t *ap_http_header_filter_handle; extern AP_DECLARE_DATA ap_filter_rec_t *ap_chunk_filter_handle; extern AP_DECLARE_DATA ap_filter_rec_t *ap_http_outerror_filter_handle; @@ -52,6 +61,14 @@ apr_status_t ap_http_filter(ap_filter_t *f, apr_bucket_brigade *b, ap_input_mode_t mode, apr_read_type_e block, apr_off_t readbytes); +/* Filter to handle a kept body on subrequests */ +apr_status_t ap_kept_body_filter(ap_filter_t *f, apr_bucket_brigade *b, + ap_input_mode_t mode, apr_read_type_e block, + apr_off_t readbytes); + +/* Initialisation of filter to handle a kept body on subrequests */ +int ap_kept_body_filter_init(ap_filter_t *f); + /* HTTP/1.1 chunked transfer encoding filter. */ apr_status_t ap_http_chunk_filter(ap_filter_t *f, apr_bucket_brigade *b); diff --git a/server/request.c b/server/request.c index 14044be44b..402f3bda05 100644 --- a/server/request.c +++ b/server/request.c @@ -1566,6 +1566,16 @@ static request_rec *make_sub_request(const request_rec *r, * until some module interjects and changes the value. */ rnew->used_path_info = AP_REQ_DEFAULT_PATH_INFO; + + /* Pass on the kept body (if any) into the new request. */ + rnew->kept_body = r->kept_body; + + /* + * Add the KEPT_BODY filter, which will insert any body marked to be + * kept for the use of a subrequest, into the subrequest. + */ + ap_add_input_filter_handle(ap_kept_body_input_filter_handle, + NULL, rnew, rnew->connection); return rnew; }