]> granicus.if.org Git - python/commit
bpo-31945: Configurable blocksize in HTTP(S)Connection (#4279)
authorNir Soffer <nirsof@gmail.com>
Mon, 6 Nov 2017 21:16:37 +0000 (23:16 +0200)
committerVictor Stinner <victor.stinner@gmail.com>
Mon, 6 Nov 2017 21:16:37 +0000 (13:16 -0800)
commitad455cd9243319b896c86074ffeb3bf78a82f4ec
tree48a4989023b45958c9fa7dfe090936506998906d
parent30f4fa456ef626ad7a92759f492ec7a268f7af4e
bpo-31945: Configurable blocksize in HTTP(S)Connection (#4279)

blocksize was hardcoded to 8192, preventing efficient upload when using
file-like body. Add blocksize argument to __init__, so users can
configure the blocksize to fit their needs.

I tested this uploading data from /dev/zero to a web server dropping the
received data, to test the overhead of the HTTPConnection.send() with a
file-like object.

Here is an example 10g upload with the default buffer size (8192):

$ time ~/src/cpython/release/python upload-httplib.py 10 https://localhost:8000/
Uploaded 10.00g in 17.53 seconds (584.00m/s)

real 0m17.574s
user 0m8.887s
sys 0m5.971s

Same with 512k blocksize:

$ time ~/src/cpython/release/python upload-httplib.py 10 https://localhost:8000/
Uploaded 10.00g in 6.60 seconds (1551.15m/s)

real 0m6.641s
user 0m3.426s
sys 0m2.162s

In real world usage the difference will be smaller, depending on the
local and remote storage and the network.

See https://github.com/nirs/http-bench for more info.
Doc/library/http.client.rst
Doc/whatsnew/3.7.rst
Lib/http/client.py
Lib/test/test_httplib.py
Misc/NEWS.d/next/Library/2017-11-05-01-17-12.bpo-31945.TLPBtS.rst [new file with mode: 0644]