TODO: 1.8 Modified buffer size approach

Thoughts around buffer sizes and what might be possible to do...

diff --git a/docs/TODO b/docs/TODO
index 41e582155e567f1bc08a22603691fbb755380e18..79bd43006b6bf1d416e448d4725122bfbb067579 100644 (file)
--- a/docs/TODO
+++ b/docs/TODO
@@ -18,6 +18,7 @@
  1.5 get rid of PATH_MAX
  1.6 progress callback without doubles
  1.7 Happy Eyeball dual stack connect
+ 1,8 Modified buffer size approach
 
  2. libcurl - multi interface
  2.1 More non-blocking
@@ -178,6 +179,28 @@
 
     http://tools.ietf.org/html/rfc6555
 
+1.8 Modified buffer size approach
+
+ Current libcurl allocates a fixed 16K size buffer for download and an
+ additional 16K for upload. They are always unconditionally part of the easy
+ handle. If CRLF translations are requested, an additional 32K "scratch
+ buffer" is allocated. A total of 64K transfer buffers in the worst case.
+
+ First, while the handles are not actually in use these buffers could be freed
+ so that lingering handles just kept in queues or whatever waste less memory.
+
+ Secondly, SFTP is a protocol that needs to handle many ~30K blocks at once
+ since each need to be individually acked and therefore libssh2 must be
+ allowed to send (or receive) many separate ones in parallel to achieve high
+ transfer speeds. A current libcurl build with a 16K buffer makes that
+ impossible, but one with a 512K buffer will reach MUCH faster transfers. But
+ allocating 512K unconditionally for all buffers just in case they would like
+ to do fast SFTP transfers at some point is not a good solution either.
+
+ Dynamically allocate buffer size depending on protocol in use in combination
+ with freeing it after each individual transfer? Other suggestions?
+
+
 2. libcurl - multi interface
 
 2.1 More non-blocking