Multiple Simultaneous Connections in Wget
This page discusses implementation of a mechanism within Wget for the use of multiple connections simultaneously fetching files or portions of files.
Requirements
- It should be possible to limit the number of connections per-host (as determined by address, not domain name). There should be a hard limit of 2 connections per host, with perhaps(?) an option to lower the limit to 1 for all or a specified selection of hosts.
- There should be a configurable limit to the total number of simultaneous connections Wget will maintain at one time.
There must be support for multiple connections to be transferring data for different portions of the same resource (necessary for support of ../Metalink).
The ability to support multiple simultaneous download streams will need to be dependent on a "multiple simultaneous streams" property of the virtual output filesystem (../AbstractedFileSystem), as obviously if we're downloading to a tarball or .mht, we will not be able to handle more than one file at a time.
Things to Consider
- Concurrent parsing during fetch
- Handling of redirections to outside hosts
- How will logging work?
Support for multiple URLs for a single resource may be a good milestone on the way to supporting downloads of multiple resources simultaneously (less need to deal with filesystem issues, etc).