wget

wget

wget is a versatile and reliable command-line utility used for retrieving files from the web. It supports multiple protocols such as HTTP, HTTPS, and FTP, and is primarily used for downloading files, websites, and even entire directories. Unlike curl, which is more suited for web requests and API interactions, wget excels in batch downloading, recursive retrieval, and file resumption.

Key Features of wget:

  • Non-Interactive: wget can download files without requiring user interaction, making it ideal for automated tasks.
  • Supports Multiple Protocols: Works with HTTP, HTTPS, and FTP.
  • Resuming Downloads: Can resume downloads from where they left off, making it useful for large or interrupted downloads.
  • Recursive Downloading: Can download entire websites or directories for offline use.
  • Customizable: It provides a wide range of options for fine-tuning downloads, such as specifying download speed, retries, or proxy settings.

Description of Options and Switches

The following options are available in the wget form to customize your download:

  • -O: Save the downloaded file with a custom name (e.g., -O filename).
  • -c: Resume an interrupted download from where it left off.
  • –recursive: Download a website or directory recursively.
  • –limit-rate: Limit the download speed (e.g., --limit-rate=200k).
  • –quiet: Suppress all output except for errors.
  • –background: Run the download in the background.
  • –user-agent: Specify a custom user-agent string to mimic a browser.
  • –header: Add custom headers to the request (e.g., for API authentication).
  • –tries: Set the number of retry attempts in case of failure.
  • –timeout: Set the connection timeout period (e.g., --timeout=10).
  • –waitretry: Set the delay between retries (e.g., --waitretry=5).
  • –proxy: Use a proxy server for the download.

Common Options and Usage

These are some of the most common and useful options for customizing wget commands:

  • Basic Usage
    Retrieve a file from a specified URL:
    > wget http://example.com/file.txt
  • Resume Downloads
    Resume a partially downloaded file:
    > wget -c http://example.com/large-file.zip
  • Recursive Downloads
    Download entire websites or directories recursively:
    > wget --recursive http://example.com
  • Custom User-Agent
    Mimic a specific browser during the download:
    > wget --user-agent="Mozilla/5.0" http://example.com
  • Limit Download Speed
    Prevent excessive bandwidth usage by limiting the download speed:
    > wget --limit-rate=200k http://example.com/large-file.zip
  • Download Retry
    Automatically retry downloads on failure:
    > wget --tries=5 http://example.com
  • Background Download
    Run the download in the background:
    > wget --background http://example.com/file.txt
  • Quiet Mode
    Suppress output except for errors:
    > wget --quiet http://example.com

Advanced Options

For more advanced and specific use cases, you can combine these options to customize wget commands:

  • Authentication
    Access protected resources using credentials:
    > wget --user=username --password=password http://example.com/protected
  • Custom Headers
    Add custom HTTP headers to your request (e.g., for API calls):
    > wget --header="Authorization: Bearer your_token" http://example.com/api
  • Timeouts and Connection Control
    Specify timeouts and retry intervals for unreliable connections:
    > wget --timeout=10 --waitretry=5 http://example.com
  • Proxy Support
    Use a proxy for downloading:
    > wget --proxy=on http://example.com

Examples of wget Commands

  • Download a File and Save it with a Specific Name
    > wget -O custom-name.txt http://example.com/file.txt
  • Mirror an Entire Website
    Download an entire website with all necessary resources:
    > wget --mirror --convert-links --adjust-extension --page-requisites http://example.com
  • Download Only Specific File Types (e.g., Images)
    Download only .jpg and .png files from a website:
    > wget --accept jpg,png --recursive http://example.com
  • Simulate a Download for Testing
    Use wget in “spider” mode to test the availability of a URL without downloading:
    > wget --spider http://example.com
  • Test Server Response with Custom Headers
    Check the response headers from a server:
    > wget --header="X-Test-Header: value" http://example.com
  • Throttle Connections for Server-Friendly Downloading
    Limit the download rate and introduce wait times between requests:
    > wget --limit-rate=50k --wait=1 http://example.com
  • Inspect Response Headers Only
    Fetch only the headers, without downloading the content:
    > wget --server-response --spider http://example.com

Why Choose wget?

Customizable: Easily adjusted with a range of options to suit any task.

Offline Mirroring: Perfect for creating offline copies of websites.

Robust Retry Mechanism: Automatically retries failed downloads.

Batch Processing: Handles lists of URLs in a single run.

Protocol Support: Works seamlessly with HTTP, HTTPS, and FTP.