How Do You Download Big Files with PowerShell?

Problem scenario
You want to download a very large file using PowerShell (from a file share on your network or URL on the Internet).  But the file is larger than the available RAM on your server.  How do you download a large file using Powershell?

Solution
Invoke-restmethod, Invoke-webrequest, curl and wget (even with the redirect ">" command) all put the file being downloaded into memory.  If RAM is scarce on your system or the file is very large, you will want to avoid bringing the file into memory.  To bypass memory consumption when downloading a file from a URL and write directly to disk using PowerShell, try these lines of code:

$webClient = New-Object System.Net.WebClient
$webURL = "https://www.continualintegration.com"
$filePath = "c:\temp\foo.html"
$Webclient.DownloadFile($webURL, $filePath)

On some systems, the above may not work.  In that case you could try to use start-bitstransfer.  The drawback to start-bitstransfer is that remotely executed scripts will not work.  This command, start-bitstransfer, only works interactively from local logon sessions or in a script that was launched by a local user.  To get around this shortcoming of start-bitstransfer you can create a Scheduled Task to run start-bitstransfer.  The Scheduled Task can be launched remotely with PowerShell.

To explain how to use start-bitstransfer in a script, follow these steps.  First write a PowerShell script that uses start-bitstransfer.  Next create a Scheduled Task to execute this PowerShell script, we will call it "foobar" for this example.  Finally run this as a PowerShell or batch command to invoke the Scheduled Task, use a command like this:

SCHTASKS /RUN /TN foobar

where "foobar" is the name of the Scheduled Task that you want to run.  Now the start-bitstransfer will run even when it is in a script and the script has been executed remotely.

---

To buy a book on PowerShell, see this page.