mathieu/granite-rust
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Auto merge of #33288 - cyplo:32834_retry_download, r=alexcrichton

Browse source 
Get a file during bootstrap to a temp location first.

When downloading a file in the bootstrap phase - get it to a temp
location first. Verify it there and only if downloaded properly move it
to the `cache` directory.

This should prevent `make` being stuck if the download was interrupted
or otherwise corrupted, as per discussion in #32834

The temporary files are deleted in case of an exception.

I was looking for some unit/integration tests around this and couldn't find any - presumably because this is being tested by just Travis launching it ? Let me know if it would be good to try to write tests around this. Thanks !
This commit is contained in:
bors 2016-05-08 14:57:57 -07:00
commit 6974800c6b
1 changed files with 38 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

49
src/bootstrap/bootstrap.py
View file

@ -16,20 +16,46 @@ import shutil
import subprocess
import sys
import tarfile
import tempfile
def get(url, path, verbose=False):
print("downloading " + url)
sha_url = url + ".sha256"
sha_path = path + ".sha256"
for _url, _path in ((url, path), (sha_url, sha_path)):
# see http://serverfault.com/questions/301128/how-to-download
if sys.platform == 'win32':
run(["PowerShell.exe", "/nologo", "-Command",
"(New-Object System.Net.WebClient)"
".DownloadFile('{}', '{}')".format(_url, _path)],
verbose=verbose)
else:
run(["curl", "-o", _path, _url], verbose=verbose)
with tempfile.NamedTemporaryFile(delete=False) as temp_file:
temp_path = temp_file.name
with tempfile.NamedTemporaryFile(suffix=".sha256", delete=False) as sha_file:
sha_path = sha_file.name
try:
download(sha_path, sha_url, verbose)
download(temp_path, url, verbose)
verify(temp_path, sha_path, verbose)
print("moving " + temp_path + " to " + path)
shutil.move(temp_path, path)
finally:
delete_if_present(sha_path)
delete_if_present(temp_path)
def delete_if_present(path):
if os.path.isfile(path):
print("removing " + path)
os.unlink(path)
def download(path, url, verbose):
print("downloading " + url + " to " + path)
# see http://serverfault.com/questions/301128/how-to-download
if sys.platform == 'win32':
run(["PowerShell.exe", "/nologo", "-Command",
"(New-Object System.Net.WebClient)"
".DownloadFile('{}', '{}')".format(url, path)],
verbose=verbose)
else:
run(["curl", "-o", path, url], verbose=verbose)
def verify(path, sha_path, verbose):
print("verifying " + path)
with open(path, "rb") as f:
found = hashlib.sha256(f.read()).hexdigest()
@ -43,6 +69,7 @@ def get(url, path, verbose=False):
raise RuntimeError(err)
sys.exit(err)
def unpack(tarball, dst, verbose=False, match=None):
print("extracting " + tarball)
fname = os.path.basename(tarball).replace(".tar.gz", "")