SUSE Conversations


Use Wget as Your Linux Download Manager



By: npelton

October 27, 2008 4:03 pm

Reads:512

Comments:6

Rating:0

When trying to determine the best fit for a download manager, I eventually decided upon using wget for a number of reasons. After looking at my example, you may decide to give it a try.

Some of the Reasons I liked this solution.

  • Not having to integrate a Download Manager with a browser(s).
  • Ability to log in to remote site.
  • Easy integration with CRON or AT, so that the downloads could be scheduled for evenings and weekends.
  • It also seemed quits simple!

Proxy anyone?

First, as many may use a proxy server, when the following line is entered in the terminal window, it will define the proxy that you will use with wget (the proxy specified in your browser will not be automatically used). Obviously, if you don’t use a proxy for your browser, you won’t need to use the command.

tux:~ # export http_proxy="http://your-proxy.your-doamin.com:1234"

Where “your-proxy.your-domain.com:1234″ would be the proxy settings used in your browser.

For Single Use–Right Now!

After logging into the destination, right click the links you want and paste them one at a time into a text file, surround each link with single quote mark on each end, with a space between the links. Next, if it is a one time download, you would paste the links you just copied after the user and password info. The following command would download the desired file into the present working directory on the Linux host. (Note that entire command is all on one line and wraps to the next. Not sure of the limit, but I have used this with what seems to me to be incredibly long strings.) If you want to schedule it for later, either once or many times, you could paste the login info into the file and run it as a bash script (below) within AT or CRON.

tux:~ # wget --http-user=yourlogin --http-password=yourpassword 'url.domain.com/path/to/first/file' 'url.domain.com/path/to/second/file' 'url.domain.com/path/to/third/file' 'url.domain.com/path/to/fourth/file' 'url.domain.com/path/to/fifth/file' 

Sample bash script to run while you are at the beach or skiing with your family. (beach.sh)

#!/bin/bash
##Following changes to directory to download files to.
cd /tux/download1
##Following is the command specified above to start the download.
wget --http-user=yourlogin --http-password=yourpassword 'url.domain.com/path/to/first/file' 'url.domain.com/path/to/second/file' 'url.domain.com/path/to/third/file' 'url.domain.com/path/to/fourth/file' 'url.domain.com/path/to/fifth/file'
##Following would allow you to change to another directory and then repeat the command above and have the 
cd /tux/download2
##Then you would use wget for files for this location, etc.

If it was a job that you will want to run occasionally and the URL’s may change from time to time, use the AT command to schedule the “beach.sh” bash script to run. This is my use–I’ll run it starting noon tomorrow (Saturday) (until ?) and download 50 Gig or more of ISO’s and such while Internet usage is low. CRON would be the choice if you wanted to download the same files on a scheduled basis, but I will use AT here.

I want it to start when I’m at the beach and out of cell range!

Start the AT daemon, schedule the time, and then enter your command into it.

tux:~ #atd
tux:~ #at noon tomorrow
at>sh beach.sh
##Exit this with a <CNTRL-D)

Tomorrow you will be at the beach or skiing and this will run for you. Hope you enjoy the beach/skiing and that you find this has worked for you when you get back.. :0)

Nic

VN:F [1.9.22_1171]
Rating: 0.0/5 (0 votes cast)

Tags:
Categories: Enterprise Linux, Technical Solutions

Disclaimer: As with everything else at SUSE Conversations, this content is definitely not supported by SUSE (so don't even think of calling Support if you try something and it blows up).  It was contributed by a community member and is published "as is." It seems to have worked for at least one person, and might work for you. But please be sure to test, test, test before you do anything drastic with it.

6 Comments

  1. By:Anonymous

    Nice article but what about securing passwords in wget ? I do not want to keep my password in plain text in wget call or even in .wgetrc file, is there a way to encode them somehow ?

  2. By:Anonymous

    Curious. Can wget be used to download files larger then 4gb? I have tried numerous times (unsuccessfully) to download dvd iso’s from download.novell.com. Ive tried IE, Firefox and Chrome. All fail either stating the file exceeds 4gb or page cannot be found.

  3. By:npelton

    There are several options. For me, the connections that I attach to are https and I probably should have pointed that out (that it will work with a site that requires https–my only use is with several sites that require https). Also, for my application the idea was to use what was on hand, so I am using a workstation that is a few years old, and I am the only one who has access to it. I then distribute the content locally if applicable.

    A thought for you would be to store the password in wgetrc or netrc and then have the passwords overwritten via a script after authentication is made. Perhaps easier and more straightforward would be to use chmod to limit access to the applicable files. More info on this can be found on the man pages for wget.

  4. By:Anonymous

    Of the three browsers you mention, I have had the best luck using a current version of Firefox when downloading larger files. There were limits with IE6 (but I thought it was for anything over 2G), and I’d heard that the limit no longer exists for IE7, but I haven’t used it. Also haven’t done much with Chrome.

    I have downloaded many ISO’s and image files that are between 2.5 and 3.7 G with wget from Novell and other sites (I should mention all using HTTPS).. I haven’t had any problems with downloading any of them, although I don’t recall any over 4 Gig….

  5. By:Anonymous

    Just an FYI–Did look and noticed that some of the downloads were over 4 GB, (one 4.3 and one 4.1), so I expect it would work for you in this regard as well.

  6. By:wramthun

    This is a nice article and is the perfect solution to downloading ISO’s to, say, a media server in a data center. However, I’m unable to download any of the ISO’s via this solution since the Novel download site seems to build single use URL’s for downloads. Or am I missing something?

Comment

RSS