Downloading full Website offline
-
@Dashrender said in Downloading full Website offline:
@dbeato said in Downloading full Website offline:
@Dashrender said in Downloading full Website offline:
I wonder how much storage is needed?
For example ML took about 24 GB of two days downloading, I stopped it because I didn't need it.
lol, not the site I was talking about
Spiceworks about 87 GB to 100 GB only on posts.
-
@dbeato said in Downloading full Website offline:
@Dashrender said in Downloading full Website offline:
@dbeato said in Downloading full Website offline:
@Dashrender said in Downloading full Website offline:
I wonder how much storage is needed?
For example ML took about 24 GB of two days downloading, I stopped it because I didn't need it.
lol, not the site I was talking about
Spiceworks about 87 GB to 100 GB only on posts.
13.5GB on ML so far.
-
ML has very little "media" on the site. So that doesn't expand very quickly.
-
Is there a way to do a dry run just too see how much storage will be consumed without actually downloading?
-
@black3dynamite said in Downloading full Website offline:
Is there a way to do a dry run just too see how much storage will be consumed without actually downloading?
No that I know of.
-
@black3dynamite said in Downloading full Website offline:
Is there a way to do a dry run just too see how much storage will be consumed without actually downloading?
No, the only way to know the size is to grab every file and add it up. You could come up with a way to store that info and not store the files, but no way to not download it all, add it up, and then know. So not really any value to a dry run, it would hit all the same things as the real deal.
-
-
@black3dynamite said in Downloading full Website offline:
Pretty cool, that's why I say that I don't know of
-
@black3dynamite said in Downloading full Website offline:
That is literally saving the website to a temp directory. Why not just do it once instead of twice?
-
@black3dynamite said in Downloading full Website offline:
But does it really not download everything? Websites don't report on the size directly AFAIK.
-
@travisdh1 said in Downloading full Website offline:
@black3dynamite said in Downloading full Website offline:
That is literally saving the website to a temp directory. Why not just do it once instead of twice?
Right, which is what I had originally said you could do. And doing so to a temp directory is STILL downloading the whole thing - the very thing you are trying to avoid.
-
@dbeato are you specifying a user account for this to run again?
-
@DustinB3403 said in Downloading full Website offline:
@dbeato are you specifying a user account for this to run again?
No, he'd have to pass a cookie to do that.
-
@DustinB3403 said in Downloading full Website offline:
@dbeato are you specifying a user account for this to run again?
I am not.
-
@scottalanmiller okay. . .
So I know exactly why @dbeato is going through this process. The followup is then, how do you sort out the things you don't want from the website and retain those for later?
-
@DustinB3403 said in Downloading full Website offline:
@scottalanmiller okay. . .
So I know exactly why @dbeato is going through this process. The followup is then, how do you sort out the things you don't want from the website and retain those for later?
That is what I would do slowly in my spare time
-
@scottalanmiller said in Downloading full Website offline:
@DustinB3403 said in Downloading full Website offline:
@dbeato are you specifying a user account for this to run again?
No, he'd have to pass a cookie to do that.
-
@dbeato said in Downloading full Website offline:
@DustinB3403 said in Downloading full Website offline:
@scottalanmiller okay. . .
So I know exactly why @dbeato is going through this process. The followup is then, how do you sort out the things you don't want from the website and retain those for later?
That is what I would do slowly in my spare time
Ugh. . .
-
@scottalanmiller said in Downloading full Website offline:
@black3dynamite said in Downloading full Website offline:
But does it really not download everything? Websites don't report on the size directly AFAIK.
Using
--spider, does not download the pages.
Using--no-directories, tells wget to create empty directories. -
@black3dynamite said in Downloading full Website offline:
@scottalanmiller said in Downloading full Website offline:
@black3dynamite said in Downloading full Website offline:
But does it really not download everything? Websites don't report on the size directly AFAIK.
Using
--spider, does not download the pages.
Using--no-directories, tells wget to create empty directories.So it gets it all from "Content Length"? Interesting. Guess that would work.
