Migrating Logs
-
As you guys know I'm reading a book on the linux command line before I read the cert book for RHCSA/RHCE. I try to consider use cases for everything and as such, started thinking about
cat. You can combine files usingcat test1.txt test2.txt > test3.txtand the filetest3.txtwill contain bothtest1andtest2.When you are migrating logs, are you using a cron job in conjunction with cat to migrate one big log file or do you just have a naming convention that makes using a wildcard easier?
-
So the quick answer is "no", cat would not be used in this way.
-
The longer answer is that if you wanted to combine your logs into a single file, which I can't think of why you would want to do that, but assuming you wanted a simple single file once a month or something with all logs to go to tape or something I suppose you could come up with a time, you would combine then with tar, not with cat because then you could retrieve them into their original logs rather than having them all merged into one, giant, unwieldy text file.
-
@scottalanmiller said in Migrating Logs:
So the quick answer is "no", cat would not be used in this way.
Am I correct in thinking that the naming convention is used with a wildcard to handle the migration of logs (as an example) that are migrated regularly?
-
Far more often, what you will be doing is not using cat or tar at all, but instead using a tool like gzip or bzip2 to compress the individual log files to a fraction of their original size and leaving them as individual files. This is mostly better because you compress incrementally as you go reducing their size hour by hour, day by day or whatever granularity that you need. Then you can backup or ship them wherever you need whenever you need rather than waiting to bundle them all up before doing something with them. And retrieving them is just grabbing one small file rather than wading through one giant one.
-
Of course you could tar up a bunch of gzipped files, but why bother? You cannot cat gzipped files, though.
-
@wirestyle22 said in Migrating Logs:
@scottalanmiller said in Migrating Logs:
So the quick answer is "no", cat would not be used in this way.
Am I correct in thinking that the naming convention is used with a wildcard to handle the migration of logs (as an example) that are migrated regularly?
You would rarely use anything to grab stuff in that way with logs. A more common thing to do would either to have a script that does something complex, or a really simple command that does something like this...
find all files over 24 hours old in the log directory whose names end in .gz and ship them to backupserver:/logarchive/
-
@scottalanmiller said in Migrating Logs:
retrieving them is just grabbing one small file rather than wading through one giant one.
The assumption here would be if it's getting done regularly it wouldn't be that big but I guess that differs depending on the situation.
-
@scottalanmiller said in Migrating Logs:
@wirestyle22 said in Migrating Logs:
@scottalanmiller said in Migrating Logs:
So the quick answer is "no", cat would not be used in this way.
Am I correct in thinking that the naming convention is used with a wildcard to handle the migration of logs (as an example) that are migrated regularly?
You would rarely use anything to grab stuff in that way with logs. A more common thing to do would either to have a script that does something complex, or a really simple command that does something like this...
find all files over 24 hours old in the log directory whose names end in .gz and ship them to backupserver:/logarchive/
Understood. Thanks!
-
@wirestyle22 said in Migrating Logs:
@scottalanmiller said in Migrating Logs:
retrieving them is just grabbing one small file rather than wading through one giant one.
The assumption here would be if it's getting done regularly it wouldn't be that big but I guess that differs depending on the situation.
If it isn't really big, what was the benefit of combining just a few?
-
@scottalanmiller said in Migrating Logs:
@wirestyle22 said in Migrating Logs:
@scottalanmiller said in Migrating Logs:
retrieving them is just grabbing one small file rather than wading through one giant one.
The assumption here would be if it's getting done regularly it wouldn't be that big but I guess that differs depending on the situation.
If it isn't really big, what was the benefit of combining just a few?
Yeah I guess the positive there is also the negative either way you go
-
@wirestyle22 said in Migrating Logs:
@scottalanmiller said in Migrating Logs:
@wirestyle22 said in Migrating Logs:
@scottalanmiller said in Migrating Logs:
retrieving them is just grabbing one small file rather than wading through one giant one.
The assumption here would be if it's getting done regularly it wouldn't be that big but I guess that differs depending on the situation.
If it isn't really big, what was the benefit of combining just a few?
Yeah I guess the positive there is also the negative either way you go
Yes, merging just a few small files would be extra effort without benefit. If you merged enough to be beneficial in any way, you'd introduce loads of problems.
-
Really, log shipping with local storage is a thing of the past as well. Not what you are looking for with your use case, but long ago people did this. Today if you want to store logs beyond what fits on the local system you look at remote log servers like syslog, rsyslog, Kiwi, Graylog, ELK, loggly, Splunk and so forth. They have more useful platforms for dealing with centralized logs, archiving and backups.
-
@scottalanmiller said in Migrating Logs:
Really, log shipping with local storage is a thing of the past as well. Not what you are looking for with your use case, but long ago people did this. Today if you want to store logs beyond what fits on the local system you look at remote log servers like syslog, rsyslog, Kiwi, Graylog, ELK, loggly, Splunk and so forth. They have more useful platforms for dealing with centralized logs, archiving and backups.
Then we get into why you would use each. What product benefits what situation
