10月 232014
 

日志 前端两台 负载均衡 后端一台 保存和分析
Nginx对已经生成日志进行按天截断

主机A
脚本

#!/bin/bash

logs_path=”/opt/nginx-1.6.2/logs/”
# for bbs.access.log
mv ${logs_path}bbs.access.log ${logs_path}bbs.access.log.$(date -d “yesterday” +”%Y%m%d”).A
# for www.access.log
mv ${logs_path}www.access.log ${logs_path}www.access.log.$(date -d “yesterday” +”%Y%m%d”).A
# for shop.access.log
mv ${logs_path}shop.access.log /mnt/backup/shop.access.log.$(date -d “yesterday” +”%Y%m%d”).A
kill -USR1 `cat /opt/nginx-1.6.2/logs/nginx.pid`

加入定时任务

00 0 * * * root /opt/nginx-1.6.2/logs/logs.sh

主机B
在备份和分析服务主机上对已经截断的前一天日志进行合并

脚本
#!/bin/bash

yesterday=`date -d ‘-1 day’ +%Y%m%d`

# for static.linuxcache.cn
mv /backup/mp/h154.110/static.access.log.$yesterday.A /backup/logs/static.access.log.$yesterday
cat /backup/mp/h154.111/static.access.log.$yesterday.B >>/backup/logs/static.access.log.$yesterday
rm -rf /backup/mp/h154.111/static.access.log.$yesterday.B

加入定时任务 执行日志生成后一段时间

安装EPEL源,安装awstats二进制版本

[root@backup ~]# yum install httpd awstats

Installing:
awstats noarch 7.0-3.el6 epel 1.2 M
Installing for dependencies:
mailcap noarch 2.1.31-2.el6 base 27 k
perl-Compress-Raw-Zlib x86_64 1:2.021-136.el6 base 69 k
perl-Compress-Zlib x86_64 2.021-136.el6 base 45 k
perl-HTML-Parser x86_64 3.64-2.el6 base 109 k
perl-HTML-Tagset noarch 3.20-4.el6 base 17 k
perl-IO-Compress-Base x86_64 2.021-136.el6 base 69 k
perl-IO-Compress-Zlib x86_64 2.021-136.el6 base 135 k
perl-URI noarch 1.40-2.el6 base 117 k
perl-libwww-perl noarch 5.833-2.el6 base 387 k
修改安装后生成的apache配置文件
新增站点配置文件

/usr/share/awstats/tools/awstats_configure.pl
You can then manually update your statistics for ‘bbs.linuxcache.cn’ with command:
> perl awstats.pl -update -config=bbs.linuxcache.cn
You can also build static report pages for ‘bbs.linuxcache.cn’ with command:
> perl awstats.pl -output=pagetype -config=bbs.linuxcache.cn
手动执行日志分析

/usr/share/awstats/wwwroot/cgi-bin/awstats.pl -update -config=bbs.linuxcache.cn

[root@backup awstats]# /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -update -config=bbs.linuxcache.cn
Create/Update database for config “/etc/awstats/awstats.bbs.linuxcache.cn.conf” by AWStats version 7.0 (build 1.971)
From data in log file “/backup/logs/bbs.access.log.20141021″…
Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
Jumped lines in file: 0
Parsed lines in file: 3
Found 0 dropped records,
Found 0 comments,
Found 0 blank records,
Found 0 corrupted records,
Found 0 old records,
Found 3 new qualified records.
[root@backup awstats]#
修改配配置文件

1修改日志路径
2禁用反向解析
3增加地理数据库
4增加UTF8解码器
http://192.168.153.111/awstats/awstats.pl?config=

增加http验证

[root@backup ~]# htpasswd -c /etc/awstats/htpasswd.users logsviewer
New password:
Re-type new password:
Adding password for user logsviewer
New password:
[root@backup ~]#
修改apache配置
<Directory “/usr/share/awstats/wwwroot”>
AuthName “Linuxcache.com LOGS ANALYSIS”
AuthType Basic
AuthUserFile /etc/awstats/htpasswd.users
Require valid-user
Options None
AllowOverride None
Order allow,deny
#Allow from 127.0.0.1
Allow from all
</Directory>

===================================增加地理数据库
[root@backup ~]# rpm -lq GeoIP
/etc/GeoIP.conf
/etc/GeoIP.conf.default
/usr/bin/geoiplookup
/usr/bin/geoiplookup6
/usr/bin/geoipupdate
/usr/lib64/libGeoIP.so.1
/usr/lib64/libGeoIP.so.1.5.1
/usr/lib64/libGeoIPUpdate.so.0
/usr/lib64/libGeoIPUpdate.so.0.0.0
/usr/share/GeoIP
/usr/share/GeoIP/GeoIP.dat
/usr/share/GeoIP/GeoIPASNum.dat
/usr/share/GeoIP/GeoIPASNumv6.dat
/usr/share/GeoIP/GeoIPv6.dat
/usr/share/GeoIP/GeoLiteASNum.dat
/usr/share/GeoIP/GeoLiteASNumv6.dat
/usr/share/GeoIP/GeoLiteCity.dat
/usr/share/GeoIP/GeoLiteCityv6.dat
/usr/share/GeoIP/GeoLiteCountry.dat
/usr/share/doc/GeoIP-1.5.1
/usr/share/doc/GeoIP-1.5.1/AUTHORS
/usr/share/doc/GeoIP-1.5.1/COPYING
/usr/share/doc/GeoIP-1.5.1/ChangeLog
/usr/share/doc/GeoIP-1.5.1/README
/usr/share/doc/GeoIP-1.5.1/TODO
/usr/share/doc/GeoIP-1.5.1/fetch-geoipdata-city.pl
/usr/share/doc/GeoIP-1.5.1/fetch-geoipdata.pl
/usr/share/man/man1/geoiplookup.1.gz
/usr/share/man/man1/geoiplookup6.1.gz
/usr/share/man/man1/geoipupdate.1.gz
[root@backup ~]#

[root@backup ~]# rpm -lq GeoIP-update
/etc/cron.weekly/geoipupdate
[root@backup ~]#

修改配置文件
#LoadPlugin=”geoip GEOIP_STANDARD /pathto/GeoIP.dat”
LoadPlugin=”geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat”
#LoadPlugin=”geoip_city_maxmind GEOIP_STANDARD /pathto/GeoIPCity.dat”
LoadPlugin=”geoip_city_maxmind GEOIP_STANDARD /usr/share/GeoIP/GeoLiteCity.dat”
错误分析

[root@backup awstats]# /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -update -config=static.linuxcache.cn
Error: Plugin load for plugin ‘geoip’ failed with return code: Error:
Can’t locate Geo/IP.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5
/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .
/usr/share/awstats/lib /usr/share/awstats/plugins) at (eval 2) line 1.
Can’t locate Geo/IP/PurePerl.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5
/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .
/usr/share/awstats/lib /usr/share/awstats/plugins) at (eval 3) line 1.
Error: Need Perl module Geo::IP or Geo::IP::PurePerl
Setup (‘/etc/awstats/awstats.static.linuxcache.cn.conf’ file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in ‘docs’ directory).
[root@backup awstats]#

[root@backup awstats]# yum install perl-Geo-IP

安装后执行
[root@backup awstats]# /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -update -config=static.linuxcache.cn
Warning: Tried to load plugin “geoip” twice. Fix config file.
Create/Update database for config “/etc/awstats/awstats.static.linuxcache.cn.conf” by AWStats version 7.0 (build 1.971)
From data in log file “/backup/logs/static.access.log.20141022″…
Phase 1 : First bypass old records, searching new record…
Direct access after last parsed record (after line 6)
Jumped lines in file: 6
Found 6 already parsed records.
Parsed lines in file: 0
Found 0 dropped records,
Found 0 comments,
Found 0 blank records,
Found 0 corrupted records,
Found 0 old records,
Found 0 new qualified records.
[root@backup awstats]#

awstats-logs-geoip

分析搜索关键词乱码问题
#LoadPlugin=”decodeutfkeys”
LoadPlugin=”decodeutfkeys”

awstats-logs-utf8-chinese-01

awstats-logs-utf8-chinese-02