python实时监控logstash日志代码


Posted in Python onApril 27, 2020

实时读取logstash日志,有异常错误keywork即触发报警。

# /usr/bin/env python3
# -*- coding: utf-8 -*-
# __author__ = caozhi
# create_time 2018-11-12,update_time 2018-11-15
# version = 1.0
# 录像高可用报警
# 1 读取日志 使用游标移动
# 2 线上业务日志文件会切割,切割后,读取上一个切割的日志

import os
import sys
import json
import requests
import time
import re

cini = conf.ini'
log_file = logstash.log'

def readconf():
 try:
 with open(cini, 'r+') as f:
  CONF = json.load(f)
 except:
 CONF = {"seek": 0, "inode": 922817, "last_file": logstash.log"}
 writeconf(CONF=CONF)
 print('conf.ini 配置文件缺失,自动创建一个新的配置文件')
 return CONF

def writeconf(CONF):
 with open(cini, 'w+') as e:
 json.dump(CONF, e)

def read_log(log_file, seek):
 try:
 f = open(log_file, 'r')
 except FileNotFoundError:
 f = open(logstash.log', 'r')
 seek = 0
 print('上一个文件读取失败了,请检查切割的日志文件')
 except:
 print('日志文件打开错误,退出程序')
 sys.exit()

f.seek(seek)
line = f.readline()
new_seek = f.tell()
if new_seek == seek:
 print('没有追加日志,退出程序')
 sys.exit()

while line:
 try:
 logstash = json.loads(line)
 except:
 CONF = {"seek": 0, "inode": 922817, "last_file": "/data/logs/lmrs/logstash.log"}
 writeconf(CONF=CONF)
 print('json数据加载错误,重新创建一个新的配置文件')
 sys.exit()

 #if '''re.search(time.strftime("%Y:%H:%M", time.localtime()), logstash.get('log_time')) and '''logstash.get('rtype') == 6 and logstash.get('uri') == '/publish' and logstash.get('event') == 0:
 if logstash.get('rtype') == 6 and logstash.get('uri') == '/publish' and logstash.get('event') == 0:
 value = 1
 stream = logstash.get('name')
 print('{} {}'.format(value, stream))
 record(value=value, stream=stream)
 else:
 value = 0
 stream = 0
 line = f.readline()
seek = f.tell()
f.close
return value, stream, seek

def record(value, stream):
 data = []
 record = {}
 record['metric'] = 'recording_high_availability_monitor'
 record['endpoint'] = os.uname()[1]
 record['timestamp'] = int(time.time())
 record['step'] = 60
 record['value'] = value
 record['counterType'] = 'GAUGE'
 record['Tags'] = '{}={}'.format(int(time.time()), stream)
 data.append(record)

if data:
 print('这是data的json数据')
 print(data)
 falcon_request = requests.post("http://127.0.0.1:1988/v1/push", data=json.dumps(data))
 #falcon_request = requests.post("http://127.0.0.1:1988/v1/push", json=data)
 print('json参数请求返回状态码为:' + str(falcon_request.status_code))
 print('json参数请求返回为:' + str(falcon_request.text))

if __name__ == '__main__':
 print()
 print('***************************************')
 print('本次执行脚本时间:{}'.format(time.strftime("%Y%m%d_%H%M", time.localtime())))
 CONF = readconf()
 print('first_CONF :{}'.format(CONF))
 print('NO1.log_file',log_file)
 last_inode = CONF['inode']
 inode = os.stat(log_file).st_ino
 print('last_inode: {} inode: {}'.format(last_inode, inode))

if inode == last_inode:
 seek = CONF['seek']
 next_file = 0
else:
 log_file = CONF['last_file'] + time.strftime("-%Y%m%d_", time.localtime()) + str(time.strftime("%H%M", time.localtime()))[:-1] + '0'
 next_file = 1
 seek = CONF['seek']

print('NO2.log_file',log_file)
value, stream, seek = read_log(log_file=log_file,seek=seek)

if next_file:
 CONF['seek'] = 0
else:
 CONF['seek'] = seek

CONF['inode'] = os.stat(logstash.log').st_ino
writeconf(CONF=CONF)
print('last_CONF :{}'.format(CONF))

补充知识:logstash 调用exec

我就废话不多说了,还是直接看代码吧!

[elk@Vsftp logstash]$ cat t3.conf 
input {
 stdin {
 } 
} 
filter {
 grok {
 match => [ "message","(?m)\s*%{TIMESTAMP_ISO8601:time}\s*(?<Level>(\S+)).*"]
 }
 date {
 match => ["time", "yyyy-MM-dd HH:mm:ss,SSS"]
 }
 mutate {
   add_field =>["type","tailong"]
   add_field =>["messager","%{type}-%{message}"]
   remove_field =>["message"]
  }
} 
output { 
 if ([Level] == "ERROR" or [messager] =~ "Exception" ) and [messager] !~ "温金服务未连接" and [messager] !~ "调用温金代理系统接口错误" and [messager] !~ "BusinessException" {
 exec {
  command => "/bin/smail.pl \"%{messager}\" \"%{type}\" "
 }
 }
 stdout { 
 codec =>rubydebug 
 } 
}
 
Vsftp:/root# cat /bin/smail.pl 
#!/usr/bin/perl 
use Net::SMTP;
use HTTP::Date qw(time2iso str2time time2iso time2isoz); 
use Data::Dumper;
use Getopt::Std;
use vars qw($opt_d );
getopts('d:');
# mail_user should be your_mail@163.com
 $message= "@ARGV";
 $env="$opt_d";
 sub send_mail{
 my $CurrTime = time2iso(time());
 my $to_address = shift;
 my $mail_user = 'zhao.yangjian@163.com';
 my $mail_pwd = 'xx';
 my $mail_server = 'smtp.163.com';
 
 my $from = "From: $mail_user\n";
 my $subject = "Subject: zjcap info\n";
 my $info = "$CurrTime--$message";
 my $message = <<CONTENT; 
 $info
CONTENT
 my $smtp = Net::SMTP->new($mail_server);
 
 $smtp->auth($mail_user, $mail_pwd) || die "Auth Error! $!";
 $smtp->mail($mail_user);
 $smtp->to($to_address);
 
 $smtp->data();  # begin the data
 $smtp->datasend($from); # set user
 $smtp->datasend($subject); # set subject
 $smtp->datasend("\n\n");
 $smtp->datasend("$message\n"); # set content
 $smtp->dataend();
 $smtp->quit();
};
 
send_mail ('zhao.yangjian@163.com'); 
 
2017-01-12 10:19:19,888 jjjjj Exception
{
 "@version" => "1",
 "@timestamp" => "2017-01-12T02:19:19.888Z",
  "host" => "Vsftp",
  "time" => "2017-01-12 10:19:19,888",
  "Level" => "jjjjj",
  "type" => "tailong",
 "messager" => "tailong-2017-01-12 10:19:19,888 jjjjj Exception"
}

以上这篇python实时监控logstash日志代码就是小编分享给大家的全部内容了,希望能给大家一个参考,也希望大家多多支持三水点靠木。

Python 相关文章推荐
Python strip lstrip rstrip使用方法
Sep 06 Python
python使用urllib模块开发的多线程豆瓣小站mp3下载器
Jan 16 Python
python操作CouchDB的方法
Oct 08 Python
Python中decorator使用实例
Apr 14 Python
在Python中使用第三方模块的教程
Apr 27 Python
PyCharm安装第三方库如Requests的图文教程
May 18 Python
python获取磁盘号下盘符步骤详解
Jun 19 Python
Python OrderedDict字典排序方法详解
May 21 Python
Django Form设置文本框为readonly操作
Jul 03 Python
python 根据列表批量下载网易云音乐的免费音乐
Dec 03 Python
python向xls写入数据(包括合并,边框,对齐,列宽)
Feb 02 Python
Django与AJAX实现网页动态数据显示的示例代码
Feb 24 Python
python实现秒杀商品的微信自动提醒功能(代码详解)
Apr 27 #Python
浅析python 定时拆分备份 nginx 日志的方法
Apr 27 #Python
python异步Web框架sanic的实现
Apr 27 #Python
python库skimage给灰度图像染色的方法示例
Apr 27 #Python
python实现密度聚类(模板代码+sklearn代码)
Apr 27 #Python
Django中文件上传和文件访问微项目的方法
Apr 27 #Python
详解Python中namedtuple的使用
Apr 27 #Python
You might like
ip签名探针
2006/10/09 PHP
PHP 第二节 数据类型之转换
2012/04/28 PHP
PHP实现搜索地理位置及计算两点地理位置间距离的实例
2016/01/08 PHP
php版微信数据统计接口用法示例
2016/10/12 PHP
PHP实现支付宝即时到账功能
2016/12/21 PHP
js登录弹出层特效
2014/03/07 Javascript
jQuery实现form表单元素序列化为json对象的方法
2015/12/09 Javascript
jQuery中Ajax全局事件引用方式及各个事件(全局/局部)执行顺序
2016/06/02 Javascript
vue.js表格组件开发的实例详解
2016/10/12 Javascript
Bootstrap选项卡动态切换效果
2016/11/28 Javascript
详谈jQuery中的一些正则匹配表达式
2017/03/08 Javascript
纯JS实现图片验证码功能并兼容IE6-8(推荐)
2017/04/19 Javascript
原生js实现日历效果
2020/03/02 Javascript
python解决汉字编码问题:Unicode Decode Error
2017/01/19 Python
python3.x实现发送邮件功能
2018/05/22 Python
python 提取tuple类型值中json格式的key值方法
2018/12/31 Python
Python提取频域特征知识点浅析
2019/03/04 Python
python nmap实现端口扫描器教程
2020/05/28 Python
python网络编程之多线程同时接受和发送
2019/09/03 Python
python 哈希表实现简单python字典代码实例
2019/09/27 Python
python模块导入的方法
2019/10/24 Python
Tensorflow 1.0之后模型文件、权重数值的读取方式
2020/02/12 Python
python模拟实现分发扑克牌
2020/04/22 Python
利用python控制Autocad:pyautocad方式
2020/06/01 Python
如何用用Python将地址标记在地图上
2021/02/07 Python
电气专业应届生求职信
2013/11/01 职场文书
白酒市场营销方案
2014/02/25 职场文书
商学院大学生求职的自我评价
2014/03/12 职场文书
《大禹治水》教学反思
2014/04/27 职场文书
优秀研究生主要事迹
2014/06/03 职场文书
学习作风建设心得体会
2014/10/22 职场文书
场地使用证明模板
2014/10/25 职场文书
保安辞职申请书应该怎么写?
2019/07/15 职场文书
熟背这些句子,让您的英语口语突飞猛进(135句)
2019/09/06 职场文书
彻底理解golang中什么是nil
2021/04/29 Golang
MySQL数据库如何查看表占用空间大小
2022/06/10 MySQL