python实时监控logstash日志代码


Posted in Python onApril 27, 2020

实时读取logstash日志,有异常错误keywork即触发报警。

# /usr/bin/env python3
# -*- coding: utf-8 -*-
# __author__ = caozhi
# create_time 2018-11-12,update_time 2018-11-15
# version = 1.0
# 录像高可用报警
# 1 读取日志 使用游标移动
# 2 线上业务日志文件会切割,切割后,读取上一个切割的日志

import os
import sys
import json
import requests
import time
import re

cini = conf.ini'
log_file = logstash.log'

def readconf():
 try:
 with open(cini, 'r+') as f:
  CONF = json.load(f)
 except:
 CONF = {"seek": 0, "inode": 922817, "last_file": logstash.log"}
 writeconf(CONF=CONF)
 print('conf.ini 配置文件缺失,自动创建一个新的配置文件')
 return CONF

def writeconf(CONF):
 with open(cini, 'w+') as e:
 json.dump(CONF, e)

def read_log(log_file, seek):
 try:
 f = open(log_file, 'r')
 except FileNotFoundError:
 f = open(logstash.log', 'r')
 seek = 0
 print('上一个文件读取失败了,请检查切割的日志文件')
 except:
 print('日志文件打开错误,退出程序')
 sys.exit()

f.seek(seek)
line = f.readline()
new_seek = f.tell()
if new_seek == seek:
 print('没有追加日志,退出程序')
 sys.exit()

while line:
 try:
 logstash = json.loads(line)
 except:
 CONF = {"seek": 0, "inode": 922817, "last_file": "/data/logs/lmrs/logstash.log"}
 writeconf(CONF=CONF)
 print('json数据加载错误,重新创建一个新的配置文件')
 sys.exit()

 #if '''re.search(time.strftime("%Y:%H:%M", time.localtime()), logstash.get('log_time')) and '''logstash.get('rtype') == 6 and logstash.get('uri') == '/publish' and logstash.get('event') == 0:
 if logstash.get('rtype') == 6 and logstash.get('uri') == '/publish' and logstash.get('event') == 0:
 value = 1
 stream = logstash.get('name')
 print('{} {}'.format(value, stream))
 record(value=value, stream=stream)
 else:
 value = 0
 stream = 0
 line = f.readline()
seek = f.tell()
f.close
return value, stream, seek

def record(value, stream):
 data = []
 record = {}
 record['metric'] = 'recording_high_availability_monitor'
 record['endpoint'] = os.uname()[1]
 record['timestamp'] = int(time.time())
 record['step'] = 60
 record['value'] = value
 record['counterType'] = 'GAUGE'
 record['Tags'] = '{}={}'.format(int(time.time()), stream)
 data.append(record)

if data:
 print('这是data的json数据')
 print(data)
 falcon_request = requests.post("http://127.0.0.1:1988/v1/push", data=json.dumps(data))
 #falcon_request = requests.post("http://127.0.0.1:1988/v1/push", json=data)
 print('json参数请求返回状态码为:' + str(falcon_request.status_code))
 print('json参数请求返回为:' + str(falcon_request.text))

if __name__ == '__main__':
 print()
 print('***************************************')
 print('本次执行脚本时间:{}'.format(time.strftime("%Y%m%d_%H%M", time.localtime())))
 CONF = readconf()
 print('first_CONF :{}'.format(CONF))
 print('NO1.log_file',log_file)
 last_inode = CONF['inode']
 inode = os.stat(log_file).st_ino
 print('last_inode: {} inode: {}'.format(last_inode, inode))

if inode == last_inode:
 seek = CONF['seek']
 next_file = 0
else:
 log_file = CONF['last_file'] + time.strftime("-%Y%m%d_", time.localtime()) + str(time.strftime("%H%M", time.localtime()))[:-1] + '0'
 next_file = 1
 seek = CONF['seek']

print('NO2.log_file',log_file)
value, stream, seek = read_log(log_file=log_file,seek=seek)

if next_file:
 CONF['seek'] = 0
else:
 CONF['seek'] = seek

CONF['inode'] = os.stat(logstash.log').st_ino
writeconf(CONF=CONF)
print('last_CONF :{}'.format(CONF))

补充知识:logstash 调用exec

我就废话不多说了,还是直接看代码吧!

[elk@Vsftp logstash]$ cat t3.conf 
input {
 stdin {
 } 
} 
filter {
 grok {
 match => [ "message","(?m)\s*%{TIMESTAMP_ISO8601:time}\s*(?<Level>(\S+)).*"]
 }
 date {
 match => ["time", "yyyy-MM-dd HH:mm:ss,SSS"]
 }
 mutate {
   add_field =>["type","tailong"]
   add_field =>["messager","%{type}-%{message}"]
   remove_field =>["message"]
  }
} 
output { 
 if ([Level] == "ERROR" or [messager] =~ "Exception" ) and [messager] !~ "温金服务未连接" and [messager] !~ "调用温金代理系统接口错误" and [messager] !~ "BusinessException" {
 exec {
  command => "/bin/smail.pl \"%{messager}\" \"%{type}\" "
 }
 }
 stdout { 
 codec =>rubydebug 
 } 
}
 
Vsftp:/root# cat /bin/smail.pl 
#!/usr/bin/perl 
use Net::SMTP;
use HTTP::Date qw(time2iso str2time time2iso time2isoz); 
use Data::Dumper;
use Getopt::Std;
use vars qw($opt_d );
getopts('d:');
# mail_user should be your_mail@163.com
 $message= "@ARGV";
 $env="$opt_d";
 sub send_mail{
 my $CurrTime = time2iso(time());
 my $to_address = shift;
 my $mail_user = 'zhao.yangjian@163.com';
 my $mail_pwd = 'xx';
 my $mail_server = 'smtp.163.com';
 
 my $from = "From: $mail_user\n";
 my $subject = "Subject: zjcap info\n";
 my $info = "$CurrTime--$message";
 my $message = <<CONTENT; 
 $info
CONTENT
 my $smtp = Net::SMTP->new($mail_server);
 
 $smtp->auth($mail_user, $mail_pwd) || die "Auth Error! $!";
 $smtp->mail($mail_user);
 $smtp->to($to_address);
 
 $smtp->data();  # begin the data
 $smtp->datasend($from); # set user
 $smtp->datasend($subject); # set subject
 $smtp->datasend("\n\n");
 $smtp->datasend("$message\n"); # set content
 $smtp->dataend();
 $smtp->quit();
};
 
send_mail ('zhao.yangjian@163.com'); 
 
2017-01-12 10:19:19,888 jjjjj Exception
{
 "@version" => "1",
 "@timestamp" => "2017-01-12T02:19:19.888Z",
  "host" => "Vsftp",
  "time" => "2017-01-12 10:19:19,888",
  "Level" => "jjjjj",
  "type" => "tailong",
 "messager" => "tailong-2017-01-12 10:19:19,888 jjjjj Exception"
}

以上这篇python实时监控logstash日志代码就是小编分享给大家的全部内容了,希望能给大家一个参考,也希望大家多多支持三水点靠木。

Python 相关文章推荐
Python写的服务监控程序实例
Jan 31 Python
Python连接PostgreSQL数据库的方法
Nov 28 Python
使用 Python 实现简单的 switch/case 语句的方法
Sep 17 Python
使用50行Python代码从零开始实现一个AI平衡小游戏
Nov 21 Python
python使用pipeline批量读写redis的方法
Feb 18 Python
pytorch中torch.max和Tensor.view函数用法详解
Jan 03 Python
tensorflow2.0与tensorflow1.0的性能区别介绍
Feb 07 Python
python+opencv3生成一个自定义纯色图教程
Feb 19 Python
解决Django中checkbox复选框的传值问题
Mar 31 Python
Django表单提交后实现获取相同name的不同value值
May 14 Python
基于python爬取链家二手房信息代码示例
Oct 21 Python
Python OpenCV超详细讲解调整大小与图像操作的实现
Apr 02 Python
python实现秒杀商品的微信自动提醒功能(代码详解)
Apr 27 #Python
浅析python 定时拆分备份 nginx 日志的方法
Apr 27 #Python
python异步Web框架sanic的实现
Apr 27 #Python
python库skimage给灰度图像染色的方法示例
Apr 27 #Python
python实现密度聚类(模板代码+sklearn代码)
Apr 27 #Python
Django中文件上传和文件访问微项目的方法
Apr 27 #Python
详解Python中namedtuple的使用
Apr 27 #Python
You might like
php利用iframe实现无刷新文件上传功能的代码
2011/09/29 PHP
遍历指定目录,并存储目录内所有文件属性信息的php代码
2016/10/28 PHP
javascript 延迟加载技术(lazyload)简单实现
2011/01/17 Javascript
十个迅速提升JQuery性能让你的JQuery跑得更快
2012/12/10 Javascript
jquery自定义函数的多种方法
2014/01/09 Javascript
Javascript中数组方法汇总(推荐)
2015/04/01 Javascript
浅谈angular.js中实现双向绑定的方法$watch $digest $apply
2015/10/14 Javascript
基于javascript实现简单的抽奖系统
2020/04/15 Javascript
个人网站留言页面(前端jQuery编写、后台php读写MySQL)
2016/05/03 Javascript
Vue实例简单方法介绍
2017/01/20 Javascript
Vue组件实例间的直接访问实现代码
2017/08/20 Javascript
详解ES6通过WeakMap解决内存泄漏问题
2018/03/09 Javascript
利用vscode调试编译后的js代码详解
2018/05/14 Javascript
Express之托管静态文件的方法
2018/06/01 Javascript
微信小程序实现tab页面切换功能
2018/07/13 Javascript
PostgreSQL Node.js实现函数计算方法示例
2019/02/12 Javascript
详解滑动穿透(锁body)终极探索
2019/04/16 Javascript
vue 验证两次输入的密码是否一致的方法示例
2020/09/29 Javascript
[01:00:04]DOTA2上海特级锦标赛B组小组赛#1 Alliance VS Spirit第二局
2016/02/26 DOTA
[00:55]深扒TI7聊天轮盘语音出处3
2017/05/11 DOTA
Django 使用Ajax进行前后台交互的示例讲解
2018/05/28 Python
selenium+python 对输入框的输入处理方法
2018/10/11 Python
python搜索包的路径的实现方法
2019/07/19 Python
Django admin禁用编辑链接和添加删除操作详解
2019/11/15 Python
tensorflow之获取tensor的shape作为max_pool的ksize实例
2020/01/04 Python
Python unittest生成测试报告过程解析
2020/09/08 Python
python中time tzset()函数实例用法
2021/02/18 Python
HTML5 canvas 瀑布流文字效果的示例代码
2018/01/31 HTML / CSS
Maje德国官网:法国女性成衣品牌
2017/02/10 全球购物
Python面试题集
2012/03/08 面试题
2014年学雷锋活动总结
2014/06/26 职场文书
2014预备党员批评与自我批评思想汇报
2014/09/20 职场文书
四风之害观后感
2015/06/09 职场文书
工作感想范文
2015/08/07 职场文书
创业计划书之婴幼儿游泳馆
2019/09/11 职场文书
2019年国庆祝福语(70句)
2019/09/19 职场文书