Python的消息队列包SnakeMQ使用初探


Posted in Python onJune 29, 2016

一、关于snakemq的官方介绍
SnakeMQ的GitHub项目页:https://github.com/dsiroky/snakemq
1.纯python实现,跨平台

2.自动重连接

3.可靠发送--可配置的消息方式与消息超时方式

4.持久化/临时 两种队列

5.支持异步 -- poll()

6.symmetrical -- 单个TCP连接可用于双工通讯

7.多数据库支持 -- SQLite、MongoDB……

8.brokerless - 类似ZeroMQ的实现原理

9.扩展模块:RPC, bandwidth throttling

以上都是官话,需要自己验证,动手封装了一下,感觉萌萌哒。

二、几个主要问题说明

1.支持自动重连,不需要自己动手写心跳逻辑,你只需要关注发送和接收就行

2.支持数据持久化,如果开始持久化,在重连之后会自动发送数据。

3.数据的接收,snakemq通过提供回调实现,你只需要写个接收方法添加到回调列表里去。

4.数据的发送,在此发送的都是bytes类型(二进制),因此需要转换。我在程序中测试的都是文本字符串,使用str.encode(‘utf-8')转换成bytes,接收时再转换回来。

5.术语解释,Connector:类似于socket的TcpClient,Lisenter:类似于socket的TcpServer,每个connector或者listener都一个一个ident标识,发送和接收数据时就知道是谁的数据了。

6.使用sqlite持久化时,需要修改源码,sqlite3.connect(filename,check_same_thread = False),用于解决多线程访问sqlite的问题。(会不会死锁?)

7.启动持久化时,如果重新连上,则会自动发送,保证可靠。

8.为了封装的需要,数据接收以后,我通过callback方式传送出去。

三、代码

说明代码中使用了自定义的日志模块

from common import nxlogger

import snakemqlogger as logger

可替换成logging的。

回调类(callbacks.py):

# -*- coding:utf-8 -*-

'''synchronized callback'''

class Callback(object):

  def __init__(self):

    self.callbacks = []

 

  def add(self, func):

    self.callbacks.append(func)

 

  def remove(self, func):

    self.callbacks.remove(func)

 

  def __call__(self, *args, **kwargs):

    for callback in self.callbacks:

      callback(*args, **kwargs)

Connector类(snakemqConnector.py):

# -*- coding:utf-8 -*-

import threading

import snakemq

import snakemq.link

import snakemq.packeter

import snakemq.messaging

import snakemq.message

from snakemq.storage.sqlite import SqliteQueuesStorage

from snakemq.message import FLAG_PERSISTENT

from common.callbacks import Callback

 

from common import nxlogger

import snakemqlogger as logger

 

class SnakemqConnector(threading.Thread):

     def __init__(self, snakemqident = None, remoteIp = "localhost", remotePort = 9090, persistent = False):

         super(SnakemqConnector,self).__init__()

         self.messaging = None

         self.link = None

         self.snakemqident = snakemqident

         self.pktr = None

         self.remoteIp = remoteIp

         self.remotePort = remotePort

         self.persistent = persistent

         self.on_recv = Callback()

         self._initConnector()

 

     def run(self):

         logger.info("connector start...")

         

         if self.link != None:

              self.link.loop()

 

         logger.info("connector end...")

    

     def terminate(self):

         logger.info("connetor terminating...")

         if self.link != None:

              self.link.stop()

              self.link.cleanup()

         logger.info("connetor terminated")

 

     def on_recv_message(self, conn, ident, message):

         try:

              self.on_recv(ident, message.data.decode('utf-8'))#dispatch received data

         except Exception as e:

              logger.error("connector recv:{0}".format(e))

              print(e)

 

     '''send message to dest host named destIdent'''

     def sendMsg(self, destIdent, byteseq):

         msg = None

         if self.persistent:

              msg = snakemq.message.Message(byteseq, ttl=60, flags=FLAG_PERSISTENT)

         else:

              msg = snakemq.message.Message(byteseq, ttl=60)

         if self.messaging == None:

              logger.error("connector:messaging is not initialized, send message failed")

              return

         self.messaging.send_message(destIdent, msg)

 

     '''

    

     '''

     def _initConnector(self):

         try:

              self.link = snakemq.link.Link()

              self.link.add_connector((self.remoteIp, self.remotePort))

 

              self.pktr = snakemq.packeter.Packeter(self.link)

 

              if self.persistent:

                  storage = SqliteQueuesStorage("SnakemqStorage.db")

                  self.messaging = snakemq.messaging.Messaging(self.snakemqident, "", self.pktr, storage)

              else:

                  self.messaging = snakemq.messaging.Messaging(self.snakemqident, "", self.pktr)

             

              self.messaging.on_message_recv.add(self.on_recv_message)

             

         except Exception as e:

              logger.error("connector:{0}".format(e))

         finally:

              logger.info("connector[{0}] loop ended...".format(self.snakemqident))

 Listener类(snakemqListener.py):

# -*- coding:utf-8 -*-

import threading

import snakemq

import snakemq.link

import snakemq.packeter

import snakemq.messaging

import snakemq.message

from common import nxlogger

import snakemqlogger as logger

from common.callbacks import Callback

class SnakemqListener(threading.Thread):

     def __init__(self, snakemqident = None, ip = "localhost", port = 9090, persistent = False):

         super(SnakemqListener,self).__init__()

         self.messaging = None

         self.link = None

         self.pktr = None

         self.snakemqident = snakemqident

         self.ip = ip;

         self.port = port

         self.connectors = {}

         self.on_recv = Callback()

         self.persistent = persistent

         self._initlistener()

 

     '''

     thread run

     '''

     def run(self):

         logger.info("listener start...")

         

         if self.link != None:

              self.link.loop()

 

         logger.info("listener end...")

 

     '''

     terminate snakemq listener thread

     '''

     def terminate(self):

         logger.info("listener terminating...")

         if self.link != None:

              self.link.stop()

              self.link.cleanup()

         logger.info("listener terminated")

 

     '''

     receive message from host named ident

     '''

     def on_recv_message(self, conn, ident, message):

         try:

              self.on_recv(ident, message.data.decode('utf-8'))#dispatch received data

              self.sendMsg('bob','hello,{0}'.format(ident).encode('utf-8'))

         except Exception as e:

              logger.error("listener recv:{0}".format(e))

              print(e)

 

     def on_drop_message(self, ident, message):

         print("message dropped", ident, message)

         logger.debug("listener:message dropped,ident:{0},message:{1}".format(ident, message))

 

     '''client connect'''

     def on_connect(self, ident):

         logger.debug("listener:{0} connected".format(ident))

         self.connectors[ident] = ident

         self.sendMsg(ident, "hello".encode('utf-8'))

 

     '''client disconnect'''

     def on_disconnect(self, ident):

         logger.debug("listener:{0} disconnected".format(ident))

         if ident in self.connectors:

              self.connectors.pop(ident)

 

     '''

     listen start loop

     '''

     def _initlistener(self):

         try:

              self.link = snakemq.link.Link()

              self.link.add_listener((self.ip, self.port))

 

              self.pktr = snakemq.packeter.Packeter(self.link)

              self.pktr.on_connect.add(self.on_connect)

              self.pktr.on_disconnect.add(self.on_disconnect)

 

              if self.persistent:

                  storage = SqliteQueuesStorage("SnakemqStorage.db")

                  self.messaging = snakemq.messaging.Messaging(self.snakemqident, "", self.pktr, storage)

              else:

                  self.messaging = snakemq.messaging.Messaging(self.snakemqident, "", self.pktr)

             

              self.messaging.on_message_recv.add(self.on_recv_message)

              self.messaging.on_message_drop.add(self.on_drop_message)

 

         except Exception as e:

              logger.error("listener:{0}".format(e))

         finally:

              logger.info("listener:loop ended...")

     '''send message to dest host named destIdent'''

     def sendMsg(self, destIdent, byteseq):

         msg = None

         if self.persistent:

              msg = snakemq.message.Message(byteseq, ttl=60, flags=FLAG_PERSISTENT)

         else:

              msg = snakemq.message.Message(byteseq, ttl=60)

         if self.messaging == None:

              logger.error("listener:messaging is not initialized, send message failed")

              return

         self.messaging.send_message(destIdent, msg)

测试代码connector(testSnakeConnector.py):

读取本地一个1M的文件,然后发送给listener,然后listener发回一个hello的信息。

from netComm.snakemq import snakemqConnector

import time

import sys

import os

def received(ident, data):

     print(data)

 

if __name__ == "__main__":

     bob = snakemqConnector.SnakemqConnector('bob',"10.16.5.45",4002,True)

     bob.on_recv.add(received)

     bob.start()

     try:

         with open("testfile.txt",encoding='utf-8') as f:

              txt = f.read()

              for i in range(100):

                  bob.sendMsg("niess",txt.encode('utf-8'))

                  time.sleep(0.1)

     except Exception as e:

         print(e)

     time.sleep(5)

     bob.terminate()   

 

测试代码listener(testSnakeListener.py):

from netComm.snakemq import snakemqListener

import time

 

def received(ident, data):

     filename = "log/recFile{0}.txt".format(time.strftime('%S',time.localtime()))

     file = open(filename,'w')

     file.writelines(data)

     file.close()

 

if __name__ == "__main__":

     niess = snakemqListener.SnakemqListener("niess","10.16.5.45",4002)

     niess.on_recv.add(received)

     niess.start()

     print("niess start...")

     time.sleep(60)

     niess.terminate()  

     print("niess end...")
Python 相关文章推荐
python 获取本机ip地址的两个方法
Feb 25 Python
python基于windows平台锁定键盘输入的方法
Mar 05 Python
Python中利用函数装饰器实现备忘功能
Mar 30 Python
python DataFrame 修改列的顺序实例
Apr 10 Python
Python3 安装PyQt5及exe打包图文教程
Jan 08 Python
python获取引用对象的个数方式
Dec 20 Python
使用Python实现Wake On Lan远程开机功能
Jan 22 Python
python网络编程socket实现服务端、客户端操作详解
Mar 24 Python
tensorflow 20:搭网络,导出模型,运行模型的实例
May 26 Python
用python监控服务器的cpu,磁盘空间,内存,超过邮件报警
Jan 29 Python
Python图片验证码降噪和8邻域降噪
Aug 30 Python
利用Python实时获取steam特惠游戏数据
Jun 25 Python
Python中线程的MQ消息队列实现以及消息队列的优点解析
Jun 29 #Python
深入理解Python中装饰器的用法
Jun 28 #Python
Python中的迭代器与生成器高级用法解析
Jun 28 #Python
Python设计足球联赛赛程表程序的思路与简单实现示例
Jun 28 #Python
详解Python中heapq模块的用法
Jun 28 #Python
Python中operator模块的操作符使用示例总结
Jun 28 #Python
基础的十进制按位运算总结与在Python中的计算示例
Jun 28 #Python
You might like
使用MaxMind 根据IP地址对访问者定位
2006/10/09 PHP
Smarty Foreach 使用说明
2010/03/23 PHP
php数组函数序列之array_unique() - 去除数组中重复的元素值
2011/10/29 PHP
PHP小技巧之函数重载
2014/06/02 PHP
PHP基于DOM创建xml文档的方法示例
2017/02/08 PHP
thinkPHP5使用Rabc实现权限管理
2019/08/28 PHP
PHP实现获取文件mime类型多种方法解析
2020/05/28 PHP
jquery+json实现的搜索加分页效果
2010/03/31 Javascript
详解js闭包
2014/09/02 Javascript
javaScript中push函数用法实例分析
2015/06/08 Javascript
Bootstrap安装环境配置教程分享
2016/05/27 Javascript
Web程序员必备的7个JavaScript函数
2016/06/14 Javascript
ajax的分页查询示例(不刷新页面)
2017/01/11 Javascript
分析JavaScript数组操作难点
2017/12/18 Javascript
node中间层实现文件上传功能
2018/06/11 Javascript
快速解决vue动态绑定多个class的官方实例语法无效的问题
2018/09/05 Javascript
微信小程序Flex布局用法深入浅出分析
2019/04/25 Javascript
Vue 图片压缩并上传至服务器功能
2020/01/15 Javascript
ElementUI中el-tree节点的操作的实现
2020/02/27 Javascript
vue3.0实现点击切换验证码(组件)及校验
2020/11/18 Vue.js
python连接字符串的方法小结
2015/07/13 Python
Python基于pygame实现的弹力球效果(附源码)
2015/11/11 Python
简单了解python单例模式的几种写法
2019/07/01 Python
Python读取分割压缩TXT文本文件实例
2020/02/14 Python
Python中的wordcloud库安装问题及解决方法
2020/05/27 Python
python实现文件+参数发送request的实例代码
2021/01/05 Python
利用css3制作3D样式按钮实现代码
2013/03/18 HTML / CSS
加拿大最大的相机店:Henry’s
2017/05/17 全球购物
类如何去实现接口
2013/12/19 面试题
小学生自我评价范例
2013/09/24 职场文书
旅游管理专业生自荐信范文
2014/01/02 职场文书
工作表扬信的范文
2014/01/10 职场文书
村党支部书记四风问题个人对照检查材料思想汇报
2014/10/06 职场文书
父亲节寄语大全
2015/02/27 职场文书
2016年暑期社会实践活动总结报告
2016/04/06 职场文书
Python+uiautomator2实现自动刷抖音视频功能
2021/04/29 Python