Python的消息队列包SnakeMQ使用初探


Posted in Python onJune 29, 2016

一、关于snakemq的官方介绍
SnakeMQ的GitHub项目页:https://github.com/dsiroky/snakemq
1.纯python实现,跨平台

2.自动重连接

3.可靠发送--可配置的消息方式与消息超时方式

4.持久化/临时 两种队列

5.支持异步 -- poll()

6.symmetrical -- 单个TCP连接可用于双工通讯

7.多数据库支持 -- SQLite、MongoDB……

8.brokerless - 类似ZeroMQ的实现原理

9.扩展模块:RPC, bandwidth throttling

以上都是官话,需要自己验证,动手封装了一下,感觉萌萌哒。

二、几个主要问题说明

1.支持自动重连,不需要自己动手写心跳逻辑,你只需要关注发送和接收就行

2.支持数据持久化,如果开始持久化,在重连之后会自动发送数据。

3.数据的接收,snakemq通过提供回调实现,你只需要写个接收方法添加到回调列表里去。

4.数据的发送,在此发送的都是bytes类型(二进制),因此需要转换。我在程序中测试的都是文本字符串,使用str.encode(‘utf-8')转换成bytes,接收时再转换回来。

5.术语解释,Connector:类似于socket的TcpClient,Lisenter:类似于socket的TcpServer,每个connector或者listener都一个一个ident标识,发送和接收数据时就知道是谁的数据了。

6.使用sqlite持久化时,需要修改源码,sqlite3.connect(filename,check_same_thread = False),用于解决多线程访问sqlite的问题。(会不会死锁?)

7.启动持久化时,如果重新连上,则会自动发送,保证可靠。

8.为了封装的需要,数据接收以后,我通过callback方式传送出去。

三、代码

说明代码中使用了自定义的日志模块

from common import nxlogger

import snakemqlogger as logger

可替换成logging的。

回调类(callbacks.py):

# -*- coding:utf-8 -*-

'''synchronized callback'''

class Callback(object):

  def __init__(self):

    self.callbacks = []

 

  def add(self, func):

    self.callbacks.append(func)

 

  def remove(self, func):

    self.callbacks.remove(func)

 

  def __call__(self, *args, **kwargs):

    for callback in self.callbacks:

      callback(*args, **kwargs)

Connector类(snakemqConnector.py):

# -*- coding:utf-8 -*-

import threading

import snakemq

import snakemq.link

import snakemq.packeter

import snakemq.messaging

import snakemq.message

from snakemq.storage.sqlite import SqliteQueuesStorage

from snakemq.message import FLAG_PERSISTENT

from common.callbacks import Callback

 

from common import nxlogger

import snakemqlogger as logger

 

class SnakemqConnector(threading.Thread):

     def __init__(self, snakemqident = None, remoteIp = "localhost", remotePort = 9090, persistent = False):

         super(SnakemqConnector,self).__init__()

         self.messaging = None

         self.link = None

         self.snakemqident = snakemqident

         self.pktr = None

         self.remoteIp = remoteIp

         self.remotePort = remotePort

         self.persistent = persistent

         self.on_recv = Callback()

         self._initConnector()

 

     def run(self):

         logger.info("connector start...")

         

         if self.link != None:

              self.link.loop()

 

         logger.info("connector end...")

    

     def terminate(self):

         logger.info("connetor terminating...")

         if self.link != None:

              self.link.stop()

              self.link.cleanup()

         logger.info("connetor terminated")

 

     def on_recv_message(self, conn, ident, message):

         try:

              self.on_recv(ident, message.data.decode('utf-8'))#dispatch received data

         except Exception as e:

              logger.error("connector recv:{0}".format(e))

              print(e)

 

     '''send message to dest host named destIdent'''

     def sendMsg(self, destIdent, byteseq):

         msg = None

         if self.persistent:

              msg = snakemq.message.Message(byteseq, ttl=60, flags=FLAG_PERSISTENT)

         else:

              msg = snakemq.message.Message(byteseq, ttl=60)

         if self.messaging == None:

              logger.error("connector:messaging is not initialized, send message failed")

              return

         self.messaging.send_message(destIdent, msg)

 

     '''

    

     '''

     def _initConnector(self):

         try:

              self.link = snakemq.link.Link()

              self.link.add_connector((self.remoteIp, self.remotePort))

 

              self.pktr = snakemq.packeter.Packeter(self.link)

 

              if self.persistent:

                  storage = SqliteQueuesStorage("SnakemqStorage.db")

                  self.messaging = snakemq.messaging.Messaging(self.snakemqident, "", self.pktr, storage)

              else:

                  self.messaging = snakemq.messaging.Messaging(self.snakemqident, "", self.pktr)

             

              self.messaging.on_message_recv.add(self.on_recv_message)

             

         except Exception as e:

              logger.error("connector:{0}".format(e))

         finally:

              logger.info("connector[{0}] loop ended...".format(self.snakemqident))

 Listener类(snakemqListener.py):

# -*- coding:utf-8 -*-

import threading

import snakemq

import snakemq.link

import snakemq.packeter

import snakemq.messaging

import snakemq.message

from common import nxlogger

import snakemqlogger as logger

from common.callbacks import Callback

class SnakemqListener(threading.Thread):

     def __init__(self, snakemqident = None, ip = "localhost", port = 9090, persistent = False):

         super(SnakemqListener,self).__init__()

         self.messaging = None

         self.link = None

         self.pktr = None

         self.snakemqident = snakemqident

         self.ip = ip;

         self.port = port

         self.connectors = {}

         self.on_recv = Callback()

         self.persistent = persistent

         self._initlistener()

 

     '''

     thread run

     '''

     def run(self):

         logger.info("listener start...")

         

         if self.link != None:

              self.link.loop()

 

         logger.info("listener end...")

 

     '''

     terminate snakemq listener thread

     '''

     def terminate(self):

         logger.info("listener terminating...")

         if self.link != None:

              self.link.stop()

              self.link.cleanup()

         logger.info("listener terminated")

 

     '''

     receive message from host named ident

     '''

     def on_recv_message(self, conn, ident, message):

         try:

              self.on_recv(ident, message.data.decode('utf-8'))#dispatch received data

              self.sendMsg('bob','hello,{0}'.format(ident).encode('utf-8'))

         except Exception as e:

              logger.error("listener recv:{0}".format(e))

              print(e)

 

     def on_drop_message(self, ident, message):

         print("message dropped", ident, message)

         logger.debug("listener:message dropped,ident:{0},message:{1}".format(ident, message))

 

     '''client connect'''

     def on_connect(self, ident):

         logger.debug("listener:{0} connected".format(ident))

         self.connectors[ident] = ident

         self.sendMsg(ident, "hello".encode('utf-8'))

 

     '''client disconnect'''

     def on_disconnect(self, ident):

         logger.debug("listener:{0} disconnected".format(ident))

         if ident in self.connectors:

              self.connectors.pop(ident)

 

     '''

     listen start loop

     '''

     def _initlistener(self):

         try:

              self.link = snakemq.link.Link()

              self.link.add_listener((self.ip, self.port))

 

              self.pktr = snakemq.packeter.Packeter(self.link)

              self.pktr.on_connect.add(self.on_connect)

              self.pktr.on_disconnect.add(self.on_disconnect)

 

              if self.persistent:

                  storage = SqliteQueuesStorage("SnakemqStorage.db")

                  self.messaging = snakemq.messaging.Messaging(self.snakemqident, "", self.pktr, storage)

              else:

                  self.messaging = snakemq.messaging.Messaging(self.snakemqident, "", self.pktr)

             

              self.messaging.on_message_recv.add(self.on_recv_message)

              self.messaging.on_message_drop.add(self.on_drop_message)

 

         except Exception as e:

              logger.error("listener:{0}".format(e))

         finally:

              logger.info("listener:loop ended...")

     '''send message to dest host named destIdent'''

     def sendMsg(self, destIdent, byteseq):

         msg = None

         if self.persistent:

              msg = snakemq.message.Message(byteseq, ttl=60, flags=FLAG_PERSISTENT)

         else:

              msg = snakemq.message.Message(byteseq, ttl=60)

         if self.messaging == None:

              logger.error("listener:messaging is not initialized, send message failed")

              return

         self.messaging.send_message(destIdent, msg)

测试代码connector(testSnakeConnector.py):

读取本地一个1M的文件,然后发送给listener,然后listener发回一个hello的信息。

from netComm.snakemq import snakemqConnector

import time

import sys

import os

def received(ident, data):

     print(data)

 

if __name__ == "__main__":

     bob = snakemqConnector.SnakemqConnector('bob',"10.16.5.45",4002,True)

     bob.on_recv.add(received)

     bob.start()

     try:

         with open("testfile.txt",encoding='utf-8') as f:

              txt = f.read()

              for i in range(100):

                  bob.sendMsg("niess",txt.encode('utf-8'))

                  time.sleep(0.1)

     except Exception as e:

         print(e)

     time.sleep(5)

     bob.terminate()   

 

测试代码listener(testSnakeListener.py):

from netComm.snakemq import snakemqListener

import time

 

def received(ident, data):

     filename = "log/recFile{0}.txt".format(time.strftime('%S',time.localtime()))

     file = open(filename,'w')

     file.writelines(data)

     file.close()

 

if __name__ == "__main__":

     niess = snakemqListener.SnakemqListener("niess","10.16.5.45",4002)

     niess.on_recv.add(received)

     niess.start()

     print("niess start...")

     time.sleep(60)

     niess.terminate()  

     print("niess end...")
Python 相关文章推荐
python调用windows api锁定计算机示例
Apr 17 Python
windows系统中python使用rar命令压缩多个文件夹示例
May 06 Python
介绍Python中几个常用的类方法
Apr 08 Python
Python获取系统默认字符编码的方法
Jun 04 Python
python:socket传输大文件示例
Jan 18 Python
Django应用程序中如何发送电子邮件详解
Feb 04 Python
Python实现扩展内置类型的方法分析
Oct 16 Python
在python中pandas的series合并方法
Nov 12 Python
python列表生成器迭代器实例解析
Dec 19 Python
selenium+python配置chrome浏览器的选项的实现
Mar 18 Python
tensorflow中tf.reduce_mean函数的使用
Apr 19 Python
pytorch实现查看当前学习率
Jun 24 Python
Python中线程的MQ消息队列实现以及消息队列的优点解析
Jun 29 #Python
深入理解Python中装饰器的用法
Jun 28 #Python
Python中的迭代器与生成器高级用法解析
Jun 28 #Python
Python设计足球联赛赛程表程序的思路与简单实现示例
Jun 28 #Python
详解Python中heapq模块的用法
Jun 28 #Python
Python中operator模块的操作符使用示例总结
Jun 28 #Python
基础的十进制按位运算总结与在Python中的计算示例
Jun 28 #Python
You might like
PHP防CC攻击实现代码
2011/12/29 PHP
用PHP编写和读取XML的几种方式
2013/01/12 PHP
php递归删除指定文件夹的方法小结
2015/04/20 PHP
Jquery数独游戏解析(一)-页面布局
2010/11/05 Javascript
js 编程笔记 无名函数
2011/06/28 Javascript
实现点击列表弹出列表索引的两种方式
2013/03/08 Javascript
jQuery中next()方法用法实例
2015/01/07 Javascript
详解webpack+vue-cli项目打包技巧
2017/06/17 Javascript
JS实现的邮箱提示补全效果示例
2018/01/30 Javascript
express框架中使用jwt实现验证的方法
2019/08/25 Javascript
vue项目实现图片上传功能
2019/12/23 Javascript
在Vue中使用HOC模式的实现
2020/08/23 Javascript
Python配置文件解析模块ConfigParser使用实例
2015/04/13 Python
python实现中文分词FMM算法实例
2015/07/10 Python
Python的Flask框架应用程序实现使用QQ账号登录的方法
2016/06/07 Python
Python3中的列表,元组,字典,字符串相关知识小结
2017/11/10 Python
python爬取网页转换为PDF文件
2018/06/07 Python
详解Python下载图片并保存本地的两种方式
2019/05/15 Python
详解Numpy数组转置的三种方法T、transpose、swapaxes
2019/05/27 Python
分享8点超级有用的Python编程建议(推荐)
2019/10/13 Python
Python3的unicode编码转换成中文的问题及解决方案
2019/12/10 Python
Python测试Kafka集群(pykafka)实例
2019/12/23 Python
python 命名规范知识点汇总
2020/02/14 Python
Python 整行读取文本方法并去掉readlines换行\n操作
2020/09/03 Python
纯CSS3代码实现switch滑动开关按钮效果
2016/08/30 HTML / CSS
美国爆米花工厂:The Popcorn Factory
2019/09/14 全球购物
线程的基本概念、线程的基本状态以及状态之间的关系
2012/10/26 面试题
简述索引存取方法的作用和建立索引的原则
2013/03/26 面试题
人力资源专员岗位职责
2014/01/30 职场文书
2014年社区庆元旦活动方案
2014/03/08 职场文书
合作意向书模板
2014/03/31 职场文书
公司贷款承诺书
2014/05/30 职场文书
基层干部群众路线教育实践活动个人对照检查材料
2014/09/23 职场文书
2015年小学生自我评价范文
2015/03/03 职场文书
圆明园纪录片观后感
2015/06/03 职场文书
Nginx利用Logrotate实现日志分割
2022/05/20 Servers