python制作mysql数据迁移脚本


Posted in Python onJanuary 01, 2019

用python写了个数据迁移脚本,主要是利用从库将大的静态表导出表空间,载导入到目标实例中。

#!/usr/bin/env python3
#-*- coding:utf8 -*-
#author:zhanbin.liu
#!!!!!DB必须同版本
#python3环境  pip3 install pymysql paramiko

import os
#from pathlib import Path
import sys
import pymysql
import paramiko

#每次只能迁移一个DB下的表,到指定DB
#GRANT SELECT, CREATE, RELOAD, ALTER, LOCK TABLES ON *.* TO 'data_migration'@'192.168.%' IDENTIFIED BY 'data_migration@123';
tables='sqlauto_cluster,sqlauto_user'    #以,分割的字符串,如a,b,c
tableList = tables.split(',')
sourceIp = '192.168.1.101'
sourceDataBase = '/data/mysql/3306/data'
sourceDbName = 'inception_web'
sourceDataDir = os.path.join(sourceDataBase,sourceDbName)
desIp = '192.168.1.102'
desDataBase = '/data/mysql/3306/data'
desDbName = 'inception_web'
desDataDir = os.path.join(desDataBase,desDbName)

# for table in tableList:
#   desFile = Path("%s/%s.ibd" %(desDataDir,table))
#   print(desFile)
#   if desFile.is_file():
#     print("ok")
#   else:
#     print("no")

comUser = 'data_migration'
comPwd = 'data_migration@123'
comPort = 3306

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

def table_judge():
  print("table_judge")
  sourceTableExist = pymysql.connect(sourceIp,comUser,comPwd,sourceDbName,comPort,charset='utf8')
  desTableExist = pymysql.connect(desIp,comUser,comPwd,desDbName,comPort,charset='utf8')
  sourceTables = []
  desTables = []
  cursor_source = sourceTableExist.cursor()
  cursor_des = desTableExist.cursor()

  for table in tableList:
    #print(table)
    cursor_source.execute("select TABLE_NAME from information_schema.TABLES where TABLE_SCHEMA='%s' and TABLE_NAME='%s';" % (sourceDbName,table))
    sourceTable_tmp = cursor_source.fetchall()
    cursor_des.execute("select TABLE_NAME from information_schema.TABLES where TABLE_SCHEMA='%s' and TABLE_NAME='%s';" % (desDbName,table))
    desTable_tmp = cursor_des.fetchall()
    #print(desTable_tmp)
    if sourceTable_tmp is ():
      sourceTables.append(table)
    if desTable_tmp is not ():
      desTables.append(desTable_tmp[0][0])
  sourceTableExist.close()
  desTableExist.close()

  s=d=0
  if sourceTables != []:
    print('迁移源不存在将要迁移的表:',sourceIp,sourceDbName, sourceTables,' 请检查')
    s=1
  if desTables != []:
    print('目标库存在将要迁移的表:',desIp,desDbName,desTables,' 请移除')
    d=1
  if s == 1 or d == 1:
    sys.exit()

def data_sync():
  print('data_sync')
  db_source = pymysql.connect(sourceIp,comUser,comPwd,sourceDbName,comPort,charset='utf8')
  db_des = pymysql.connect(desIp,comUser,comPwd,desDbName,comPort,charset='utf8')
  cursor_db_source = db_source.cursor()
  cursor_db_des = db_des.cursor()

  for table in tableList:
    print("正在同步表:",table)
    cursor_db_source.execute("show create table %s;" % (table))
    createTableSQL = cursor_db_source.fetchall()[0][1]
    print(createTableSQL)
    try:
      cursor_db_des.execute(createTableSQL)
    except Exception as error:
      print(error)
    cursor_db_source.execute("flush table %s with read lock;" % (table))
    cursor_db_des.execute("alter table %s discard tablespace;" % (table))

    client.connect(sourceIp, 22, 'root')
    stdin1, stdout1, stderr1 = client.exec_command("scp %s %s:%s " % (sourceDataDir+"/"+table+".ibd", desIp, desDataDir))
    stdin2, stdout2, stderr2 = client.exec_command("scp %s %s:%s " % (sourceDataDir+"/"+table+".cfg", desIp, desDataDir))
    a_e_1 = stderr1.readlines()
    a_e_2 = stderr2.readlines()
    if a_e_1 != [] or a_e_2 != []:
      print(a_e_1,a_e_2)
      sys.exit()
    client.close()

    client.connect(desIp, 22, 'root')
    stdin3, stdout3, stderr3 = client.exec_command("chown -R mysql.mysql %s*" % (desDataDir+"/"+table))
    a_e_3 = stderr3.readlines()
    if a_e_3 != []:
      print(a_e_1, a_e_2)
      sys.exit()
    client.close()
    #cursor_db_source.execute("select sleep(10);")
    cursor_db_source.execute("unlock tables;")
    cursor_db_des.execute("alter table %s import tablespace;" % (table))
    print("同步完成")

  cursor_db_source.close()
  cursor_db_des.close()

def data_checksum():
  print('data_checksum')
  db_source = pymysql.connect(sourceIp,comUser,comPwd,sourceDbName,comPort,charset='utf8')
  db_des = pymysql.connect(desIp,comUser,comPwd,desDbName,comPort,charset='utf8')
  cursor_db_source = db_source.cursor()
  cursor_db_des = db_des.cursor()

  for table in tableList:
    print("正在校验表:", table)
    cursor_db_source.execute("checksum table %s;" % (table))
    ck_s = cursor_db_source.fetchall()[0][1]
    cursor_db_des.execute("checksum table %s;" % (table))
    ck_d = cursor_db_des.fetchall()[0][1]
    if ck_s != ck_d:
      print("表不一致:",table)
    else:
      print("表一致:",table)

  cursor_db_source.close()
  cursor_db_des.close()

if __name__ == "__main__":
  table_judge()
  data_sync()
  data_checksum()
  print('haha')
Python 相关文章推荐
Python中__name__的使用实例
Apr 14 Python
Python学习之Anaconda的使用与配置方法
Jan 04 Python
Python基于FTP模块实现ftp文件上传操作示例
Apr 23 Python
Python实现自定义函数的5种常见形式分析
Jun 16 Python
浅析python中numpy包中的argsort函数的使用
Aug 30 Python
Python实现通过解析域名获取ip地址的方法分析
May 17 Python
python实现自动化上线脚本的示例
Jul 01 Python
python pyinstaller打包exe报错的解决方法
Nov 02 Python
解决django-xadmin列表页filter关联对象搜索问题
Nov 15 Python
浅谈Python里面None True False之间的区别
Jul 09 Python
解析python 类方法、对象方法、静态方法
Aug 15 Python
Python利用socket模块开发简单的端口扫描工具的实现
Jan 27 Python
在python中将字符串转为json对象并取值的方法
Dec 31 #Python
对python中Json与object转化的方法详解
Dec 31 #Python
python使用zip将list转为json的方法
Dec 31 #Python
python 获取utc时间转化为本地时间的方法
Dec 31 #Python
python 实现UTC时间加减的方法
Dec 31 #Python
Python从单元素字典中获取key和value的实例
Dec 31 #Python
对Python 两大环境管理神器 pyenv 和 virtualenv详解
Dec 31 #Python
You might like
实用函数2
2007/11/08 PHP
php使用pack处理二进制文件的方法
2014/07/03 PHP
PHP查询附近的人及其距离的实现方法
2016/05/11 PHP
laravel框架语言包拓展实现方法分析
2019/11/22 PHP
javascript支持firefox,ie7页面布局拖拽效果代码
2007/12/20 Javascript
给Function做的OOP扩展
2009/05/07 Javascript
js function使用心得
2010/05/10 Javascript
JS控件ASP.NET的treeview控件全选或者取消(示例代码)
2013/12/16 Javascript
jquery ajax,ashx,json的用法总结
2014/02/12 Javascript
jQuery链式调用与show知识浅析
2016/05/11 Javascript
js获取页面引用的css样式表中的属性值方法(推荐)
2016/08/19 Javascript
JS图片左右无缝隙滚动的实现(兼容IE,Firefox 遵循W3C标准)
2016/09/23 Javascript
javascript算法之二叉搜索树的示例代码
2017/09/12 Javascript
浅谈Node Inspector 代理实现
2017/10/19 Javascript
layerUI下的绑定事件实例代码
2018/08/17 Javascript
Python元字符的用法实例解析
2018/01/17 Python
PyQt5打开文件对话框QFileDialog实例代码
2018/02/07 Python
Python实现的建造者模式示例
2018/08/06 Python
python的schedule定时任务模块二次封装方法
2019/02/19 Python
使用Python制作简单的小程序IP查看器功能
2019/04/16 Python
Python中如何导入类示例详解
2019/04/17 Python
Django框架模板语言实例小结【变量,标签,过滤器,继承,html转义】
2019/05/23 Python
利用anaconda保证64位和32位的python共存
2021/03/09 Python
pyqt5 QlistView列表显示的实现示例
2020/03/24 Python
Python SMTP配置参数并发送邮件
2020/06/16 Python
浅谈TensorFlow之稀疏张量表示
2020/06/30 Python
Python数据可视化常用4大绘图库原理详解
2020/10/23 Python
PatPat德国:妈妈的每日优惠
2019/10/02 全球购物
求职简历自荐信范文
2013/10/21 职场文书
工作失职检讨书范文
2014/01/16 职场文书
爱祖国爱家乡演讲稿
2014/09/02 职场文书
党员四风问题对照检查材料
2014/09/27 职场文书
绍兴鲁迅故居导游词
2015/02/09 职场文书
小学教师个人工作总结2015
2015/04/20 职场文书
python的netCDF4批量处理NC格式文件的操作方法
2022/03/21 Python
win10键盘驱动怎么修复?Win10键盘驱动修复小技巧
2022/04/06 数码科技