python制作mysql数据迁移脚本


Posted in Python onJanuary 01, 2019

用python写了个数据迁移脚本,主要是利用从库将大的静态表导出表空间,载导入到目标实例中。

#!/usr/bin/env python3
#-*- coding:utf8 -*-
#author:zhanbin.liu
#!!!!!DB必须同版本
#python3环境  pip3 install pymysql paramiko

import os
#from pathlib import Path
import sys
import pymysql
import paramiko

#每次只能迁移一个DB下的表,到指定DB
#GRANT SELECT, CREATE, RELOAD, ALTER, LOCK TABLES ON *.* TO 'data_migration'@'192.168.%' IDENTIFIED BY 'data_migration@123';
tables='sqlauto_cluster,sqlauto_user'    #以,分割的字符串,如a,b,c
tableList = tables.split(',')
sourceIp = '192.168.1.101'
sourceDataBase = '/data/mysql/3306/data'
sourceDbName = 'inception_web'
sourceDataDir = os.path.join(sourceDataBase,sourceDbName)
desIp = '192.168.1.102'
desDataBase = '/data/mysql/3306/data'
desDbName = 'inception_web'
desDataDir = os.path.join(desDataBase,desDbName)

# for table in tableList:
#   desFile = Path("%s/%s.ibd" %(desDataDir,table))
#   print(desFile)
#   if desFile.is_file():
#     print("ok")
#   else:
#     print("no")

comUser = 'data_migration'
comPwd = 'data_migration@123'
comPort = 3306

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

def table_judge():
  print("table_judge")
  sourceTableExist = pymysql.connect(sourceIp,comUser,comPwd,sourceDbName,comPort,charset='utf8')
  desTableExist = pymysql.connect(desIp,comUser,comPwd,desDbName,comPort,charset='utf8')
  sourceTables = []
  desTables = []
  cursor_source = sourceTableExist.cursor()
  cursor_des = desTableExist.cursor()

  for table in tableList:
    #print(table)
    cursor_source.execute("select TABLE_NAME from information_schema.TABLES where TABLE_SCHEMA='%s' and TABLE_NAME='%s';" % (sourceDbName,table))
    sourceTable_tmp = cursor_source.fetchall()
    cursor_des.execute("select TABLE_NAME from information_schema.TABLES where TABLE_SCHEMA='%s' and TABLE_NAME='%s';" % (desDbName,table))
    desTable_tmp = cursor_des.fetchall()
    #print(desTable_tmp)
    if sourceTable_tmp is ():
      sourceTables.append(table)
    if desTable_tmp is not ():
      desTables.append(desTable_tmp[0][0])
  sourceTableExist.close()
  desTableExist.close()

  s=d=0
  if sourceTables != []:
    print('迁移源不存在将要迁移的表:',sourceIp,sourceDbName, sourceTables,' 请检查')
    s=1
  if desTables != []:
    print('目标库存在将要迁移的表:',desIp,desDbName,desTables,' 请移除')
    d=1
  if s == 1 or d == 1:
    sys.exit()

def data_sync():
  print('data_sync')
  db_source = pymysql.connect(sourceIp,comUser,comPwd,sourceDbName,comPort,charset='utf8')
  db_des = pymysql.connect(desIp,comUser,comPwd,desDbName,comPort,charset='utf8')
  cursor_db_source = db_source.cursor()
  cursor_db_des = db_des.cursor()

  for table in tableList:
    print("正在同步表:",table)
    cursor_db_source.execute("show create table %s;" % (table))
    createTableSQL = cursor_db_source.fetchall()[0][1]
    print(createTableSQL)
    try:
      cursor_db_des.execute(createTableSQL)
    except Exception as error:
      print(error)
    cursor_db_source.execute("flush table %s with read lock;" % (table))
    cursor_db_des.execute("alter table %s discard tablespace;" % (table))

    client.connect(sourceIp, 22, 'root')
    stdin1, stdout1, stderr1 = client.exec_command("scp %s %s:%s " % (sourceDataDir+"/"+table+".ibd", desIp, desDataDir))
    stdin2, stdout2, stderr2 = client.exec_command("scp %s %s:%s " % (sourceDataDir+"/"+table+".cfg", desIp, desDataDir))
    a_e_1 = stderr1.readlines()
    a_e_2 = stderr2.readlines()
    if a_e_1 != [] or a_e_2 != []:
      print(a_e_1,a_e_2)
      sys.exit()
    client.close()

    client.connect(desIp, 22, 'root')
    stdin3, stdout3, stderr3 = client.exec_command("chown -R mysql.mysql %s*" % (desDataDir+"/"+table))
    a_e_3 = stderr3.readlines()
    if a_e_3 != []:
      print(a_e_1, a_e_2)
      sys.exit()
    client.close()
    #cursor_db_source.execute("select sleep(10);")
    cursor_db_source.execute("unlock tables;")
    cursor_db_des.execute("alter table %s import tablespace;" % (table))
    print("同步完成")

  cursor_db_source.close()
  cursor_db_des.close()

def data_checksum():
  print('data_checksum')
  db_source = pymysql.connect(sourceIp,comUser,comPwd,sourceDbName,comPort,charset='utf8')
  db_des = pymysql.connect(desIp,comUser,comPwd,desDbName,comPort,charset='utf8')
  cursor_db_source = db_source.cursor()
  cursor_db_des = db_des.cursor()

  for table in tableList:
    print("正在校验表:", table)
    cursor_db_source.execute("checksum table %s;" % (table))
    ck_s = cursor_db_source.fetchall()[0][1]
    cursor_db_des.execute("checksum table %s;" % (table))
    ck_d = cursor_db_des.fetchall()[0][1]
    if ck_s != ck_d:
      print("表不一致:",table)
    else:
      print("表一致:",table)

  cursor_db_source.close()
  cursor_db_des.close()

if __name__ == "__main__":
  table_judge()
  data_sync()
  data_checksum()
  print('haha')
Python 相关文章推荐
跟老齐学Python之正规地说一句话
Sep 28 Python
Python中类型检查的详细介绍
Feb 13 Python
基于python时间处理方法(详解)
Aug 14 Python
JavaScript实现一维数组转化为二维数组
Apr 17 Python
Python XML转Json之XML2Dict的使用方法
Jan 15 Python
python实现Virginia无密钥解密
Mar 20 Python
Python redis操作实例分析【连接、管道、发布和订阅等】
May 16 Python
pyQt5实时刷新界面的示例
Jun 25 Python
python绘制已知点的坐标的直线实例
Jul 04 Python
python模拟鼠标点击和键盘输入的操作
Aug 04 Python
Pytorch在dataloader类中设置shuffle的随机数种子方式
Jan 14 Python
Python如何解决secure_filename对中文不支持问题
Jul 16 Python
在python中将字符串转为json对象并取值的方法
Dec 31 #Python
对python中Json与object转化的方法详解
Dec 31 #Python
python使用zip将list转为json的方法
Dec 31 #Python
python 获取utc时间转化为本地时间的方法
Dec 31 #Python
python 实现UTC时间加减的方法
Dec 31 #Python
Python从单元素字典中获取key和value的实例
Dec 31 #Python
对Python 两大环境管理神器 pyenv 和 virtualenv详解
Dec 31 #Python
You might like
php中取得URL的根域名的代码
2011/03/23 PHP
Zend Framework框架之Zend_Mail实现发送Email邮件验证功能及解决标题乱码的方法
2016/03/21 PHP
Laravel 自带的Auth验证登录方法
2019/09/30 PHP
JavaScript 产生不重复的随机数三种实现思路
2012/12/13 Javascript
js单例模式的两种方案
2013/10/22 Javascript
JavaScript使用pop方法移除数组最后一个元素用法实例
2015/04/06 Javascript
webpack+vue.js实现组件化详解
2016/10/12 Javascript
node学习记录之搭建web服务器教程
2017/02/16 Javascript
js实现简易垂直滚动条
2017/02/22 Javascript
canvas实现简易的圆环进度条效果
2017/02/28 Javascript
Vue开发中整合axios的文件整理
2017/04/29 Javascript
bootstrap table服务端实现分页效果
2017/08/10 Javascript
关于vue编译版本引入的问题的解决
2018/09/17 Javascript
VUE 配置vue-devtools调试工具及安装方法
2018/09/30 Javascript
angular 实现同步验证器跨字段验证的方法
2019/04/11 Javascript
Vue的H5页面唤起支付宝支付功能
2019/04/18 Javascript
Webpack3+React16代码分割的实现
2021/03/03 Javascript
[03:01]完美世界DOTA2联赛PWL S2 集锦第二期
2020/12/03 DOTA
用python 制作图片转pdf工具
2015/01/30 Python
qpython3 读取安卓lastpass Cookies
2016/06/19 Python
详解关于Django中ORM数据库迁移的配置
2018/10/08 Python
python  Django中的apps.py的目的是什么
2018/10/15 Python
pandas分别写入excel的不同sheet方法
2018/12/11 Python
对Python中class和instance以及self的用法详解
2019/06/26 Python
python3 使用openpyxl将mysql数据写入xlsx的操作
2020/05/15 Python
详解python百行有效代码实现汉诺塔小游戏(简约版)
2020/10/30 Python
用纯CSS3实现网页中常见的小箭头
2017/10/16 HTML / CSS
幼儿园美术教学反思
2014/01/31 职场文书
幼儿园教育教学反思
2014/01/31 职场文书
我的中国梦演讲稿400字
2014/08/19 职场文书
教师岗位职责
2015/02/03 职场文书
迎客户欢迎词三篇
2019/09/27 职场文书
简单了解 MySQL 中相关的锁
2021/05/25 MySQL
Python基础知识学习之类的继承
2021/05/31 Python
Redis可视化客户端小结
2021/06/10 Redis
吉利入股戴姆勒后smart“长大了”
2022/04/21 数码科技