python制作mysql数据迁移脚本


Posted in Python onJanuary 01, 2019

用python写了个数据迁移脚本,主要是利用从库将大的静态表导出表空间,载导入到目标实例中。

#!/usr/bin/env python3
#-*- coding:utf8 -*-
#author:zhanbin.liu
#!!!!!DB必须同版本
#python3环境  pip3 install pymysql paramiko

import os
#from pathlib import Path
import sys
import pymysql
import paramiko

#每次只能迁移一个DB下的表,到指定DB
#GRANT SELECT, CREATE, RELOAD, ALTER, LOCK TABLES ON *.* TO 'data_migration'@'192.168.%' IDENTIFIED BY 'data_migration@123';
tables='sqlauto_cluster,sqlauto_user'    #以,分割的字符串,如a,b,c
tableList = tables.split(',')
sourceIp = '192.168.1.101'
sourceDataBase = '/data/mysql/3306/data'
sourceDbName = 'inception_web'
sourceDataDir = os.path.join(sourceDataBase,sourceDbName)
desIp = '192.168.1.102'
desDataBase = '/data/mysql/3306/data'
desDbName = 'inception_web'
desDataDir = os.path.join(desDataBase,desDbName)

# for table in tableList:
#   desFile = Path("%s/%s.ibd" %(desDataDir,table))
#   print(desFile)
#   if desFile.is_file():
#     print("ok")
#   else:
#     print("no")

comUser = 'data_migration'
comPwd = 'data_migration@123'
comPort = 3306

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

def table_judge():
  print("table_judge")
  sourceTableExist = pymysql.connect(sourceIp,comUser,comPwd,sourceDbName,comPort,charset='utf8')
  desTableExist = pymysql.connect(desIp,comUser,comPwd,desDbName,comPort,charset='utf8')
  sourceTables = []
  desTables = []
  cursor_source = sourceTableExist.cursor()
  cursor_des = desTableExist.cursor()

  for table in tableList:
    #print(table)
    cursor_source.execute("select TABLE_NAME from information_schema.TABLES where TABLE_SCHEMA='%s' and TABLE_NAME='%s';" % (sourceDbName,table))
    sourceTable_tmp = cursor_source.fetchall()
    cursor_des.execute("select TABLE_NAME from information_schema.TABLES where TABLE_SCHEMA='%s' and TABLE_NAME='%s';" % (desDbName,table))
    desTable_tmp = cursor_des.fetchall()
    #print(desTable_tmp)
    if sourceTable_tmp is ():
      sourceTables.append(table)
    if desTable_tmp is not ():
      desTables.append(desTable_tmp[0][0])
  sourceTableExist.close()
  desTableExist.close()

  s=d=0
  if sourceTables != []:
    print('迁移源不存在将要迁移的表:',sourceIp,sourceDbName, sourceTables,' 请检查')
    s=1
  if desTables != []:
    print('目标库存在将要迁移的表:',desIp,desDbName,desTables,' 请移除')
    d=1
  if s == 1 or d == 1:
    sys.exit()

def data_sync():
  print('data_sync')
  db_source = pymysql.connect(sourceIp,comUser,comPwd,sourceDbName,comPort,charset='utf8')
  db_des = pymysql.connect(desIp,comUser,comPwd,desDbName,comPort,charset='utf8')
  cursor_db_source = db_source.cursor()
  cursor_db_des = db_des.cursor()

  for table in tableList:
    print("正在同步表:",table)
    cursor_db_source.execute("show create table %s;" % (table))
    createTableSQL = cursor_db_source.fetchall()[0][1]
    print(createTableSQL)
    try:
      cursor_db_des.execute(createTableSQL)
    except Exception as error:
      print(error)
    cursor_db_source.execute("flush table %s with read lock;" % (table))
    cursor_db_des.execute("alter table %s discard tablespace;" % (table))

    client.connect(sourceIp, 22, 'root')
    stdin1, stdout1, stderr1 = client.exec_command("scp %s %s:%s " % (sourceDataDir+"/"+table+".ibd", desIp, desDataDir))
    stdin2, stdout2, stderr2 = client.exec_command("scp %s %s:%s " % (sourceDataDir+"/"+table+".cfg", desIp, desDataDir))
    a_e_1 = stderr1.readlines()
    a_e_2 = stderr2.readlines()
    if a_e_1 != [] or a_e_2 != []:
      print(a_e_1,a_e_2)
      sys.exit()
    client.close()

    client.connect(desIp, 22, 'root')
    stdin3, stdout3, stderr3 = client.exec_command("chown -R mysql.mysql %s*" % (desDataDir+"/"+table))
    a_e_3 = stderr3.readlines()
    if a_e_3 != []:
      print(a_e_1, a_e_2)
      sys.exit()
    client.close()
    #cursor_db_source.execute("select sleep(10);")
    cursor_db_source.execute("unlock tables;")
    cursor_db_des.execute("alter table %s import tablespace;" % (table))
    print("同步完成")

  cursor_db_source.close()
  cursor_db_des.close()

def data_checksum():
  print('data_checksum')
  db_source = pymysql.connect(sourceIp,comUser,comPwd,sourceDbName,comPort,charset='utf8')
  db_des = pymysql.connect(desIp,comUser,comPwd,desDbName,comPort,charset='utf8')
  cursor_db_source = db_source.cursor()
  cursor_db_des = db_des.cursor()

  for table in tableList:
    print("正在校验表:", table)
    cursor_db_source.execute("checksum table %s;" % (table))
    ck_s = cursor_db_source.fetchall()[0][1]
    cursor_db_des.execute("checksum table %s;" % (table))
    ck_d = cursor_db_des.fetchall()[0][1]
    if ck_s != ck_d:
      print("表不一致:",table)
    else:
      print("表一致:",table)

  cursor_db_source.close()
  cursor_db_des.close()

if __name__ == "__main__":
  table_judge()
  data_sync()
  data_checksum()
  print('haha')
Python 相关文章推荐
Python的Flask开发框架简单上手笔记
Nov 16 Python
理解Python中的With语句
Mar 18 Python
Python中绑定与未绑定的类方法用法分析
Apr 29 Python
Django学习笔记之Class-Based-View
Feb 15 Python
Python求出0~100以内的所有素数
Jan 23 Python
Django CSRF跨站请求伪造防护过程解析
Jul 31 Python
python with (as)语句实例详解
Feb 04 Python
Python利用逻辑回归分类实现模板
Feb 15 Python
Python中用pyinstaller打包时的图标问题及解决方法
Feb 17 Python
python GUI库图形界面开发之PyQt5输入对话框QInputDialog详细使用方法与实例
Feb 27 Python
PyTorch的torch.cat用法
Jun 28 Python
Python中的特殊方法以及应用详解
Sep 20 Python
在python中将字符串转为json对象并取值的方法
Dec 31 #Python
对python中Json与object转化的方法详解
Dec 31 #Python
python使用zip将list转为json的方法
Dec 31 #Python
python 获取utc时间转化为本地时间的方法
Dec 31 #Python
python 实现UTC时间加减的方法
Dec 31 #Python
Python从单元素字典中获取key和value的实例
Dec 31 #Python
对Python 两大环境管理神器 pyenv 和 virtualenv详解
Dec 31 #Python
You might like
虚拟主机中对PHP的特殊设置
2006/10/09 PHP
php变量作用域的深入解析
2013/06/03 PHP
学习php设计模式 php实现模板方法模式
2015/12/08 PHP
php的闭包(Closure)匿名函数初探
2016/02/14 PHP
tp5框架使用composer实现日志记录功能示例
2019/01/10 PHP
laravel实现一个上传图片的接口,并建立软链接,访问图片的方法
2019/10/12 PHP
js触发onchange事件的方法说明
2014/03/08 Javascript
javascript模拟评分控件实现方法
2015/05/13 Javascript
JS设置下拉列表框当前所选值的方法
2015/12/22 Javascript
jquery模拟实现鼠标指针停止运动事件
2016/01/12 Javascript
详解javascript实现自定义事件
2016/01/19 Javascript
JavaScript实现复制或剪切内容到剪贴板功能的方法
2016/05/23 Javascript
vue-cli 脚手架基于Nightwatch的端到端测试环境的过程
2018/09/30 Javascript
Phaser.js实现简单的跑酷游戏附源码下载
2018/10/26 Javascript
webpack中如何使用雪碧图的示例代码
2018/11/11 Javascript
如何为vue的项目添加单元测试
2018/12/19 Javascript
seajs和requirejs模块化简单案例分析
2019/08/26 Javascript
js实现图片区域可点击大小随意改变(适用移动端)代码实例
2019/09/11 Javascript
Vue路由对象属性 .meta $route.matched详解
2019/11/04 Javascript
vue element-ui实现input输入框金额数字添加千分位
2019/12/29 Javascript
JavaScript中条件语句的优化技巧总结
2020/12/04 Javascript
Python标准库之随机数 (math包、random包)介绍
2014/11/25 Python
用实例解释Python中的继承和多态的概念
2015/04/27 Python
python3制作捧腹网段子页爬虫
2017/02/12 Python
python笔记_将循环内容在一行输出的方法
2019/08/08 Python
Python importlib动态导入模块实现代码
2020/04/16 Python
python实现录音功能(可随时停止录音)
2020/10/26 Python
Python descriptor(描述符)的实现
2020/11/15 Python
Django中ORM的基本使用教程
2020/12/22 Python
CSS3 3D酷炫立方体变换动画的实现
2019/03/26 HTML / CSS
Melissa鞋马来西亚官方网站:MDreams马来西亚
2018/04/05 全球购物
Crucial英睿达法国官网:内存条及SSD固态硬盘升级
2018/07/13 全球购物
《找不到快乐的波斯猫》教学反思
2014/02/24 职场文书
作风整顿个人剖析材料
2014/10/06 职场文书
2015年行政人事工作总结
2015/05/21 职场文书
2016年第二十五次全国助残日活动总结
2016/04/01 职场文书