Pytorch中Softmax和LogSoftmax的使用详解


Posted in Python onJune 05, 2021

一、函数解释

1.Softmax函数常用的用法是指定参数dim就可以:

(1)dim=0:对每一列的所有元素进行softmax运算,并使得每一列所有元素和为1。

(2)dim=1:对每一行的所有元素进行softmax运算,并使得每一行所有元素和为1。

class Softmax(Module):
    r"""Applies the Softmax function to an n-dimensional input Tensor
    rescaling them so that the elements of the n-dimensional output Tensor
    lie in the range [0,1] and sum to 1.
    Softmax is defined as:
    .. math::
        \text{Softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)}
    Shape:
        - Input: :math:`(*)` where `*` means, any number of additional
          dimensions
        - Output: :math:`(*)`, same shape as the input
    Returns:
        a Tensor of the same dimension and shape as the input with
        values in the range [0, 1]
    Arguments:
        dim (int): A dimension along which Softmax will be computed (so every slice
            along dim will sum to 1).
    .. note::
        This module doesn't work directly with NLLLoss,
        which expects the Log to be computed between the Softmax and itself.
        Use `LogSoftmax` instead (it's faster and has better numerical properties).
    Examples::
        >>> m = nn.Softmax(dim=1)
        >>> input = torch.randn(2, 3)
        >>> output = m(input)
    """
    __constants__ = ['dim']
 
    def __init__(self, dim=None):
        super(Softmax, self).__init__()
        self.dim = dim
 
    def __setstate__(self, state):
        self.__dict__.update(state)
        if not hasattr(self, 'dim'):
            self.dim = None
 
    def forward(self, input):
        return F.softmax(input, self.dim, _stacklevel=5)
 
    def extra_repr(self):
        return 'dim={dim}'.format(dim=self.dim)

2.LogSoftmax其实就是对softmax的结果进行log,即Log(Softmax(x))

class LogSoftmax(Module):
    r"""Applies the :math:`\log(\text{Softmax}(x))` function to an n-dimensional
    input Tensor. The LogSoftmax formulation can be simplified as:
    .. math::
        \text{LogSoftmax}(x_{i}) = \log\left(\frac{\exp(x_i) }{ \sum_j \exp(x_j)} \right)
    Shape:
        - Input: :math:`(*)` where `*` means, any number of additional
          dimensions
        - Output: :math:`(*)`, same shape as the input
    Arguments:
        dim (int): A dimension along which LogSoftmax will be computed.
    Returns:
        a Tensor of the same dimension and shape as the input with
        values in the range [-inf, 0)
    Examples::
        >>> m = nn.LogSoftmax()
        >>> input = torch.randn(2, 3)
        >>> output = m(input)
    """
    __constants__ = ['dim']
 
    def __init__(self, dim=None):
        super(LogSoftmax, self).__init__()
        self.dim = dim
 
    def __setstate__(self, state):
        self.__dict__.update(state)
        if not hasattr(self, 'dim'):
            self.dim = None
 
    def forward(self, input):
        return F.log_softmax(input, self.dim, _stacklevel=5)

二、代码示例

输入代码

import torch
import torch.nn as nn
import numpy as np
 
batch_size = 4
class_num = 6
inputs = torch.randn(batch_size, class_num)
for i in range(batch_size):
    for j in range(class_num):
        inputs[i][j] = (i + 1) * (j + 1)
 
print("inputs:", inputs)

得到大小batch_size为4,类别数为6的向量(可以理解为经过最后一层得到)

tensor([[ 1., 2., 3., 4., 5., 6.],
[ 2., 4., 6., 8., 10., 12.],
[ 3., 6., 9., 12., 15., 18.],
[ 4., 8., 12., 16., 20., 24.]])

接着我们对该向量每一行进行Softmax

Softmax = nn.Softmax(dim=1)
probs = Softmax(inputs)
print("probs:\n", probs)

得到

tensor([[4.2698e-03, 1.1606e-02, 3.1550e-02, 8.5761e-02, 2.3312e-01, 6.3369e-01],
[3.9256e-05, 2.9006e-04, 2.1433e-03, 1.5837e-02, 1.1702e-01, 8.6467e-01],
[2.9067e-07, 5.8383e-06, 1.1727e-04, 2.3553e-03, 4.7308e-02, 9.5021e-01],
[2.0234e-09, 1.1047e-07, 6.0317e-06, 3.2932e-04, 1.7980e-02, 9.8168e-01]])

此外,我们对该向量每一行进行LogSoftmax

LogSoftmax = nn.LogSoftmax(dim=1)
log_probs = LogSoftmax(inputs)
print("log_probs:\n", log_probs)

得到

tensor([[-5.4562e+00, -4.4562e+00, -3.4562e+00, -2.4562e+00, -1.4562e+00, -4.5619e-01],
[-1.0145e+01, -8.1454e+00, -6.1454e+00, -4.1454e+00, -2.1454e+00, -1.4541e-01],
[-1.5051e+01, -1.2051e+01, -9.0511e+00, -6.0511e+00, -3.0511e+00, -5.1069e-02],
[-2.0018e+01, -1.6018e+01, -1.2018e+01, -8.0185e+00, -4.0185e+00, -1.8485e-02]])

验证每一行元素和是否为1

# probs_sum in dim=1
probs_sum = [0 for i in range(batch_size)]
 
for i in range(batch_size):
    for j in range(class_num):
        probs_sum[i] += probs[i][j]
    print(i, "row probs sum:", probs_sum[i])

得到每一行的和,看到确实为1

0 row probs sum: tensor(1.)
1 row probs sum: tensor(1.0000)
2 row probs sum: tensor(1.)
3 row probs sum: tensor(1.)

验证LogSoftmax是对Softmax的结果进行Log

# to numpy
np_probs = probs.data.numpy()
print("numpy probs:\n", np_probs)
 
# np.log()
log_np_probs = np.log(np_probs)
print("log numpy probs:\n", log_np_probs)

得到

numpy probs:
[[4.26977826e-03 1.16064614e-02 3.15496325e-02 8.57607946e-02 2.33122006e-01 6.33691311e-01]
[3.92559559e-05 2.90064461e-04 2.14330270e-03 1.58369839e-02 1.17020354e-01 8.64669979e-01]
[2.90672347e-07 5.83831024e-06 1.17265590e-04 2.35534250e-03 4.73083146e-02 9.50212955e-01]
[2.02340233e-09 1.10474026e-07 6.03167746e-06 3.29318427e-04 1.79801770e-02 9.81684387e-01]]
log numpy probs:
[[-5.4561934e+00 -4.4561934e+00 -3.4561934e+00 -2.4561932e+00 -1.4561933e+00 -4.5619333e-01]
[-1.0145408e+01 -8.1454077e+00 -6.1454072e+00 -4.1454072e+00 -2.1454074e+00 -1.4540738e-01]
[-1.5051069e+01 -1.2051069e+01 -9.0510693e+00 -6.0510693e+00 -3.0510693e+00 -5.1069155e-02]
[-2.0018486e+01 -1.6018486e+01 -1.2018485e+01 -8.0184851e+00 -4.0184855e+00 -1.8485421e-02]]

验证完毕

三、整体代码

import torch
import torch.nn as nn
import numpy as np
 
batch_size = 4
class_num = 6
inputs = torch.randn(batch_size, class_num)
for i in range(batch_size):
    for j in range(class_num):
        inputs[i][j] = (i + 1) * (j + 1)
 
print("inputs:", inputs)
Softmax = nn.Softmax(dim=1)
probs = Softmax(inputs)
print("probs:\n", probs)
 
LogSoftmax = nn.LogSoftmax(dim=1)
log_probs = LogSoftmax(inputs)
print("log_probs:\n", log_probs)
 
# probs_sum in dim=1
probs_sum = [0 for i in range(batch_size)]
 
for i in range(batch_size):
    for j in range(class_num):
        probs_sum[i] += probs[i][j]
    print(i, "row probs sum:", probs_sum[i])
 
# to numpy
np_probs = probs.data.numpy()
print("numpy probs:\n", np_probs)
 
# np.log()
log_np_probs = np.log(np_probs)
print("log numpy probs:\n", log_np_probs)

基于pytorch softmax,logsoftmax 表达

import torch
import numpy as np
input = torch.autograd.Variable(torch.rand(1, 3))

print(input)
print('softmax={}'.format(torch.nn.functional.softmax(input, dim=1)))
print('logsoftmax={}'.format(np.log(torch.nn.functional.softmax(input, dim=1))))

以上为个人经验,希望能给大家一个参考,也希望大家多多支持三水点靠木。

Python 相关文章推荐
python线程池(threadpool)模块使用笔记详解
Nov 17 Python
Python虚拟环境项目实例
Nov 20 Python
Python爬虫获取图片并下载保存至本地的实例
Jun 01 Python
pycharm运行出现ImportError:No module named的解决方法
Oct 13 Python
Python中flatten( )函数及函数用法详解
Nov 02 Python
python 通过麦克风录音 生成wav文件的方法
Jan 09 Python
Python实现的对本地host127.0.0.1主机进行扫描端口功能示例
Feb 15 Python
python字符串查找函数的用法详解
Jul 08 Python
Tensorflow 自定义loss的情况下初始化部分变量方式
Jan 06 Python
Pytorch模型转onnx模型实例
Jan 15 Python
django日志默认打印request请求信息的方法示例
May 17 Python
Python中for后接else的语法使用
May 18 Python
Pytorch中Softmax与LogSigmoid的对比分析
Jun 05 #Python
Pytorch反向传播中的细节-计算梯度时的默认累加操作
pytorch 梯度NAN异常值的解决方案
Jun 05 #Python
pytorch 权重weight 与 梯度grad 可视化操作
PyTorch 如何检查模型梯度是否可导
python-opencv 中值滤波{cv2.medianBlur(src, ksize)}的用法
解决Pytorch修改预训练模型时遇到key不匹配的情况
Jun 05 #Python
You might like
php文章内容分页并生成相应的htm静态页面代码
2010/06/07 PHP
php获取指定日期之间的各个周和月的起止时间
2014/11/24 PHP
网站防止被刷票的一些思路与方法
2015/01/08 PHP
Laravel中批量赋值Mass-Assignment的真正含义详解
2017/09/29 PHP
jQuery 数据缓存data(name, value)详解及实现
2010/01/04 Javascript
JQuery仿小米手机抢购页面倒计时效果
2014/12/16 Javascript
jQuery中弹出iframe内嵌页面元素到父页面并全屏化的实例代码
2016/12/27 Javascript
Javascript设计模式之装饰者模式详解篇
2017/01/17 Javascript
基于AngularJS的拖拽文件上传的实例代码
2017/07/15 Javascript
利用vue组件自定义v-model实现一个Tab组件方法示例
2017/12/06 Javascript
微信小程序radio组件使用详解
2018/01/31 Javascript
Node.js爬取豆瓣数据实例分析
2018/03/05 Javascript
vue中v-for循环给标签属性赋值的方法
2018/10/18 Javascript
Vue.js中对css的操作(修改)具体方式详解
2018/10/30 Javascript
小程序登录态管理的方法示例
2018/11/13 Javascript
微信小程序实现吸顶特效
2020/01/08 Javascript
[49:08]Secret vs VP 2018国际邀请赛小组赛BO2 第一场 8.17
2018/08/20 DOTA
Python中使用不同编码读写txt文件详解
2015/05/28 Python
python获取文件扩展名的方法
2015/07/06 Python
Python正则替换字符串函数re.sub用法示例
2017/01/19 Python
pandas 使用apply同时处理两列数据的方法
2018/04/20 Python
Python基于SMTP协议实现发送邮件功能详解
2018/08/14 Python
解决pycharm安装后代码区不能编辑的问题
2018/10/28 Python
不到20行代码用Python做一个智能聊天机器人
2019/04/19 Python
Python实现Linux监控的方法
2019/05/16 Python
python同步两个文件夹下的内容
2019/08/29 Python
python 求两个向量的顺时针夹角操作
2021/03/04 Python
美国的Eastbay旗下的运动款子品牌:Final-Score
2018/01/01 全球购物
90后毕业生的求职信范文
2013/09/21 职场文书
计算机专业推荐信范文
2013/11/27 职场文书
大学毕业通用个人的求职信
2013/12/08 职场文书
小学社团活动总结
2014/06/27 职场文书
健康教育主题班会
2015/08/14 职场文书
工作总结之小学教师体育工作范文(3篇)
2019/10/07 职场文书
PyQt5爬取12306车票信息程序的实现
2021/05/14 Python
redis lua限流算法实现示例
2022/07/15 Redis