Pytorch中Softmax和LogSoftmax的使用详解


Posted in Python onJune 05, 2021

一、函数解释

1.Softmax函数常用的用法是指定参数dim就可以:

(1)dim=0:对每一列的所有元素进行softmax运算,并使得每一列所有元素和为1。

(2)dim=1:对每一行的所有元素进行softmax运算,并使得每一行所有元素和为1。

class Softmax(Module):
    r"""Applies the Softmax function to an n-dimensional input Tensor
    rescaling them so that the elements of the n-dimensional output Tensor
    lie in the range [0,1] and sum to 1.
    Softmax is defined as:
    .. math::
        \text{Softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)}
    Shape:
        - Input: :math:`(*)` where `*` means, any number of additional
          dimensions
        - Output: :math:`(*)`, same shape as the input
    Returns:
        a Tensor of the same dimension and shape as the input with
        values in the range [0, 1]
    Arguments:
        dim (int): A dimension along which Softmax will be computed (so every slice
            along dim will sum to 1).
    .. note::
        This module doesn't work directly with NLLLoss,
        which expects the Log to be computed between the Softmax and itself.
        Use `LogSoftmax` instead (it's faster and has better numerical properties).
    Examples::
        >>> m = nn.Softmax(dim=1)
        >>> input = torch.randn(2, 3)
        >>> output = m(input)
    """
    __constants__ = ['dim']
 
    def __init__(self, dim=None):
        super(Softmax, self).__init__()
        self.dim = dim
 
    def __setstate__(self, state):
        self.__dict__.update(state)
        if not hasattr(self, 'dim'):
            self.dim = None
 
    def forward(self, input):
        return F.softmax(input, self.dim, _stacklevel=5)
 
    def extra_repr(self):
        return 'dim={dim}'.format(dim=self.dim)

2.LogSoftmax其实就是对softmax的结果进行log,即Log(Softmax(x))

class LogSoftmax(Module):
    r"""Applies the :math:`\log(\text{Softmax}(x))` function to an n-dimensional
    input Tensor. The LogSoftmax formulation can be simplified as:
    .. math::
        \text{LogSoftmax}(x_{i}) = \log\left(\frac{\exp(x_i) }{ \sum_j \exp(x_j)} \right)
    Shape:
        - Input: :math:`(*)` where `*` means, any number of additional
          dimensions
        - Output: :math:`(*)`, same shape as the input
    Arguments:
        dim (int): A dimension along which LogSoftmax will be computed.
    Returns:
        a Tensor of the same dimension and shape as the input with
        values in the range [-inf, 0)
    Examples::
        >>> m = nn.LogSoftmax()
        >>> input = torch.randn(2, 3)
        >>> output = m(input)
    """
    __constants__ = ['dim']
 
    def __init__(self, dim=None):
        super(LogSoftmax, self).__init__()
        self.dim = dim
 
    def __setstate__(self, state):
        self.__dict__.update(state)
        if not hasattr(self, 'dim'):
            self.dim = None
 
    def forward(self, input):
        return F.log_softmax(input, self.dim, _stacklevel=5)

二、代码示例

输入代码

import torch
import torch.nn as nn
import numpy as np
 
batch_size = 4
class_num = 6
inputs = torch.randn(batch_size, class_num)
for i in range(batch_size):
    for j in range(class_num):
        inputs[i][j] = (i + 1) * (j + 1)
 
print("inputs:", inputs)

得到大小batch_size为4,类别数为6的向量(可以理解为经过最后一层得到)

tensor([[ 1., 2., 3., 4., 5., 6.],
[ 2., 4., 6., 8., 10., 12.],
[ 3., 6., 9., 12., 15., 18.],
[ 4., 8., 12., 16., 20., 24.]])

接着我们对该向量每一行进行Softmax

Softmax = nn.Softmax(dim=1)
probs = Softmax(inputs)
print("probs:\n", probs)

得到

tensor([[4.2698e-03, 1.1606e-02, 3.1550e-02, 8.5761e-02, 2.3312e-01, 6.3369e-01],
[3.9256e-05, 2.9006e-04, 2.1433e-03, 1.5837e-02, 1.1702e-01, 8.6467e-01],
[2.9067e-07, 5.8383e-06, 1.1727e-04, 2.3553e-03, 4.7308e-02, 9.5021e-01],
[2.0234e-09, 1.1047e-07, 6.0317e-06, 3.2932e-04, 1.7980e-02, 9.8168e-01]])

此外,我们对该向量每一行进行LogSoftmax

LogSoftmax = nn.LogSoftmax(dim=1)
log_probs = LogSoftmax(inputs)
print("log_probs:\n", log_probs)

得到

tensor([[-5.4562e+00, -4.4562e+00, -3.4562e+00, -2.4562e+00, -1.4562e+00, -4.5619e-01],
[-1.0145e+01, -8.1454e+00, -6.1454e+00, -4.1454e+00, -2.1454e+00, -1.4541e-01],
[-1.5051e+01, -1.2051e+01, -9.0511e+00, -6.0511e+00, -3.0511e+00, -5.1069e-02],
[-2.0018e+01, -1.6018e+01, -1.2018e+01, -8.0185e+00, -4.0185e+00, -1.8485e-02]])

验证每一行元素和是否为1

# probs_sum in dim=1
probs_sum = [0 for i in range(batch_size)]
 
for i in range(batch_size):
    for j in range(class_num):
        probs_sum[i] += probs[i][j]
    print(i, "row probs sum:", probs_sum[i])

得到每一行的和,看到确实为1

0 row probs sum: tensor(1.)
1 row probs sum: tensor(1.0000)
2 row probs sum: tensor(1.)
3 row probs sum: tensor(1.)

验证LogSoftmax是对Softmax的结果进行Log

# to numpy
np_probs = probs.data.numpy()
print("numpy probs:\n", np_probs)
 
# np.log()
log_np_probs = np.log(np_probs)
print("log numpy probs:\n", log_np_probs)

得到

numpy probs:
[[4.26977826e-03 1.16064614e-02 3.15496325e-02 8.57607946e-02 2.33122006e-01 6.33691311e-01]
[3.92559559e-05 2.90064461e-04 2.14330270e-03 1.58369839e-02 1.17020354e-01 8.64669979e-01]
[2.90672347e-07 5.83831024e-06 1.17265590e-04 2.35534250e-03 4.73083146e-02 9.50212955e-01]
[2.02340233e-09 1.10474026e-07 6.03167746e-06 3.29318427e-04 1.79801770e-02 9.81684387e-01]]
log numpy probs:
[[-5.4561934e+00 -4.4561934e+00 -3.4561934e+00 -2.4561932e+00 -1.4561933e+00 -4.5619333e-01]
[-1.0145408e+01 -8.1454077e+00 -6.1454072e+00 -4.1454072e+00 -2.1454074e+00 -1.4540738e-01]
[-1.5051069e+01 -1.2051069e+01 -9.0510693e+00 -6.0510693e+00 -3.0510693e+00 -5.1069155e-02]
[-2.0018486e+01 -1.6018486e+01 -1.2018485e+01 -8.0184851e+00 -4.0184855e+00 -1.8485421e-02]]

验证完毕

三、整体代码

import torch
import torch.nn as nn
import numpy as np
 
batch_size = 4
class_num = 6
inputs = torch.randn(batch_size, class_num)
for i in range(batch_size):
    for j in range(class_num):
        inputs[i][j] = (i + 1) * (j + 1)
 
print("inputs:", inputs)
Softmax = nn.Softmax(dim=1)
probs = Softmax(inputs)
print("probs:\n", probs)
 
LogSoftmax = nn.LogSoftmax(dim=1)
log_probs = LogSoftmax(inputs)
print("log_probs:\n", log_probs)
 
# probs_sum in dim=1
probs_sum = [0 for i in range(batch_size)]
 
for i in range(batch_size):
    for j in range(class_num):
        probs_sum[i] += probs[i][j]
    print(i, "row probs sum:", probs_sum[i])
 
# to numpy
np_probs = probs.data.numpy()
print("numpy probs:\n", np_probs)
 
# np.log()
log_np_probs = np.log(np_probs)
print("log numpy probs:\n", log_np_probs)

基于pytorch softmax,logsoftmax 表达

import torch
import numpy as np
input = torch.autograd.Variable(torch.rand(1, 3))

print(input)
print('softmax={}'.format(torch.nn.functional.softmax(input, dim=1)))
print('logsoftmax={}'.format(np.log(torch.nn.functional.softmax(input, dim=1))))

以上为个人经验,希望能给大家一个参考,也希望大家多多支持三水点靠木。

Python 相关文章推荐
python处理csv数据的方法
Mar 11 Python
Python批量重命名同一文件夹下文件的方法
May 25 Python
Python数组遍历的简单实现方法小结
Apr 27 Python
Python 自动刷博客浏览量实例代码
Jun 14 Python
python使用folium库绘制地图点击框
Sep 21 Python
Python中捕获键盘的方式详解
Mar 28 Python
python算法与数据结构之单链表的实现代码
Jun 27 Python
使用python实现滑动验证码功能
Aug 05 Python
Python利用requests模块下载图片实例代码
Aug 12 Python
在python中利用pycharm自定义代码块教程(三步搞定)
Apr 15 Python
keras得到每层的系数方式
Jun 15 Python
总结Python连接CS2000的详细步骤
Jun 23 Python
Pytorch中Softmax与LogSigmoid的对比分析
Jun 05 #Python
Pytorch反向传播中的细节-计算梯度时的默认累加操作
pytorch 梯度NAN异常值的解决方案
Jun 05 #Python
pytorch 权重weight 与 梯度grad 可视化操作
PyTorch 如何检查模型梯度是否可导
python-opencv 中值滤波{cv2.medianBlur(src, ksize)}的用法
解决Pytorch修改预训练模型时遇到key不匹配的情况
Jun 05 #Python
You might like
全国FM电台频率大全 - 1 北京市
2020/03/11 无线电
mysql 的 like 问题,超强毕杀记!!!
2007/01/18 PHP
PHP Session变量不能传送到下一页的解决方法
2009/11/27 PHP
PHP+ACCESS 文章管理程序代码
2010/06/21 PHP
使用PHP实现密保卡功能实现代码<打包下载直接运行>
2011/10/09 PHP
基于PHP文件操作的详解
2013/06/05 PHP
php脚本守护进程原理与实现方法详解
2017/07/20 PHP
推荐11款jQuery开发的复选框和单选框美化插件
2011/08/02 Javascript
回车直接实现点击某按钮的效果即触发单击事件
2014/02/27 Javascript
FF(火狐)浏览器无法执行window.close()解决方案
2014/11/13 Javascript
JavaScript实现在页面间传值的方法
2015/04/07 Javascript
JS替换字符串中空格方法
2015/04/17 Javascript
JQ实现新浪游戏首页幻灯片
2015/07/29 Javascript
详解Angularjs filter过滤器
2016/02/06 Javascript
AngularJS实现的JSONP跨域访问数据传输功能详解
2017/07/20 Javascript
es6在react中的应用代码解析
2017/11/08 Javascript
jquery ztree实现右键收藏功能
2017/11/20 jQuery
AngularJS实现的生成随机数与猜数字大小功能示例
2017/12/25 Javascript
解决vue项目报错webpackJsonp is not defined问题
2018/03/14 Javascript
Layui给数据表格动态添加一行并跳转到添加行所在页的方法
2018/08/20 Javascript
javascript设计模式 ? 观察者模式原理与用法实例分析
2020/04/22 Javascript
JS数组的常用10种方法详解
2020/05/08 Javascript
Element Tooltip 文字提示的使用示例
2020/07/26 Javascript
JavaScript实现雪花飘落效果
2020/12/27 Javascript
关于Python如何避免循环导入问题详解
2017/09/14 Python
Django+Xadmin构建项目的方法步骤
2019/03/06 Python
使用python serial 获取所有的串口名称的实例
2019/07/02 Python
基于Python的微信机器人开发 微信登录和获取好友列表实现解析
2019/08/21 Python
flask实现验证码并验证功能
2019/12/05 Python
Python MOCK SERVER moco模拟接口测试过程解析
2020/04/13 Python
Pycharm操作Git及GitHub的步骤详解
2020/10/27 Python
加拿大票务网站:Ticketmaster加拿大
2017/07/17 全球购物
乌克兰网上珠宝商店:GoldSoveren
2020/03/31 全球购物
广告学专业应届生求职信
2013/10/01 职场文书
高中生自我评价个人范文
2013/11/09 职场文书
62句有关感恩节文案(推荐收藏)
2019/11/28 职场文书