Deep Learning from Zero to Kang
Notes on the building blocks of deep learning, written while I was learning them: activation functions, losses, sequence models, vision backbones and positional embeddings, plus my earlier machine-learning study notes. Every post has an English and a Chinese version. Older posts also live on Zhihu.我在学习深度学习时写下的笔记：激活函数、损失函数、序列模型、视觉骨干网络和位置编码，以及更早的机器学习自学笔记。每篇文章都有中文和英文两个版本。更早的文章也可以在知乎上看到。
Activation functions激活函数7
SigmoidSigmoid 激活函数
The Sigmoid function, also known as the logistic function, is one of the earliest activation functions used in neural networks. However, due to some i…Sigmoid函数，又称logistic函数，是最早使用的激活函数之一。但是由于其固有存在的一些缺点，如今很少将其作为激活函数，但是依然常用于二分类问题中的概率…
TanhTanh 激活函数
The hyperbolic tangent function (tanh) is closely related to the sigmoid function, serving as an amplified version of it. In practical usage, the tanh…双曲正切1函数(tanh)，其图像与sigmoid函数十分相近，相当于sigmoid函数的放大版。在实际的使用中，tanh函数要优先于sigmoid函数。
ReLUReLU 激活函数
Rectified Linear Unit (ReLU), often abbreviated as ReLU, is the most commonly used activation function in modern neural networks, with the majority of…整流线性单元(ReLU)，全称Rectified linear unit，是现代神经网络中最常用的激活函数，大多数前馈神经网络都默认使用该激活函数。
Leaky ReLULeaky ReLU 激活函数
The Leaky ReLU activation function was introduced to address the "dead ReLU" problem, where neurons die due to the gradient being zero for x < 0 in th…Leaky ReLU激活函数是为了解决ReLU激活函数会出现的dead relu神经元死亡的现象，而这一现象的根本原因是ReLU函数在x<0的范围内梯度恒为0，…
PReLUPReLU 激活函数
Parametric Rectified Linear Unit (PReLU) is designed to address the dead ReLU problem by introducing a learnable parameter to control the activation f…参数整流线性激活函数(PReLU)，全称Parametric ReLU，在解决ReLU激活函数会出现的dead relu神经元死亡的现象的同时，引入一个新的可学…
RReLURReLU 激活函数
Randomized Rectified Linear Unit (RReLU) is an extension of ReLU designed to address the dead ReLU problem by introducing a random factor.随机修正整流线性单元(RReLU)，全称Randomized ReLU，是ReLU的延伸版，也是为了解决dead relu神经元坏死现象而提出的一种激活函数。
ELUELU 激活函数
Exponential Linear Units (ELU) incorporate an exponential component into the activation function.指数线性单元(ELU)，全称Exponential Linear Units，使用了指数作为激活函数的一部分。
Loss functions损失函数3
NLL LossNLL 损失函数
Negative Log-Likelihood Loss (NLL Loss) measures the difference between predicted and true distributions. In some tasks, we aim not to predict a speci…NLL损失函数的全称是负对数似然损失（Negative Log-Likelihood Loss），是用来衡量预测分布与真实分布的一种损失函数。在有些任务中，我们…
Bivariate Gaussian NLL Loss双变量高斯负对数似然损失
BiNLL Loss stands for Bivariate Gaussian Negative Log-Likelihood Loss. In the blog post on the single variable Gaussian negative log-likelihood loss,…BiNLL Loss是Bivariate Gaussian Negative Log-Likelihood Loss的缩写。在单变量高斯负对数似然损失函数那篇博…
NLLLoss, KLDivLoss and CrossEntropyLossNLLLoss、KLDivLoss 与 CrossEntropyLoss 对比
These three functions are very common in deep learning models, especially in the field of knowledge distillation, where these three functions are ofte…这三个函数在深度学习模型中十分常见，尤其是在知识蒸馏领域，经常会将这三个函数进行比较
Sequence models序列模型2
Recurrent Neural Networks (RNN)循环神经网络（RNN）
For example, let's consider the scenario of using neural networks to process pedestrian motion trajectories. If we were to use a CNN, the convolutiona…如果我们用 FNN 前馈神经网络去处理，那么我们需要让每个权重自己去学习参数，很容易忽略权重之间的关联性；如果我们用 CNN 卷积神经网络去处理，那么对于这个人…
Long Short-Term Memory (LSTM)长短期记忆网络（LSTM）
The design of LSTM addresses this issue by enhancing the neural network's ability to remember earlier input data. However, enabling the neural network…相信大伙在网上也看过 LSTM 内部各种门的各种各样的画法，这里我自己画了一种觉得方便理解。 LSTM 将“短期记忆线” h{t-1} 和正常的输入数据 x{t…
Vision backbones视觉骨干网络4
LeNetLeNet
LeNet, proposed by the deep learning pioneer Yann LeCun in 1998, can be considered as the first successful application of multi-layer convolutional ne…LeNet是是由深度学习巨头Yann LeCun在1998年提出，可以算作多层卷积网络在图像识别领域的首次成功应用。我们现在通常说的LeNet是指LeNet-5…
AlexNetAlexNet
Additionally, AlexNet highlighted the importance of neural network depth for model effectiveness and demonstrated the significant acceleration possibl…AlexNet 包含八层：前五层是卷积层，其中一些是最大池化层，后三层是全连接层。除最后一层外，网络被拆分为两个部分，每个部分在一个 GPU 上运行。整个结构可…
VGGVGG
It's been a decade since the introduction of the VGG network, and it's amazing to think about a model that has thrived for ten years in the rapidly ev…距离VGG网络的提出已经约十年，很难想象在深度学习高速发展的今天，一个模型能够历经十年而不衰。虽然如今已经有VGG的大量替代品，但是笔者研究的一些领域仍然有大量…
GoogLeNetGoogLeNet
GoogleNet won the ImageNet competition in 2014, bringing the Inception architecture into the spotlight. From another perspective, it inspired and adva…GoogleNet在2014年中的ImageNet夺冠，将Inception这一结构推向了热潮。从另外一个角度来看，CV魔改网络结构也从中得到启发或者说推动，拓…
Positional embeddings位置编码2
Sinusoidal Position Embeddings正弦位置编码
In the past, I worked on some sequence processing tasks and briefly experimented with Transformers, but didn't delve into the details. Recently, while…下面开一期新坑，关于位置编码的一系列介绍。之前做过时序处理的一些问题，草草调包用过transformer，但是并没有对其细节进行了解。最近在琢磨扩散模型DDPM…
Learned Sinusoidal Position Embeddings可学习的正弦位置编码
In the blog post on Sinusoidal Positional Embeds, the basic concept and implementation of Sinusoidal Position Embeds were introduced. Here, I'd like t…在位置编码那篇博客中，介绍了位置编码的基本思想以及Sinusoidal Position Embeds的实现。这里要介绍一下其中的一个变体：Random Or…
Machine-learning study notes机器学习自学笔记33
From my undergraduate years, originally written in Chinese.本科阶段的自学笔记。