Complete network function \(\hat{y}(x,\theta)\)

We have \(W^{[1]} = \begin{bmatrix} w_{1 1}^{[1]} & w_{1 2}^{[1]}\\ \vdots & \vdots\\ w_{d 1}^{[1]} & w_{d 2}^{[1]} \end{bmatrix} \quad ; \quad\) \(b^{[1]} = \begin{bmatrix} b_1^{[1]}\\ \vdots\\ b_d^{[1]} \end{bmatrix} \quad ; \quad\) \(W^{[2]} = \begin{bmatrix} w_1^{[2]} & \dots & w_d^{[2]} \end{bmatrix} \quad ; \quad\) \(b^{[2]} \in \mathbb{R}\)

We note \(\theta = \left(W^{[1]}, b^{[1]}, W^{[2]}, b^{[2]} \right)\).

We have \(x=(x_1,x_2) \in \mathbb{R}^2\).

Gradients

Followingly, the gradient with respect to \(y[m]\) is given by \(\boxed{\nabla_{y}\mathcal{L} = \frac{y_{pred} - y}{y_{pred}(1-y_{pred})}}\).

where \(\mathbf{1}\) is a matrix of the same size where all coefficients are equal to 1.

We have \(\boxed{\nabla_{W^{[2]}} \mathcal{L} = \nabla_{Z^{[2]}} \mathcal{L} \times \left(A^{[1]}\right)'}\)

We find \(\boxed{\nabla_{A^{[1]}} \mathcal{L} = \left(W^{[2]}\right)' \nabla_{Z^{[2]}} \mathcal{L}}\)

We find that \(\boxed{\nabla_{W^{[1]}} \mathcal{L} = \nabla_{Z^{[1]}} \mathcal{L} \times x}\)

Model training based on different initialisations

The network will be trained repeatedly for different initialisation settings, including different weights, different values of d, and different activation functions g (tanh and ReLU). Then, the error function values obtained from the different initialisation settings are compared. In addition, the resulting plane segmentations for these different initialisation settings will be displayed. These graphs will be displayed in the same order as the associated error function values from the displayed table. The influence of the number of iterations will be observed in the next section.

We start by choosing small weights in \([-0.1 ; 0.1]\) and \(d \in \{2,5,10,50\}\) and \(g \in \{tanh, ReLU\}\).

set.seed(1)
# Values for d
d_values = c(2,5,10,50)

# Intervall for initial weights --> [-0.1 ; 0.1]
k=0.1

loss_results_1 = matrix(NA, nrow=2, ncol=length(d_values), dimnames=list(c("tanh","ReLU")))

plots_1=list() ; i=1
for(g in c("tanh","ReLU")){
for (d in d_values){
	NNet_model = NNet(X=X, y=y, rate=0.01, niter=10000, d=d, g, k)
	loss_results_1[which(c("tanh","ReLU")==g), which(d_values==d)] = loss(NNet_model$y,y)
	plots_1[[i]]=segmentation(NNet_model$w1, NNet_model$b1, NNet_model$w2, NNet_model$b2) +
	theme_void() + theme(legend.position = "none") ; i = i+1
}
}
								

Loss function values

# Table with loss results
kable(loss_results_1, col.names=c("d=2","d=5","d=10","d=50"))
								

Plane segmentations

# Plane segmentations
grid.arrange(grobs=as.list(plots_1), ncol=4)
								

For \(g=tanh\) and \(g=ReLU\) with \(d=2\) and \(d \in \{2;5;10\}\) respectively the neural network provides results that are not very satisfactory. For other initialisations, however, the loss of the network is very low. Graphically, one can also observe the different plane segmentations for the different initialisations of the network. For networks with large loss function values, the plane segmentation seems to be rather linear, which is not very consistent with the data. We would now like to determine whether a different initialisation setting of the weights affects these results. We repeat this procedure, but this time we choose larger weights, more precisely in \([-2;2]\).

set.seed(2)
# Values for d
d_values = c(2,5,10,50)

# Intervall for initial weights --> [-2 ; 2]
k=2

loss_results_2 = matrix(NA, nrow=2, ncol=length(d_values), dimnames=list(c("tanh","ReLU")))

plots_2=list() ; i=1
for(g in c("tanh","ReLU")){
for (d in d_values){
	NNet_model = NNet(X=X, y=y, rate=0.01, niter=10000, d=d, g, k)
	loss_results_2[which(c("tanh","ReLU")==g), which(d_values==d)] = loss(NNet_model$y,y)
	plots_2[[i]]=segmentation(NNet_model$w1, NNet_model$b1, NNet_model$w2, NNet_model$b2) +
	theme_void() + theme(legend.position = "none") ; i = i+1
}
}
								

Loss function values

# Loss function values
kable(loss_results_2, col.names=c("d=2","d=5","d=10","d=50"))
								

Plane segmentations

# Plane segmentations
grid.arrange(grobs=as.list(plots_2), ncol=4)
								

We notice that the results are very similar to the results of the first simulation, but still improved. We can therefore see that the initialization of the weights has an influence on the result. On the other hand, once again the network is not able to provide optimal results for \(d=2\) and \(g=tanh\) and for \(d \in \{2;5\}\) with \(g=ReLU\). But this time for \(g=ReLU\) and \(d=10\), the network loss function is able to approach 0, which was not the case before. One could assume that this could be due to the non-convexity of the cross entropy loss function. A bad initialisation of the weights then prevents the gradient descent from converging to the global optimum. Only a local optimum is then found, which could explain the poor results for some of the initialisation settings tested.

Intermediate display of the loss function value

To be able to display the value of the error function every 100 iterations just add the line if (iter%%100 == 0){…} in the network training algorithm. This line allows to detect if for the n-th iteration n is a multiple of 100. The following code is the previously modified code such that this new NNet_intermed_losses function additionally returns the vector containing the loss function value every 100 iterations.

NNet_intermed_losses = function(X, y, rate, niter, d, g, k){

# Initialisation of weights
w1 = matrix(runif(n=2*d,-k,k),d,2)
b1 = matrix(runif(n=d,-k,k),d,1)
w2 = matrix(runif(n=d,-k,k),1,d)
b2 = runif(n=1,-k,k)

loss_vector=c()  # intermediate loss function values

for(iter in 1:niter){
	
	y_pred = prediction(X,w1,b1,w2,b2,g)
	
	Z1 <- w1 %*% t(X) + b1%*% matrix(1,1,dim(X)[1])
	if (g=="tanh") {A1 <- tanh(Z1)}
	if (g=="ReLU") {A1 <- ReLU(Z1)}
	Z2 <- w2 %*% A1 + b2 * matrix(1,1,dim(X)[1])
	
	# Computation of gradients
	grad_y_pred = (y_pred - y)/(y_pred*(1-y_pred))
	grad_Z2 = as.vector(sigmoid(Z2)*(1-sigmoid(Z2))) * grad_y_pred
	grad_w2 = grad_Z2 %*% t(A1)
	grad_b2 = sum(grad_Z2)
	grad_A1 = t(w2) %*% grad_Z2
	if(g=="tanh"){
	grad_Z1 = grad_A1 * (1-A1^2)}
	if(g=="ReLU"){
	grad_Z1 = grad_A1 * (Z1>0)}
	grad_w1 = grad_Z1 %*% X
	grad_b1 = rowSums(grad_Z1)
	
	# Updating the parameters
	w2 =w2 - rate*grad_w2
	b2 =b2 - rate*grad_b2
	w1 =w1 - rate*grad_w1
	b1 =b1 - rate*grad_b1
	
	if (iter%%100 == 0){loss_vector=c(loss_vector, loss(y_pred,y))}
}
return(list(y=y_pred,w1=w1,b1=b1,w2=w2,b2=b2,loss_vector=loss_vector))
}
								

set.seed(4)
# Number of iterations and values fo d
d_values = c(2,5,10,50)
niter=1000

data = data.frame(matrix(NA, nrow = floor(niter/100), ncol = 0))

# Model training for g=tanh and different values of d
g = "tanh"
for (d in d_values){
NNet_model = NNet_intermed_losses(X=X, y=y, rate=0.01, niter, d, g, k=1)
data = cbind(data,NNet_model$loss_vector)
}

# Model training for g=ReLU and different values of d
g = "ReLU"
for (d in d_values){
NNet_model = NNet_intermed_losses(X=X, y=y, rate=0.01, niter, d, g, k=1)
data = cbind(data,NNet_model$loss_vector)
}

# Graphic of loss function value training history
par(xpd = T, mar = par()$mar + c(0,0,0,7))
plot(-10,-10,xlim=c(0,niter),ylim=c(min(data),max(data)),
	xlab = "Itérations", ylab= "Erreur")
lines(100*(1:dim(data)[1]), data[,1], col="green")
lines(100*(1:dim(data)[1]), data[,2], col="red")
lines(100*(1:dim(data)[1]), data[,3], col="blue")
lines(100*(1:dim(data)[1]), data[,4], col="black")
lines(100*(1:dim(data)[1]), data[,5], col="green",lty=2)
lines(100*(1:dim(data)[1]), data[,6], col="red",lty=2)
lines(100*(1:dim(data)[1]), data[,7], col="blue",lty=2)
lines(100*(1:dim(data)[1]), data[,8], col="black",lty=2)
legend("topright", inset=c(-0.2,0), legend=d_values, col=c("green","red","blue","black"),pch=20, title="d")
legend("topright", inset=c(-0.3,0.5), legend=c("tanh","ReLU"), lty=c(1,2))
								

par(mar=c(5, 4, 4, 2) + 0.1)
								

It is clear, as expected, that in general, performance is better if d is large. On the other hand, it is difficult to determine whether to prefer the tanh or ReLU function as a choice for the g activation function, as it strongly depends on the value of d. While the ReLU function gives better results for large values of d (\(d=50\)), it is strongly advised to prefer \(g=tanh\) if d is small (\(\leq 5\)). However, if \(d=2\) the network loss function value does not approach 0 for both activation functions.