Skip to content
GitLab
Menu
Projects
Groups
Snippets
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
Nicolas Schuler
MockupFFNN
Commits
4156233f
Commit
4156233f
authored
Jul 16, 2021
by
Nicolas Schuler
Browse files
Update MockupFFNN.py
parent
0f8dd845
Changes
1
Hide whitespace changes
Inline
Side-by-side
MockupFFNN.py
View file @
4156233f
...
...
@@ -17,7 +17,7 @@ Since the neurons are always the same number and feeding all the other neurons,
the inputData used must have the same dimension as the number of neurons.
Contains different activation functions that can be used to transform the
output data of a given layer, before feeding it to the next layer. Note that
output
Data
data of a given layer, before feeding it to the next layer. Note that
in this implementation we use the same function for all layers of a given
network.
...
...
@@ -85,9 +85,9 @@ class Layer:
class
Network
:
"""Mockup of a neural network with layers (see Layer-class).
input
D
ata():
input
_d
ata():
Sets the used inputData data to feed through the network.
Numpy copies the arrays it handles the inputData is unchanged
Numpy copies the arrays it handles
, thus
the inputData is unchanged
through different feeding processes.
Since the number of neurons is the same for each layer and we are
feeding all the other neurons, the inputData used must have the same
...
...
@@ -97,8 +97,8 @@ class Network:
feedData():
Main function of the module. After a neural network has been created
and feed with the correct inputData data we can feed that data through
the different layers. The resulting output, after going through all
layers, can be called with output
D
ata()
the different layers. The resulting output
Data
, after going through all
layers, can be called with output
_d
ata()
activateOutput():
Applies the given activation function to the provided
...
...
@@ -113,33 +113,33 @@ class Network:
"""
def
__init__
(
self
,
numberOfLayers
=
5
,
numberOfNeurons
=
10
):
self
.
input
=
None
self
.
output
=
None
self
.
input
Data
=
None
self
.
output
Data
=
None
self
.
layers
=
[
Layer
(
numberOfNeurons
)
for
_
in
range
(
numberOfLayers
)]
def
input
D
ata
(
self
,
data
):
self
.
input
=
data
def
input
_d
ata
(
self
,
data
):
self
.
input
Data
=
data
def
output
D
ata
(
self
):
return
self
.
output
def
output
_d
ata
(
self
):
return
self
.
output
Data
def
feedData
(
self
,
activationFunction
=
"Sigmoid"
):
inputData
=
np
.
array
(
self
.
input
,
copy
=
True
,
dtype
=
"float64"
)
inputData
=
np
.
array
(
self
.
input
Data
,
copy
=
True
,
dtype
=
"float64"
)
for
layer
in
self
.
layers
:
output
=
np
.
array
(
range
(
10
),
dtype
=
"float64"
)
for
neuron
in
range
(
10
):
# Multiply inputData with weights via dot product, then add bias of the given neuron
# Finally, calculate output value with the chosen activation function (exception: softmax)
output
[
neuron
]
=
self
.
activateOutput
(
np
.
dot
(
inputData
,
layer
.
weights
[
neuron
]
+
layer
.
biases
[
neuron
]
)
,
activationFunction
)
# Finally, calculate output
Data
value with the chosen activation function (exception: softmax)
output
[
neuron
]
=
self
.
activateOutput
(
np
.
dot
(
inputData
,
layer
.
weights
[
neuron
]
)
+
layer
.
biases
[
neuron
],
activationFunction
)
# For softmax we calculate the output in the end, since we need the entire output
# For softmax we calculate the output
Data
in the end, since we need the entire output
Data
if
(
activationFunction
==
"Softmax"
):
output
=
softmax
(
output
)
inputData
=
output
self
.
output
=
output
self
.
output
Data
=
output
def
activateOutput
(
self
,
x
,
activationFunction
):
if
(
activationFunction
==
"Identity"
):
return
identity
(
x
)
...
...
@@ -170,33 +170,33 @@ if __name__ == "__main__":
network2
=
Network
(
10
,
10
)
inputData
=
np
.
array
(
range
(
10
),
dtype
=
"float64"
)
network
.
input
D
ata
(
inputData
)
network2
.
input
D
ata
(
inputData
)
network
.
input
_d
ata
(
inputData
)
network2
.
input
_d
ata
(
inputData
)
print
(
"Input"
)
print
(
network
.
input
)
print
(
network
.
input
Data
)
network
.
feedData
()
print
(
"
\n
Identity"
)
network
.
feedData
(
activationFunction
=
"Identity"
)
print
(
network
.
output
D
ata
())
print
(
network
.
output
_d
ata
())
print
(
"
\n
RectLinUnit"
)
network
.
feedData
(
activationFunction
=
"RectLinUnit"
)
print
(
network
.
output
D
ata
())
print
(
network
.
output
_d
ata
())
print
(
"
\n
Sigmoid"
)
network
.
feedData
(
activationFunction
=
"Sigmoid"
)
print
(
network
.
output
D
ata
())
print
(
network
.
output
_d
ata
())
print
(
"
\n
Softmax"
)
network
.
feedData
(
activationFunction
=
"Softmax"
)
print
(
network
.
output
D
ata
())
print
(
network
.
output
_d
ata
())
print
(
"
\n
Softmax"
)
network2
.
feedData
(
activationFunction
=
"Softmax"
)
print
(
network2
.
output
D
ata
())
print
(
network2
.
output
_d
ata
())
####################
# Exercise Answers #
...
...
@@ -207,17 +207,17 @@ What effect do different activation functions have?
Since I only used positive values (between 0 and 1) as weights/biases, the
behavior of Identity and RectLinUnit are identical. Both functions keep
the output (effectively, since we have no negative values), thus each
consecutive layer has a larger and larger output.
the output
Data
(effectively, since we have no negative values), thus each
consecutive layer has a larger and larger output
Data
.
Sigmoid converts towards 1 (for x --> infinity). Since, again, we only operate
with positive values the relatively large values will convert towards 1. That is
true for every iteration, so the values of each layer will always be squashed to
1 and then be given to the next layer.
Finally, Softmax is calculated over the entire output. Like Sigmoid, its range is
between 0 and 1. But since the sum of the entire output (vector) for Softmax is = 1
we are always way below the 1 for the individual values. Example output after
Finally, Softmax is calculated over the entire output
Data
. Like Sigmoid, its range is
between 0 and 1. But since the sum of the entire output
Data
(vector) for Softmax is = 1
we are always way below the 1 for the individual values. Example output
Data
after
feeding: [0.10320238 0.16048054 0.07592815 0.09503466 0.13095142 0.07203039 0.11736508
0.06868714 0.08664903 0.08967121] = 1
...
...
@@ -228,7 +228,7 @@ How did you initialize your weights and biases?
In the given exercise, the module does not learn/improve, so I just initialized the weights
and biases with random values between 0 and 1. I also did not initialize with only 0, since
then both weights and values would give the same output, regardless of inputData (since there is
then both weights and values would give the same output
Data
, regardless of inputData (since there is
no learning).
To my knowledge, neither method is ideal (random or zero), since especially the random
...
...
@@ -237,6 +237,6 @@ functions like Sigmoid, which squash the given values in a small range (of [0,1]
On the other hand, RectLU can counter this (since we have no squashing down, as seen above).
So ideally, you would like to initialize the values of the weights in a way, that minimizes the
danger for something like that to occur. While I'm aware of th
is fac
t, I do not know what the
danger for something like that to occur. While I'm aware of th
a
t, I do not know what the
proper way would be to do so yet.
"""
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment