Variable tensors are used when we expect the values to require updating during the course of a session. This is, therefore, what we would use for creating the parameters like the weights and biases because these values will be updated as we train our model.
Declaring them in the graph is just a matter of making use of the tf.Variable() function, eg:
tf_a = tf.Variable(tf.ones([2,3], dtype=tf.float32))
There is a quirky feature about them that makes them behave a little differently from constant tensors. When declaring a constant tensor in a graph, the values are initialized automatically. In the above code, it may appear as if we have also initialized the variable tensor as a 2 by 3 matrix populated with ones. However, tensorflow interprets this as something like: "Hey!, we want to create a tensor, please make some room for it, but don't actually initialize it just yet, just store how we would want to initialize it when we get round to it."
So if we run the following code, we get an error that says:
FailedPreconditionError: Attempting to use uninitialized value Variable
Run it to see for yourself.
graph = tf.Graph() with graph.as_default(): tf_a = tf.Variable(tf.ones([2,3], dtype=tf.float32)) with tf.Session(graph=graph) as session: a = session.run(tf_a) print(a)
In order to avoid this, we need to explicitly initialize the variables everytime we start a new session. We make use of the session.run(tf.global_variables_initializer()) function to initialize all the variables in the graph. So, now if we run the following code, things work smoothly
graph = tf.Graph() with graph.as_default(): tf_a = tf.Variable(tf.ones([2,3], dtype=tf.float32)) with tf.Session(graph=graph) as session: session.run(tf.global_variables_initializer()) a = session.run(tf_a) print(a)
[OUTPUT]
[[ 1. 1. 1.]
[ 1. 1. 1.]]
You can pass any tensorflow constant, or numpy array as the first argument to a variable tensor, and that will be the value that it gets initialized to. So any of the functions mentioned in the previous section about constants can be passed to it.
The values you chose to initialize the weights in a neural network can have a big effect on how well it learns. One very common method for choosing those values is called the Xavier initialization. It initializes the weights using random values but normalized based on the number of nodes. We can make use of this technique by first creating an object that will generate the weights for us.
weights_initializer = tf.contrib.layers.xavier_initializer(uniform=False, dtype=tf.float32)
Setting the uniform argument to True makes it use random uniform values. Setting it to False makes it use random normal values.
Putting it into a full example, looks something like this:
graph = tf.Graph() with graph.as_default(): weights_initializer = tf.contrib.layers.xavier_initializer(uniform=True, dtype=tf.float32) tf_a = tf.Variable(weights_initializer(shape=[2,3]), dtype=tf.float32) with tf.Session(graph=graph) as session: session.run(tf.global_variables_initializer()) a = session.run(tf_a) print(a)
[OUTPUT]
[[-0.81394684 0.64682126 0.95366645]
[-0.60882092 1.06681776 -0.90973783]]
The values of variable tensors change over time. In the discussions above we initialized the variables to their default values. But, we could have specified that we want to use values from a previously saved session instead, or from a pre-trained model we got off the internet instead. You will see how to initialize variables this way in a future section about saving and restoring.