• How to Develop a 1D Generative Adversarial Network From Scratch in Keras
  • Originally by Jason Brownlee
  • The Nuggets translation Project
  • Permanent link to this article: github.com/xitu/gold-m…
  • Translator: TokenJan
  • Proofreader: Haiyang-Tju, TodayCoder001

How to build a one-dimensional generative adversarial network from scratch with Keras

Generative adversarial networks, or GANs for short, are a deep learning framework for training powerful generator models.

Generator models can be used to generate new spurious samples, most likely from existing sample distributions.

GANs consists of a generator model and a discriminator model. The generator is responsible for generating new samples from the domain, and the discriminator is responsible for sensing the authenticity (generated) of these samples. Importantly, the performance of the discriminator model is used to update the model weights of the discriminator itself and the generator. This means that the generator is not aware of samples from the domain, but makes adjustments based on the performance of the discriminator.

It’s a complex model to understand and train.

One way to better understand the nature of GAN models and how to train them is to build a model from scratch based on simple tasks.

The simple task of one-dimensional functions provides a good environment for building a simple GAN from scratch. This is because both real and generated samples can be plotted and visualized to check what has been learned. A simple function also does not require a complex neural network model, which means that the use of specific generators and discriminators in the architecture can be easily understood.

In this tutorial, we will build and evaluate a generative adversarial network from scratch using the Keras deep learning library based on a simple one-dimensional function.

After completing this tutorial, you will have learned:

  • The benefits of constructing a generative adversarial network from scratch using a simple one-dimensional function.
  • How to build independent discriminator and generator models, and a composite model to train generators by discriminator predictive behavior.
  • How to generate samples subjectively in the context of real data in the problem domain.

In my new GANs book, you can find how to build DCGANs, Conditional GANs, Pix2Pix, CycleGANs and more, along with 29 step-by-step tutorials and complete source code.

Let’s get started.

How to Construct a one-dimensional function GAN from Scratch with Keras This photograph was taken by Chris Bambrick, right reserved.

Tutorial overview

This tutorial is divided into six parts, which are:

  1. Pick a one-dimensional function
  2. Define a discriminator model
  3. Define a generator model
  4. Training generator model
  5. Evaluate the performance of GAN
  6. A complete example of training GAN

Pick a one-dimensional function

The first step is to choose a one-dimensional function to model.

The function looks like:

y = f(x)
Copy the code

Where x and y are the input and output values of the function.

In particular, we need a function that is easy to understand and draw. This will help in setting expectations for what the model should generate and in visually examining the generated samples to understand their quality.

We’re going to use a simple function x^2; This function returns the square of the input value. You may remember this function from high school algebra, which is a U-shaped function.

We can define this function in Python like this:

# Simple function
def calculate(x):
	return x * x
Copy the code

We can define the input fields as real numbers between -0.5 and 0.5, calculate the output values for each input value in the linear range, and then plot the results to see how the inputs and outputs relate.

The complete example is below.

# Demonstrate a simple x^2 function
from matplotlib import pyplot

# Simple function
def calculate(x):
	return x * x

# define input values
inputs = [0.5.0.4.0.3.0.2.0.1.0.0.1.0.2.0.3.0.4.0.5]
# calculate the output value
outputs = [calculate(x) for x in inputs]
# draw result
pyplot.plot(inputs, outputs)
pyplot.show()
Copy the code

Run the example to calculate the output value for each input value and draw a graph of the input and output values.

We can see that values farther away from 0 give a larger output, and values closer to 0 give a smaller output, and that the behavior is symmetric with respect to the origin.

This is the u-shaped diagram of the famous one-dimensional function X^2.

Input and output diagram of the X^2 function.

We can randomly generate samples or points from this function.

This can be done by generating a random value between -0.5 and 0.5 and calculating its corresponding output value. Repeat this step several times to get a sample point for the function, such as a “real sample.”

Plotting these samples with a scatter plot will show the same U-shaped plot, even though these are made up of separate random samples.

The complete example is described below.

First, we generate random values evenly between 0 and 1, then offset them to the range of -0.5 and 0.5. We then compute the corresponding output value for each randomly generated input value and combine these matrices into a single Numpy array with n rows (100) and 2 columns.

Example of generating a random sample from X^2
from numpy.random import rand
from numpy import hstack
from matplotlib import pyplot

Generate a random sample of x squared
def generate_samples(n=100):
	Generate random input between [-0.5, 0.5]
	X1 = rand(n) - 0.5
	# Generates X^2 (quadratic) output
	X2 = X1 * X1
	# stack array
	X1 = X1.reshape(n, 1)
	X2 = X2.reshape(n, 1)
	return hstack((X1, X2))

# generate sample
data = generate_samples()
# draw sample
pyplot.scatter(data[:, 0], data[:, 1])
pyplot.show()
Copy the code

Running this example produces 100 random inputs, computed output, and a scatter plot of the sample, a familiar U-shaped plot.

Draws a randomly generated input sample and computed output value for the X^2 function.

We can use this function as a starting point for generating real samples for the discriminator function. In particular, a sample is made up of a vector of two elements, one as an input and one as an output of our one-dimensional function.

We can also imagine how a generator model generates new samples that we can plot and compare to the desired U-shaped X^2 function. In particular, a generator can output a vector consisting of two elements: one as input and one as output of a one-dimensional function.

Define a discriminator model

The next step is to define the discriminator model.

The model must take a sample from our problem domain, such as a two-element vector, and output a classification prediction to distinguish between the true and false samples.

This is a dichotomous problem.

  • Input: a sample of two real numbers.
  • Output: dichotomous, the probability that the sample is true (or false).

This problem is very simple, which means we don’t need a complex neural network to model it.

The discriminator model has a hidden layer containing 25 neurons, using ReLU activation function and He initialization with appropriate weights.

The output layer contains a neuron that performs dichotomies using the Sigmoid activation function.

This model will minimize the bisection cross entropy loss function, as well as the Adam version of stochastic gradient descent, because it is very efficient.

The define_discriminator() function below defines and returns the discriminator model. This function parameterizes the expected number of inputs. The default value is 2.

Define a separate discriminator model
def define_discriminator(n_inputs=2):
	model = Sequential()
	model.add(Dense(25, activation='relu', kernel_initializer='he_uniform', input_dim=n_inputs))
	model.add(Dense(1, activation='sigmoid'))
	# build model
	model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
	return model
Copy the code

We can use this function to define and summarize the discriminator model. The complete example is shown below.

Define the discriminator model
from keras.models import Sequential
from keras.layers import Dense
from keras.utils.vis_utils import plot_model

Define a separate discriminator model
def define_discriminator(n_inputs=2):
	model = Sequential()
	model.add(Dense(25, activation='relu', kernel_initializer='he_uniform', input_dim=n_inputs))
	model.add(Dense(1, activation='sigmoid'))
	# build model
	model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
	return model

# Define the discriminant model
model = define_discriminator()
# Summary model
model.summary()
# Draw the model
plot_model(model, to_file='discriminator_plot.png', show_shapes=True, show_layer_names=True)
Copy the code

Run this example, which defines and summarizes the discriminator model.

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense_1 (Dense)              (None, 25)                75
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 26
=================================================================
Total params: 101
Trainable params: 101
Non-trainable params: 0
_________________________________________________________________
Copy the code

A graph of the model is also generated, and you can see that the model has two inputs and one output.

Note: The PyDot and Graphviz libraries were installed to generate this model diagram. If the installation is having problems, you can comment out the import statement that introduced the plot_model function and the call to the plot_model method.

Generate generator model graph in adversarial network

Now you can train the model with real data labeled 1 and randomly generated data labeled 0.

We don’t need to do this, but the elements we developed will be helpful later, and it helps us realize that generators are just a generic neural network model.

First, we can update our generate_samples() method from the predicted part, named generate_real_samples(), which will return the output label of the real sample, which is an array of ones, where 1 represents the real sample.

# Generate n real samples and classification tags
def generate_real_samples(n):
	Generate input values in the range [-0.5, 0.5]
	X1 = rand(n) - 0.5
	# Generate the output value X^2
	X2 = X1 * X1
	# stack array
	X1 = X1.reshape(n, 1)
	X2 = X2.reshape(n, 1)
	X = hstack((X1, X2))
	Generate category tags
	y = ones((n, 1))
	return X, y
Copy the code

Next, we can create a copy of this method to generate a dummy sample.

In this case, we generate random values between -1 and 1 for the two elements of the sample. All of these samples have an output classification label of 0.

This method will serve as the dummy data generator model.

Generate n plus sample and category tags
def generate_fake_samples(n):
	Generate input values in the range [-1, 1]
	X1 = - 1 + rand(n) * 2
	Generate output values in the range [-1, 1]
	X2 = - 1 + rand(n) * 2
	# stack array
	X1 = X1.reshape(n, 1)
	X2 = X2.reshape(n, 1)
	X = hstack((X1, X2))
	Generate category tags
	y = zeros((n, 1))
	return X, y
Copy the code

Next, we need a way to train and evaluate the generator model.

This can be done by manually iterating through the trained epochs, generating half of the real samples and half of the fake samples within each epoch, and then updating the model over the entire batch. The train() method can be used for training, but in this case we will use the train_on_batch() method directly.

This model can be evaluated according to the generated samples, and we can generate reports on the classification accuracy of true and false samples.

The train_discriminator() method implements the training of 1000 batches for a model, with each batch containing 128 samples (64 false samples and 64 true samples).

# Train the discriminator model
def train_discriminator(model, n_epochs=1000, n_batch=128):
	half_batch = int(n_batch / 2)
	Run the epoch manually
	for i in range(n_epochs):
		# Generate real samples
		X_real, y_real = generate_real_samples(half_batch)
		# Update model
		model.train_on_batch(X_real, y_real)
		Generate false samples
		X_fake, y_fake = generate_fake_samples(half_batch)
		# Update model
		model.train_on_batch(X_fake, y_fake)
		# Evaluation model
		_, acc_real = model.evaluate(X_real, y_real, verbose=0)
		_, acc_fake = model.evaluate(X_fake, y_fake, verbose=0)
		print(i, acc_real, acc_fake)
Copy the code

We can link these together and then train the discriminator model on real and fake samples.

The complete example is shown below.

Define and load a discriminator model
from numpy import zeros
from numpy import ones
from numpy import hstack
from numpy.random import rand
from numpy.random import randn
from keras.models import Sequential
from keras.layers import Dense

Define a separate discriminator model
def define_discriminator(n_inputs=2):
	model = Sequential()
	model.add(Dense(25, activation='relu', kernel_initializer='he_uniform', input_dim=n_inputs))
	model.add(Dense(1, activation='sigmoid'))
	# build model
	model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
	return model

# Generate n real samples and classification tags
def generate_real_samples(n):
	Generate input values in the range [-0.5, 0.5]
	X1 = rand(n) - 0.5
	# Generate output X^2
	X2 = X1 * X1
	# stack array
	X1 = X1.reshape(n, 1)
	X2 = X2.reshape(n, 1)
	X = hstack((X1, X2))
	Generate category tags
	y = ones((n, 1))
	return X, y

Generate n false samples and classification tags
def generate_fake_samples(n):
	Generate input values in the range [-1, 1]
	X1 = - 1 + rand(n) * 2
	Generate output values in the range [-1, 1]
	X2 = - 1 + rand(n) * 2
	# stack array
	X1 = X1.reshape(n, 1)
	X2 = X2.reshape(n, 1)
	X = hstack((X1, X2))
	Generate category tags
	y = zeros((n, 1))
	return X, y

# Train the discriminator model
def train_discriminator(model, n_epochs=1000, n_batch=128):
	half_batch = int(n_batch / 2)
	Run the epoch manually
	for i in range(n_epochs):
		# Generate real samples
		X_real, y_real = generate_real_samples(half_batch)
		# Update model
		model.train_on_batch(X_real, y_real)
		Generate false samples
		X_fake, y_fake = generate_fake_samples(half_batch)
		# Update model
		model.train_on_batch(X_fake, y_fake)
		# Evaluation model
		_, acc_real = model.evaluate(X_real, y_real, verbose=0)
		_, acc_fake = model.evaluate(X_fake, y_fake, verbose=0)
		print(i, acc_real, acc_fake)

Define the discriminator model
model = define_discriminator()
# Load model
train_discriminator(model)
Copy the code

Running the above code generates real and fake samples and updates the model, then evaluates the model on the same samples and prints out the accuracy of the classification.

The results may vary but the model learns quickly, correctly identifies real samples with perfect accuracy, and is very good at identifying fake samples with between 80% and 90% accuracy.

. 995 1.0 0.875 996 1.0 0.921875 997 1.0 0.859375 998 1.0 0.9375 999 1.0 0.8125Copy the code

The process of training the discriminator model is straightforward. Our goal is to train a generator model, not a discriminator model, which is where generating GANs gets really complicated.

Define a generator model

The next step is to define the generator model.

The generator model takes a point from hidden space as an input and generates a new sample, such as the input and output elements of a function as a vector, such as x and x^2.

A hidden variable is a hidden or unobserved variable, and a hidden space is a multidimensional vector space composed of these variables. We can define the dimension size of the hidden space and its shape or distribution of variables for the problem.

Hidden Spaces are meaningless until the generator model begins to learn and assign meaning to points in the space. After training, points in the hidden space will be associated with points in the output space, such as generating sample space.

We define a small five-dimensional hidden space and use the method of generating a standard adversarial network literature, that is, every variable in the hidden space uses a Gaussian distribution. We will get random numbers from a standard Gaussian distribution to generate input values, such as mean 0 and standard deviation 1.

  • Input: points in hidden space, such as a vector composed of five random Gaussian numbers.
  • Output: A vector of two elements representing the samples (x and x^2) generated for our function.

The generator model will be as small as the discriminator model.

It has only one hidden layer with five neurons, using ReLU activation function and He weight initialization method. The output layer has two neurons representing two elements of the spanning vector and uses a linear activation function.

The last reason to use the linear activation function is that you want the generator to print a vector of real numbers, with the first element in the range [-0.5, 0.5] and the second element in the range [0.0, 0.25].

The model is not compiled. The reason is that the generator model is not loaded directly.

The define_generator() method below defines and returns the generator model.

The dimensionality of the hidden space is parameterized in case it needs to be changed later, and the dimensionality of the output of the model is also parameterized, matching the function of the defined discriminator model.

Define a separate generator model
def define_generator(latent_dim, n_outputs=2):
	model = Sequential()
	model.add(Dense(15, activation='relu', kernel_initializer='he_uniform', input_dim=latent_dim))
	model.add(Dense(n_outputs, activation='linear'))
	return model
Copy the code

We can summarize this model to help better understand input and output shaping.

The complete example is shown below.

Define the generator model
from keras.models import Sequential
from keras.layers import Dense
from keras.utils.vis_utils import plot_model

Define a separate generator model
def define_generator(latent_dim, n_outputs=2):
	model = Sequential()
	model.add(Dense(15, activation='relu', kernel_initializer='he_uniform', input_dim=latent_dim))
	model.add(Dense(n_outputs, activation='linear'))
	return model

Define the generator model
model = define_generator(5)
# Summary model
model.summary()
# Draw the model
plot_model(model, to_file='generator_plot.png', show_shapes=True, show_layer_names=True)
Copy the code

Run this example, which defines and summarizes the generator model.

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense_1 (Dense)              (None, 15)                90
_________________________________________________________________
dense_2 (Dense)              (None, 2)                 32
=================================================================
Total params: 122
Trainable params: 122
Non-trainable params: 0
_________________________________________________________________
Copy the code

The model graph is also generated, and we can see that the model expects a five-vector input from hidden space and predicts a two-vector output.

Note: The PyDot and Graphviz libraries were installed to generate this model diagram. If the installation is having problems, you can comment out the import statement that introduced the plot_model function and the call to the plot_model method.

Draws a generator model in a generative adversarial network

We can see that the model takes a random five-vector from the hidden space and outputs a two-vector for the one-dimensional function.

The model doesn’t do much yet. However, we can use it to demonstrate how to use it to generate samples. This is not necessary, but again, some of these elements might be useful later.

The first step is to generate new points in hidden space. We can generate a random array of numbers from the standard Gaussian distribution by calling the Randn () NumPy method.

The random number array can then be scaled to the size of the sample dimension: that’s n rows with five elements per row. The generate_latent_points() method below implements this and generates a number of points in the hidden space that can be used as inputs to the generated model.

# Generating points in hidden space as input to the generator
def generate_latent_points(latent_dim, n):
	# Spawn in hidden space
	x_input = randn(latent_dim * n)
	Resize the dimension of the batch input sample for the network
	x_input = x_input.reshape(n, latent_dim)
	return x_input
Copy the code

Next, we can use these generated points as input to the generator model to generate new samples, which we can then draw.

The generate_fake_samples() method below implements this, passing in the dimensions of the defined generator and hidden space and the number of model generation points as parameters.

Use the generator to generate n false samples and then draw the result
def generate_fake_samples(generator, latent_dim, n):
	# Spawn in hidden space
	x_input = generate_latent_points(latent_dim, n)
	# Forecast output
	X = generator.predict(x_input)
	# draw result
	pyplot.scatter(X[:, 0], X[:, 1])
	pyplot.show()
Copy the code

Put them together, the full example is shown below.

Define and use the generator model
from numpy.random import randn
from keras.models import Sequential
from keras.layers import Dense
from matplotlib import pyplot

Define a separate generator model
def define_generator(latent_dim, n_outputs=2):
	model = Sequential()
	model.add(Dense(15, activation='relu', kernel_initializer='he_uniform', input_dim=latent_dim))
	model.add(Dense(n_outputs, activation='linear'))
	return model

# Generate points in hidden space as input to the generator
def generate_latent_points(latent_dim, n):
	# Spawn in hidden space
	x_input = randn(latent_dim * n)
	# Adjust the dimension size of network batch input
	x_input = x_input.reshape(n, latent_dim)
	return x_input

The generator generates n false samples to draw the result
def generate_fake_samples(generator, latent_dim, n):
	# Spawn in hidden space
	x_input = generate_latent_points(latent_dim, n)
	# Forecast output
	X = generator.predict(x_input)
	# draw result
	pyplot.scatter(X[:, 0], X[:, 1])
	pyplot.show()

# Dimension size of hidden space
latent_dim = 5
Define the discriminator model
model = define_generator(latent_dim)
Generate and draw the generated sample
generate_fake_samples(model, latent_dim, 100)
Copy the code

Running this example will generate 100 random points from hidden space, use them as input to the generator and generate 100 false samples from our one-dimensional function domain.

Since the generator has not been trained, the resulting points are as “garbage” as we would expect, but we can imagine that as the model is trained, these points will slowly start to move towards the target function and its U-shape.

Scatter plots of pseudosamples predicted by generator models.

We’ve seen how to define and use a generator model. We need to use the generator model in this way to generate samples for the discriminator to classify.

We have not yet seen how the generator model is trained; That’s the next step.

Training generator model

The weights in the generator model are updated based on the representation of the discriminator model.

The generator updates larger when the discriminator is good at detecting false samples, and smaller when the discriminator is relatively bad at detecting false samples or is confused.

This defines the zero-sum or adversarial relationship between the two models.

There are many ways to do this using the Keras API, but perhaps the easiest way is to create a new model that contains or encapsulates the generator and discriminator models.

In particular, a new generation against network model can be defined as stacked discriminant, generator and a generator in the implicit space receive random points as input, directly is provided to generate sample discriminant model samples, and then classification, finally, the big model output can be used to update the weight of the generator model.

To be clear, we are not talking about a new third-party model, just a logically third-party model that uses the layers and weights defined in the separate generator and discriminator model.

Only discriminators are involved in distinguishing between true and false data; Therefore, the discriminator model can be trained separately from true and false data.

The generator model is only concerned with the performance of the discriminator model on bogus data. Therefore, when a discriminator is part of a model that generates adversarial networks, we mark all layers in the discriminator as untrainable, and they do not update parameters on fake data to prevent overtraining.

There is one more important change that needs to be made when training generators with this combined generative adversarial network model. We want the discriminator to think that the sample output from the generator is real, not fake. Therefore, when the generator is training as part of the generative adversarial network, we set the token generated sample to true (class tag 1).

We can imagine that the discriminator classifies the generated sample as either untrue (class label 0) or less likely to be true (0.3 or 0.5). The backpropagation process used to update the model weight treats it as a large error and then updates the model weight (such as only the weight in the generator) to correct the error, which in turn makes the generator better and more reasonable to generate false samples.

Let’s be specific.

  • Input: a point in a hidden space, such as a five-element vector of random Gaussian numbers.
  • Output: dichotomous, the probability that the sample is true (or false).

The define_gan() method below takes the already defined generator and discriminator as parameters and creates a logical third new model containing both models. The weights in the discriminator are marked as untrainable, which only affects the weights in the generative adversarial network, not the independent discriminator model.

The adversarial network model uses the same dichoric cross entropy loss function as the discriminator as well as the efficient Adam version of stochastic gradient descent.

# Define the merged generator and discriminator models for the update generator
def define_gan(generator, discriminator):
	The weight in the tag discriminator is untrainable
	discriminator.trainable = False
	# Connect them
	model = Sequential()
	# Add generator
	model.add(generator)
	Add a discriminator
	model.add(discriminator)
	# build model
	model.compile(loss='binary_crossentropy', optimizer='adam')
	return model
Copy the code

Making the discriminator untrainable is a clever trick in the Keras API.

The Trainable property affects the model when it is compiled. The discriminator models are compiled using trainable layers, so calling train_on_batch() updates the standalone models, as well as the weight models in those layers.

The discriminator model is marked as untrainable, the generative adversarial network model is added, and compiled. In this model, the weights in the discriminator model are untrainable and cannot be changed when updating the generated adversarial network model by calling train_on_batch().

This behavior is described in the Keras API documentation:

  • How can I “freeze” the Keras layer?

Complete examples of creating discriminators, generators, and composite models are listed below.

Demonstrate the creation of three models for generating adversarial networks
from keras.models import Sequential
from keras.layers import Dense
from keras.utils.vis_utils import plot_model

Define a separate discriminator model
def define_discriminator(n_inputs=2):
	model = Sequential()
	model.add(Dense(25, activation='relu', kernel_initializer='he_uniform', input_dim=n_inputs))
	model.add(Dense(1, activation='sigmoid'))
	# build model
	model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
	return model

Define a separate generator model
def define_generator(latent_dim, n_outputs=2):
	model = Sequential()
	model.add(Dense(15, activation='relu', kernel_initializer='he_uniform', input_dim=latent_dim))
	model.add(Dense(n_outputs, activation='linear'))
	return model

# Define the merged generator and discriminator models in order to update the generator
def define_gan(generator, discriminator):
	The weights in the tag discriminator model are untrainable
	discriminator.trainable = False
	# Connect them
	model = Sequential()
	# Add generator
	model.add(generator)
	Add a discriminator
	model.add(discriminator)
	# build model
	model.compile(loss='binary_crossentropy', optimizer='adam')
	return model

# Dimension size of hidden space
latent_dim = 5
Create a discriminator
discriminator = define_discriminator()
# create generator
generator = define_generator(latent_dim)
Create a generative adversarial network
gan_model = define_gan(generator, discriminator)
# Summarize and generate adversarial network model
gan_model.summary()
# Draw and generate adversarial network model
plot_model(gan_model, to_file='gan_plot.png', show_shapes=True, show_layer_names=True)
Copy the code

Running this example first creates a summary of the composite model.

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
sequential_2 (Sequential)    (None, 2)                 122
_________________________________________________________________
sequential_1 (Sequential)    (None, 1)                 101
=================================================================
Total params: 223
Trainable params: 122
Non-trainable params: 101
_________________________________________________________________
Copy the code

The model diagram is also created, and we can see that the model expects a five-point input in hidden space, as well as a classification label that predicts an output.

Note: The PyDot and Graphviz libraries were installed to generate this model diagram. If the installation is having problems, you can comment out the import statement that introduced the plot_model function and the call to the plot_model method.

Generate generator and discriminator combination model graph in adversarial network

The training composition model consists of generating a batch of points in hidden space through the generate_latent_points() method in the previous section and calling the train_on_batch() method with the class=1 tag.

The train_gan() method below demonstrates this process, although it is not very interesting because only the generator is updated in each epoch and the discriminator retains the default model weights.

# Train the combination model
def train_gan(gan_model, latent_dim, n_epochs=10000, n_batch=128):
	Manually iterate over the epoch
	for i in range(n_epochs):
		Prepare points in hidden space as input for the generator
		x_gan = generate_latent_points(latent_dim, n_batch)
		# Create a counter tag for the fake sample
		y_gan = ones((n_batch, 1))
		# Error update generator via discriminator
		gan_model.train_on_batch(x_gan, y_gan)
Copy the code

We first need to update the discriminator model with true and false samples, and then update the generator with the composite model.

This requires merging elements from the train_discriminator() method defined in the discriminator with the train_gan() method defined above. The generate_fake_samples() method is also required to use the generator model to generate dummy samples instead of random numbers.

The complete training method for updating the discriminator model and generator (by combining models) is shown below.

Train generators and discriminators
def train(g_model, d_model, gan_model, latent_dim, n_epochs=10000, n_batch=128):
	Half of the batch quantity is used to update the discriminator
	half_batch = int(n_batch / 2)
	Manually iterate over the epoch
	for i in range(n_epochs):
		# Prepare real samples
		x_real, y_real = generate_real_samples(half_batch)
		Prepare fake samples
		x_fake, y_fake = generate_fake_samples(g_model, latent_dim, half_batch)
		Update the discriminator
		d_model.train_on_batch(x_real, y_real)
		d_model.train_on_batch(x_fake, y_fake)
		Prepare points in hidden space as inputs in the generator
		x_gan = generate_latent_points(latent_dim, n_batch)
		# Create a counter tag for the fake sample
		y_gan = ones((n_batch, 1))
		# Error update generator via discriminator
		gan_model.train_on_batch(x_gan, y_gan)
Copy the code

We have almost everything we need to build a generative adversarial network using one-dimensional functions.

The rest is model evaluation.

Evaluate the performance of generating adversarial networks

In general, there is no objective way to evaluate the performance of generative adversarial network models.

In this particular case, we can design an objective measure for the generated sample because we know the potentially true input fields and objective functions, and can calculate an objective error measure.

However, we will not calculate this objective error value in this tutorial. Instead, we will use the subjective approach used in most generative adversarial network applications. In particular, we’ll use generators to generate new samples and then check how they differ from the real samples in the domain.

First, we can use the generate_real_samples() method created earlier in the discriminator section to generate a new sample. Drawing a scatter plot from these samples yields the familiar U-shaped objective function.

Generate n real samples and class tags
def generate_real_samples(n):
	Generate input values in the range [-0.5, 0.5]
	X1 = rand(n) - 0.5
	# Generate the output value X^2
	X2 = X1 * X1
	# stack array
	X1 = X1.reshape(n, 1)
	X2 = X2.reshape(n, 1)
	X = hstack((X1, X2))
	Generate class tags
	y = ones((n, 1))
	return X, y
Copy the code

Next, we can use the generator model to generate the same number of fake samples.

This first requires generating the same number of points in hidden space through the generate_latent_points() method created in the generator section above. These points can be passed into the generator model and generate samples that can be plotted on the same scatter diagram.

# Generating points in hidden space as input to the generator
def generate_latent_points(latent_dim, n):
	# Spawn in hidden space
	x_input = randn(latent_dim * n)
	Adjust a batch input dimension size for the network
	x_input = x_input.reshape(n, latent_dim)
	return x_input
Copy the code

The generate_fake_samples() method below generates these fake samples and the associated class tag 0, which will be useful later.

Generate n false samples and class tags with generator
def generate_fake_samples(generator, latent_dim, n):
	# Spawn in hidden space
	x_input = generate_latent_points(latent_dim, n)
	# Forecast output
	X = generator.predict(x_input)
	Create a class tag
	y = zeros((n, 1))
	return X, y
Copy the code

The two samples are drawn on the same graph so that they can be compared directly by subjectively seeing whether the same input and output fields are covered and whether the desired shape of the objective function is properly drawn.

The summarize_performance() method below, which can be called at any point in time in training, allows you to draw real and generated scatter diagrams to get a sense of the current ability to generate the model.

# draw true and false points
def summarize_performance(generator, latent_dim, n=100):
	# Prepare real samples
	x_real, y_real = generate_real_samples(n)
	Prepare fake samples
	x_fake, y_fake = generate_fake_samples(generator, latent_dim, n)
	Draw scatter diagram of true and false data points
	pyplot.scatter(x_real[:, 0], x_real[:, 1], color='red')
	pyplot.scatter(x_fake[:, 0], x_fake[:, 1], color='blue')
	pyplot.show()
Copy the code

In addition, we may also be interested in the performance of the discriminator model.

Specifically, we are interested in understanding the ability of the discriminator model to correctly distinguish between true and false samples. A good generator model should confuse the discriminator model, resulting in close to 50% classification accuracy on true and false samples.

We can update the Summarize_performance () method to take a discriminator and the current epoch number as parameters, and generate a report of the accuracy of true and false samples.

Evaluate the discriminator and draw true and false points
def summarize_performance(epoch, generator, discriminator, latent_dim, n=100):
	# Prepare real samples
	x_real, y_real = generate_real_samples(n)
	Evaluate the discriminator on a real sample
	_, acc_real = discriminator.evaluate(x_real, y_real, verbose=0)
	Prepare fake samples
	x_fake, y_fake = generate_fake_samples(generator, latent_dim, n)
	# Evaluate the discriminator on the dummy sample
	_, acc_fake = discriminator.evaluate(x_fake, y_fake, verbose=0)
	Summarize the discriminator performance
	print(epoch, acc_real, acc_fake)
	Draw scatter plots of true and false data
	pyplot.scatter(x_real[:, 0], x_real[:, 1], color='red')
	pyplot.scatter(x_fake[:, 0], x_fake[:, 1], color='blue')
	pyplot.show()
Copy the code

This method can be called periodically during training.

For example, if we train the model 10,000 times, we check the performance of the model every 2,000 iterations.

We can parameterize the frequency of the check with the n_eval line argument and, after a certain number of iterations, call the Summarize_performance () method from the train() method.

The updated train() method is shown below.

Train generators and discriminators
def train(g_model, d_model, gan_model, latent_dim, n_epochs=10000, n_batch=128, n_eval=2000):
	Update the discriminator with half the batch quantity
	half_batch = int(n_batch / 2)
	Manually iterate over the epoch
	for i in range(n_epochs):
		# Prepare real samples
		x_real, y_real = generate_real_samples(half_batch)
		Prepare fake samples
		x_fake, y_fake = generate_fake_samples(g_model, latent_dim, half_batch)
		Update the discriminator
		d_model.train_on_batch(x_real, y_real)
		d_model.train_on_batch(x_fake, y_fake)
		Prepare points in hidden space as input to the generator
		x_gan = generate_latent_points(latent_dim, n_batch)
		# Create a counter tag for the fake sample
		y_gan = ones((n_batch, 1))
		# Error update generator via discriminator
		gan_model.train_on_batch(x_gan, y_gan)
		# evaluate the model per n_eval epoch
		if (i+1) % n_eval == 0:
			summarize_performance(i, g_model, d_model, latent_dim)
Copy the code

A complete example of training generative adversarial networks

We now have all the conditions necessary to train and evaluate generating adversarial networks for one-dimensional functions.

The complete example is shown below.

# Train a generative adversarial network on a one-dimensional function
from numpy import hstack
from numpy import zeros
from numpy import ones
from numpy.random import rand
from numpy.random import randn
from keras.models import Sequential
from keras.layers import Dense
from matplotlib import pyplot

Define a separate discriminator model
def define_discriminator(n_inputs=2):
	model = Sequential()
	model.add(Dense(25, activation='relu', kernel_initializer='he_uniform', input_dim=n_inputs))
	model.add(Dense(1, activation='sigmoid'))
	# build model
	model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
	return model

Define a separate generator model
def define_generator(latent_dim, n_outputs=2):
	model = Sequential()
	model.add(Dense(15, activation='relu', kernel_initializer='he_uniform', input_dim=latent_dim))
	model.add(Dense(n_outputs, activation='linear'))
	return model

Update the generator by defining the merged generator and discriminator models
def define_gan(generator, discriminator):
	Set the weight of the discriminator to untrainable
	discriminator.trainable = False
	# Connect them
	model = Sequential()
	# Add generator
	model.add(generator)
	Add a discriminator
	model.add(discriminator)
	# build model
	model.compile(loss='binary_crossentropy', optimizer='adam')
	return model

Generate n real samples and class tags
def generate_real_samples(n):
	Generate input values in the range [-0.5, 0.5]
	X1 = rand(n) - 0.5
	# Generate the output value X^2
	X2 = X1 * X1
	# stack array
	X1 = X1.reshape(n, 1)
	X2 = X2.reshape(n, 1)
	X = hstack((X1, X2))
	Generate class tags
	y = ones((n, 1))
	return X, y

# Generate points in hidden space as input to the generator
def generate_latent_points(latent_dim, n):
	# Spawn in hidden space
	x_input = randn(latent_dim * n)
	Adjust a batch input dimension size for the network
	x_input = x_input.reshape(n, latent_dim)
	return x_input

Generate n false samples and class tags with generator
def generate_fake_samples(generator, latent_dim, n):
	# Spawn in hidden space
	x_input = generate_latent_points(latent_dim, n)
	# Predict the output value
	X = generator.predict(x_input)
	Create a class tag
	y = zeros((n, 1))
	return X, y

Evaluate the discriminator and draw true and false points
def summarize_performance(epoch, generator, discriminator, latent_dim, n=100):
	# Prepare real samples
	x_real, y_real = generate_real_samples(n)
	Evaluate the discriminator on a real sample
	_, acc_real = discriminator.evaluate(x_real, y_real, verbose=0)
	Prepare fake samples
	x_fake, y_fake = generate_fake_samples(generator, latent_dim, n)
	# Evaluate the discriminator on the dummy sample
	_, acc_fake = discriminator.evaluate(x_fake, y_fake, verbose=0)
	Summarize the discriminator performance
	print(epoch, acc_real, acc_fake)
	Draw scatter plots of true and false data
	pyplot.scatter(x_real[:, 0], x_real[:, 1], color='red')
	pyplot.scatter(x_fake[:, 0], x_fake[:, 1], color='blue')
	pyplot.show()

Train generators and discriminators
def train(g_model, d_model, gan_model, latent_dim, n_epochs=10000, n_batch=128, n_eval=2000):
	Use half the batch number to train the discriminator
	half_batch = int(n_batch / 2)
	Manually iterate over the epoch
	for i in range(n_epochs):
		# Prepare real samples
		x_real, y_real = generate_real_samples(half_batch)
		Prepare fake samples
		x_fake, y_fake = generate_fake_samples(g_model, latent_dim, half_batch)
		Update the discriminator
		d_model.train_on_batch(x_real, y_real)
		d_model.train_on_batch(x_fake, y_fake)
		Prepare points in hidden space as input to the generator
		x_gan = generate_latent_points(latent_dim, n_batch)
		# Create a counter tag for the fake sample
		y_gan = ones((n_batch, 1))
		# Error update generator via discriminator
		gan_model.train_on_batch(x_gan, y_gan)
		# Evaluate each N_EVAL EPOCH model
		if (i+1) % n_eval == 0:
			summarize_performance(i, g_model, d_model, latent_dim)

# Dimensions of hidden space
latent_dim = 5
Create a discriminator
discriminator = define_discriminator()
# create generator
generator = define_generator(latent_dim)
Create a generative adversarial network
gan_model = define_gan(generator, discriminator)
# Training model
train(generator, discriminator, gan_model, latent_dim)
Copy the code

Running this example will train each 2000 batches to generate a report on model performance and draw a scatter plot.

Your own results might be different because of the random nature of the training algorithm and the generating model itself.

We can see that the training process is relatively unstable. The first column is the number of iterations, the second column is the discriminator’s classification accuracy for the real sample, and the third column is the discriminator’s classification accuracy for the generated (false) sample.

In this case, we can see that the discriminator is still quite confused about the real sample, and its performance in identifying the fake sample is quite different.

1999 0.45 1.0
3999 0.45 0.91
5999 0.86 0.16
7999 0.6 0.41
9999 0.15 0.93
Copy the code

For simplicity, I will omit the five created scatter plots; We’ll just look at two of them.

The first diagram, created after 2,000 iterations, shows a comparison of the real (red) and fake (blue) samples. The model did not perform well at first, generating points only in the positive input field, although the functional relationship was correct.

Scatter plots of real and generated samples drawn for the objective function after 2000 iterations.

The second scatter plot is a comparison of the real (red) and fake (blue) samples after 10,000 iterations.

Here we can see that the generation model does produce realistic samples, the input fields are in the correct range between -0.5 and 0.5, and the output values show a functional relationship approximating X^2.

Scatter plots of real and generated samples drawn for the objective function after 10000 iterations.

expand

This section lists some ideas you may wish to explore outside of the tutorial.

  • Model architectures: Experiment with model architectures of other discriminators and generators, such as more or fewer neurons, layers, and alternative activation functions such as Leaky ReLU.
  • Data size: Use other activation functions such as Hyperbolic Tangent (TANH) and train data size as needed.
  • Other objective functions: Use other objective functions, such as a simple sine curve, gaussian distribution, a different quadratic equation or even a multimodal polynomial function.

If you explore these extensions, I’d love to hear about them. Leave your findings in the comments below.

Develop reading

This section provides additional resources on this topic if you want to learn more about it.

API

  • Keras API
  • How can I “freeze” the Keras layer?
  • MatplotLib API
  • numpy.random.rand API
  • numpy.random.randn API
  • numpy.zeros API
  • numpy.ones API
  • numpy.hstack API

conclusion

In this tutorial, you learned how to build a generative adversarial network from scratch using a one-dimensional function.

Specifically, you learned:

  • The benefits of constructing a generative adversarial network from scratch using a simple one-dimensional function.
  • How to build independent discriminator and generator models, and a combined model to train generators by discriminator predictive behavior.
  • How to evaluate the generated samples subjectively in the context of real data in the problem domain.

Do you have any questions? Leave your questions in the comments below and I’ll do my best to answer them.

If you find any mistakes in your translation or other areas that need to be improved, you are welcome to the Nuggets Translation Program to revise and PR your translation, and you can also get the corresponding reward points. The permanent link to this article at the beginning of this article is the MarkDown link to this article on GitHub.

The Nuggets Translation Project is a community that translates quality Internet technical articles from English sharing articles on nuggets. The content covers Android, iOS, front-end, back-end, blockchain, products, design, artificial intelligence and other fields. If you want to see more high-quality translation, please continue to pay attention to the Translation plan of Digging Gold, the official Weibo, Zhihu column.