Random Search for Hyperparameters

Class Reference

class pykitml.RandomSearch

This class is used to search for hyperparameters.

search(nsamples, nzoom, zoomratio, *args)

Generator function to loop through randomly generated hyperparameters. Total number of hyperparameters sampled will be nsamples*nzoom. First nsamples points will be sampled, then the function will ‘zoom in’ around the best sample, and nsamples more points will be sampled. This will be repeated nzoom times. The range for each hyperparameter should be passed as a list to *args. The range should be [from, to, 'type'], for e.g. [0.8, 1, 'float']. Three range types are available, 'float', 'int', 'log'.

Parameters:

• nsamples (int) – Number of hyperparameters to sample.
• nzoom (int) – Number of times to zoom in.
• zoomratio (float) – How much to zoom in.
• *args – Range type for each hyperparameter.

set_cost(cost)

Set the cost for current hyperparameter.

Parameters:cost (float) – The cost corresponding to current set of hyperparameters.

best

If the last generated hyperparameters is the best so far.

Note

This property has to be used AFTER calling set_cost()

Example: Tuning Feed-forward network for fashion-MNIST

import os

import pykitml as pk
from pykitml.datasets import mnist

# If the dataset is not available then download it
if not os.path.exists('mnist.pkl'):
    mnist.get(type='fashion')

# Load dataset
training_data, training_targets, testing_data, testing_targets = mnist.load()

# Search for hyperparameters
#   Learning rate alpha = 10^-4 to 10^-2
#   Decay rate = 0.8 to 1
#   Decay frequency = 10 to 30
#   Batch size = 10 to 100
search = pk.RandomSearch()
for alpha, decay, decay_freq, bsize in search.search(
    10, 3, 5, [-4, -2, 'log'], [0.8, 1, 'float'], [10, 30, 'int'], [10, 100, 'int']):

    # Create a new neural network
    fashion_classifier = pk.NeuralNetwork([784, 100, 10])

    # Train it
    fashion_classifier.train(
        training_data=training_data,
        targets=training_targets,
        batch_size=bsize,
        epochs=1200,
        optimizer=pk.Adam(learning_rate=alpha, decay_rate=decay),
        testing_freq=100,
        decay_freq=decay_freq
    )

    cost = fashion_classifier.cost(testing_data, testing_targets)
    search.set_cost(cost)

    # Save the best model
    if search.best:
        pk.save(fashion_classifier, 'best.pkl')

# Load the best model
fashion_classifier = pk.load('best.pkl')

# Show performance
accuracy = fashion_classifier.accuracy(testing_data, testing_targets)
print('Test Accuracy:', accuracy)