drop_regularization.py

# -*- coding: utf-8 -*-
"""Drop_Regularization.ipynb

Automatically generated by Colab.

Original file is located at
    https://colab.research.google.com/drive/1BNBbpnp87cTZzo6WS0rc8UWTH6tnZD93

Drop regularization, often referred to simply as dropout, is a technique used in neural networks to prevent overfitting. It works by randomly dropping (setting to zero) a proportion of input units during training time, which helps prevent neurons from co-adapting too much to the training data. This technique was introduced by Geoffrey Hinton and his colleagues at the University of Toronto in 2012.

Purpose of Dropout Regularization:
Preventing Overfitting: By dropping random units during training, dropout effectively prevents complex co-adaptations on the training data, which tends to improve generalization to new data.
How Dropout Works:
During Training:

For each training example, dropout randomly sets some of the activations in the hidden layers to zero.
The probability of dropping out a unit (i.e., setting it to zero) is typically set between 0.2 and 0.5.
After training, the activations are scaled by the dropout rate to compensate for the units that are dropped during training.
During Testing or Inference:

Dropout is not applied during testing or when making predictions. Instead, the full network is used.
"""

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout

# Example input and output dimensions
input_dim = 100  # Example: dimensionality of your input data
output_dim = 10  # Example: number of output classes

# Define the model
model = Sequential()
model.add(Dense(128, activation='relu', input_shape=(input_dim,)))
model.add(Dropout(0.5))  # Dropout with a rate of 0.5
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.3))  # Dropout with a rate of 0.3
model.add(Dense(output_dim, activation='softmax'))

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

import numpy as np

X_train = np.random.rand(100, input_dim)
y_train = np.random.randint(0, output_dim, size=100)
X_val = np.random.rand(20, input_dim)
y_val = np.random.randint(0, output_dim, size=20)

# Train the model
model.fit(X_train, y_train, epochs=10, batch_size=32, validation_data=(X_val, y_val))

# Evaluate the model
X_test = np.random.rand(30, input_dim)
y_test = np.random.randint(0, output_dim, size=30)
loss, accuracy = model.evaluate(X_test, y_test)
print(f'Test accuracy: {accuracy}')