Free Ebook: Applications of Deep Neural Networks

Free Ebook: Applications of Deep Neural Networks

by admin
Maret 6, 2021
0 comments

Where to get it:

ArXiv Article: https://arxiv.org/abs/2009.05673
PDF: https://arxiv.org/pdf/2009.05673.pdf

Abstract

Deep learning is a group of exciting new technologies for neural networks. Through a combination of advanced training techniques and neural network architectural components, it is now possible to create neural networks that can handle tabular data, images, text, and audio as both input and output. Deep learning allows a neural network to learn hierarchies of information in a way that is like the function of the human brain. This course will introduce the student to classic neural network structures, Convolution Neural Networks (CNN), Long Short-Term Memory (LSTM), Gated Recurrent Neural Networks (GRU), General Adversarial Networks (GAN), and reinforcement learning. Application of these architectures to computer vision, time series, security, natural language processing (NLP), and data generation will be covered. High-Performance Computing (HPC) aspects will demonstrate how deep learning can be leveraged both on graphical processing units (GPUs), as well as grids. Focus is primarily upon the application of deep learning to problems, with some introduction to mathematical foundations. Readers will use the Python programming language to implement deep learning using Google TensorFlow and Keras. It is not necessary to know Python prior to this book; however, familiarity with at least one programming language is assumed.

Introduction

Python Preliminaries

Assignments
Your Instructor: Jeff Heaton
Course Resources
What is Deep Learning
What is Machine Learning
Regression
Classification
Beyond Classification and Regression
What are Neural Networks
Why Deep Learning?
Python for Deep Learning
Software Installation
Python Introduction
Jupyter Notebooks
Python Versions
Module Assignment
Introduction to Python
Python Lists, Dictionaries, Sets and JSON
Lists and Tuples
Sets
Maps/Dictionaries/Hash Tables
More Advanced Lists
An Introduction to JSON
File Handling
Read a CSV File
Read (stream) a Large CSV File
Read a Text File
Read an Image
Functions, Lambdas, and Map/Reduce
Map
Filter
Lambda
Reduce

Python for Machine Learning

Introduction to Pandas
Missing Values
Dealing with Outliers
Dropping Fields
Concatenating Rows and Columns
Training and Validation
Converting a Dataframe to a Matrix
Saving a Dataframe to CSV
Saving a Dataframe to Pickle
Module Assignment
Categorical and Continuous Values
Encoding Continuous Values
Encoding Categorical Values as Dummies
Target Encoding for Categoricals
Encoding Categorical Values as Ordinal
Grouping, Sorting, and Shuffling
Shuffling a Dataset
Sorting a Data Set
Grouping a Data Set
Apply and Map
Using Map with Dataframes
Using Apply with Dataframes
Feature Engineering with Apply and Map
Feature Engineering
Calculated Fields
Google API Keys
Other Examples: Dealing with Addresses

Introduction to TensorFlow

Deep Learning and Neural Network Introduction
Classification or Regression
Neurons and Layers
Types of Neurons
Input and Output Neurons
Hidden Neurons
Bias Neurons
Context Neurons
Other Neuron Types
Why are Bias Neurons Needed?
Modern Activation Functions
Linear Activation Function
Rectified Linear Units (ReLU)
Softmax Activation Function
Classic Activation Functions
Step Activation Function
Sigmoid Activation Function
Hyperbolic Tangent Activation Function
Why ReLU?
Module Assignment
Introduction to Tensorflow and Keras
Why TensorFlow
Deep Learning Tools
Using TensorFlow Directly
TensorFlow Linear Algebra Examples
TensorFlow Mandelbrot Set Example
Introduction to Keras
Simple TensorFlow Regression: MPG
Introduction to Neural Network Hyperparameters
Controlling the Amount of Output
Regression Prediction
Simple TensorFlow Classification: Iris
Saving and Loading a Keras Neural Network
Early Stopping in Keras to Prevent Overfitting
Early Stopping with Classification
Early Stopping with Regression
Extracting Weights and Manual Network Calculation
Weight Initialization
Manual Neural Network Calculation

Training for Tabular Data

Encoding a Feature Vector for Keras Deep Learning
Generate X and Y for a Classification Neural Network
Generate X and Y for a Regression Neural Network
Module Assignment
Multiclass Classification with ROC and AUC
Binary Classification and ROC Charts
ROC Chart Example
Multiclass Classification Error Metrics
Calculate Classification Accuracy
Calculate Classification Log Loss
Keras Regression for Deep Neural Networks with RMSE
Mean Square Error
Root Mean Square Error
Lift Chart
Training Neural Networks
Classic Backpropagation
Momentum Backpropagation
Batch and Online Backpropagation
Stochastic Gradient Descent
Other Techniques
ADAM Update
Methods Compared
Specifying the Update Rule in Tensorflow
Error Calculation from Scratch
Regression
Classification

Regularization and Dropout

Introduction to Regularization: Ridge and Lasso
L and L Regularization
Linear Regression
L (Lasso) Regularization
L (Ridge) Regularization
ElasticNet Regularization
Using K-Fold Cross-validation with Keras
Regression vs Classification K-Fold Cross-Validation
Out-of-Sample Regression Predictions with K-Fold Cross-Validation
Classification with Stratified K-Fold Cross-Validation
Training with both a Cross-Validation and a Holdout Set
L and L Regularization to Decrease Overfitting
Drop Out for Keras to Decrease Overfitting
Benchmarking Regularization Techniques
Additional Reading on Hyperparameter Tuning
Bootstrapping for Regression
Bootstrapping for Classification
Benchmarking

Convolutional Neural Networks (CNN) for Computer Vision

Image Processing in Python
Creating Images (from pixels) in Python
Transform Images in Python (at the pixel level)
Standardize Images
Adding Noise to an Image
Module Assignment
Keras Neural Networks for Digits and Fashion MNIST
Computer Vision
Computer Vision Data Sets
MNIST Digits Data Set
MNIST Fashion Data Set
CIFAR Data Set
Other Resources
Convolutional Neural Networks (CNNs)
Convolution Layers
Max Pooling Layers
TensorFlow with CNNs
Access to Data Sets – DIGITS
Display the Digits
Training/Fitting CNN – DIGITS
Evaluate Accuracy – DIGITS
MNIST Fashion
Display the Apparel
Training/Fitting CNN – Fashion
Implementing a ResNet in Keras
Keras Sequence vs Functional Model API
CIFAR Dataset
ResNet V
CONTENTS ix
ResNet V
Using Your Own Images with Keras
Recognizing Multiple Images with Darknet
How Does DarkNet/YOLO Work?
Using YOLO in Python
Installing YoloV-TF
Transfering Weights
Running DarkFlow (YOLO)
Module Assignment

Generative Adversarial Networks

Introduction to GANS for Image and Data Generation
Implementing DCGANs in Keras
Face Generation with StyleGAN and Python
Keras Sequence vs Functional Model API
Generating High Rez GAN Faces with Google CoLab
Run StyleGan From Command Line
Run StyleGAN From Python Code
Examining the Latent Vector
GANS for Semi-Supervised Training in Keras
Semi-Supervised Classification Training
Semi-Supervised Regression Training
Application of Semi-Supervised Regression
An Overview of GAN Research
Select Projects

Kaggle Data Sets

Introduction to Kaggle
Kaggle Ranks
Typical Kaggle Competition
How Kaggle Competition Scoring
Preparing a Kaggle Submission
Select Kaggle Competitions
Module Assignment
Building Ensembles with Scikit-Learn and Keras
Evaluating Feature Importance
Classification and Input Perturbation Ranking
Regression and Input Perturbation Ranking
Biological Response with Neural Network
What Features/Columns are Important
Neural Network Ensemble
Architecting Network: Hyperparameters
Number of Hidden Layers and Neuron Counts
Activation Functions
Advanced Activation Functions
Regularization: L, L, Dropout
Batch Normalization
Training Parameters
Experimenting with Hyperparameters
x CONTENTS
Bayesian Hyperparameter Optimization for Keras
Current Semester’s Kaggle
Iris as a Kaggle Competition
MPG as a Kaggle Competition (Regression)
Module Assignment

Transfer Learning

Introduction to Keras Transfer Learning
Transfer Learning Example
Module Assignment
Popular Pretrained Neural Networks for Keras
DenseNet
InceptionResNetV and InceptionV
MobileNet
MobileNetV
NASNet
ResNet, ResNetV, ResNeXt
VGG and VGG
Xception
Transfer Learning for Computer Vision and Keras
Transfer
Transfer Learning for Languages and Keras
Transfer Learning for Keras Feature Engineering

Time Series in Keras

Time Series Data Encoding
Module Assignment
Programming LSTM with Keras and TensorFlow
Understanding LSTM
Simple TensorFlow LSTM Example
Sun Spots Example
Further Reading for LSTM
Text Generation with LSTM
Additional Information
Character-Level Text Generation
Image Captioning with Keras and TensorFlow
Needed Data
Running Locally
Clean/Build Dataset From Flickrk
Choosing a Computer Vision Neural Network to Transfer
Creating the Training Set
Using a Data Generator
Loading Glove Embeddings
Building the Neural Network
Train the Neural Network
Generating Captions
Evaluate Performance on Test Data from Flickerk
Evaluate Performance on My Photos
Module Assignment
CONTENTS xi
Temporal CNN in Keras and TensorFlow
Sun Spots Example – CNN

Natural Language Processing and Speech Recognition

Getting Started with Spacy in Python
Installing Spacy
Tokenization
Sentence Diagramming
Stop Words
WordVec and Text Classification
Suggested Software for WordVec
What are Embedding Layers in Keras
Simple Embedding Layer Example
Transferring An Embedding
Training an Embedding
Natural Language Processing with Spacy and Keras
Word-Level Text Generation
Learning English from Scratch with Keras and TensorFlow
Imports and Utility Functions
Getting the Data
Building the Vocabulary
Building the Training and Test Data
Compile the Neural Network
Train the Neural Network
Evaluate Accuracy
Adhoc Query

Reinforcement Learning

Introduction to the OpenAI Gym
OpenAI Gym Leaderboard
Looking at Gym Environments
Render OpenAI Gym Environments from CoLab
Introduction to Q-Learning
Introducing the Mountain Car
Programmed Car
Reinforcement Learning
Running and Observing the Agent
Inspecting the Q-Table
Keras Q-Learning in the OpenAI Gym
DQN and the Cart-Pole Problem
Hyperparameters
Environment
Agent
Policies
Metrics and Evaluation
Replay Buffer
Data Collection
Training the agent
Visualization
KS-Statistic
Detecting Drift between Training and Testing Datasets by Training
Using a Keras Deep Neural Network with a Web Application
Converting Keras to CoreML
Creating an IOS CoreML Application
More Reading
Other Neural Network Techniques
What is AutoML
AutoML from your Local Computer
AutoML from Google Cloud
A Simple AutoML System
Running My Sample AutoML Program
Using Denoising AutoEncoders in Keras
Function Approximation
Multi-Output Regression
Simple Autoencoder
Autoencode (single image)
Standardize Images
Image Autoencoder (multi-image)
Adding Noise to an Image
Denoising Autoencoder
Anomaly Detection in Keras
Read in KDD Data Set
Preprocessing
Training the Autoencoder
Detecting an Anomaly
Training an Intrusion Detection System with KDD
Read in Raw KDD- Dataset
Analyzing a Dataset
Encode the feature vector
Train the Neural Network
New Technologies
Neural Structured Learning (NSL)
Bert, AlBert, and Other NLP Technologies
Explainability Frameworks

Advanced/Other Topics

Flask and Deep Learning Web Services
Flask Hello World
MPG Flask
Flask MPG Client
Images and Web Services
Part : Deploying a Model to AWS
Train Model (optionally, outside of AWS)
Next Step: Convert Model (must use AWS SageMaker Notebook)
Step Set up
Step Load the Keras model using the JSON and weights file
Step Export the Keras model to the TensorFlow ProtoBuf format (must use AWS
SageMaker Notebook)
Step Convert TensorFlow model to a SageMaker readable format (must use AWS
SageMaker Notebook)
Tar the entire directory and upload to S
Step Deploy the trained model (must use AWS SageMaker Notebook)
Test Model Deployment (optionally, outside of AWS)
Call the end point
Additional Reading
Using a Keras Deep Neural Network with a Web Application
When to Retrain Your Neural Network
Preprocessing the Sberbank Russian Housing Market Data
KS-Statistic
Detecting Drift between Training and Testing Datasets by Training
Using a Keras Deep Neural Network with a Web Application
Converting Keras to CoreML
Creating an IOS CoreML Application
More Reading
Other Neural Network Techniques
What is AutoML
AutoML from your Local Computer
AutoML from Google Cloud
A Simple AutoML System
Running My Sample AutoML Program
Using Denoising AutoEncoders in Keras
Function Approximation
Multi-Output Regression
Simple Autoencoder
Autoencode (single image)
Standardize Images
Image Autoencoder (multi-image)
Adding Noise to an Image
Denoising Autoencoder
Anomaly Detection in Keras
Read in KDD Data Set
Preprocessing
Training the Autoencoder
Detecting an Anomaly
Training an Intrusion Detection System with KDD
Read in Raw KDD- Dataset
Analyzing a Dataset
Encode the feature vector
Train the Neural Network
New Technologies
Neural Structured Learning (NSL)
Bert, AlBert, and Other NLP Technologies
Explainability Frameworks