Where to get it:
- ArXiv Article: https://arxiv.org/abs/2009.05673
- PDF: https://arxiv.org/pdf/2009.05673.pdf
Abstract
Deep learning is a group of exciting new technologies for neural networks. Through a combination of advanced training techniques and neural network architectural components, it is now possible to create neural networks that can handle tabular data, images, text, and audio as both input and output. Deep learning allows a neural network to learn hierarchies of information in a way that is like the function of the human brain. This course will introduce the student to classic neural network structures, Convolution Neural Networks (CNN), Long Short-Term Memory (LSTM), Gated Recurrent Neural Networks (GRU), General Adversarial Networks (GAN), and reinforcement learning. Application of these architectures to computer vision, time series, security, natural language processing (NLP), and data generation will be covered. High-Performance Computing (HPC) aspects will demonstrate how deep learning can be leveraged both on graphical processing units (GPUs), as well as grids. Focus is primarily upon the application of deep learning to problems, with some introduction to mathematical foundations. Readers will use the Python programming language to implement deep learning using Google TensorFlow and Keras. It is not necessary to know Python prior to this book; however, familiarity with at least one programming language is assumed.
Contents
Introduction
Python Preliminaries
- Assignments
- Your Instructor: Jeff Heaton
- Course Resources
- What is Deep Learning
- What is Machine Learning
- Regression
- Classification
- Beyond Classification and Regression
- What are Neural Networks
- Why Deep Learning?
- Python for Deep Learning
- Software Installation
- Python Introduction
- Jupyter Notebooks
- Python Versions
- Module Assignment
- Introduction to Python
- Python Lists, Dictionaries, Sets and JSON
- Lists and Tuples
- Sets
- Maps/Dictionaries/Hash Tables
- More Advanced Lists
- An Introduction to JSON
- File Handling
- Read a CSV File
- Read (stream) a Large CSV File
- Read a Text File
- Read an Image
- Functions, Lambdas, and Map/Reduce
- Map
- Filter
- Lambda
- Reduce
Python for Machine Learning
- Introduction to Pandas
- Missing Values
- Dealing with Outliers
- Dropping Fields
- Concatenating Rows and Columns
- Training and Validation
- Converting a Dataframe to a Matrix
- Saving a Dataframe to CSV
- Saving a Dataframe to Pickle
- Module Assignment
- Categorical and Continuous Values
- Encoding Continuous Values
- Encoding Categorical Values as Dummies
- Target Encoding for Categoricals
- Encoding Categorical Values as Ordinal
- Grouping, Sorting, and Shuffling
- Shuffling a Dataset
- Sorting a Data Set
- Grouping a Data Set
- Apply and Map
- Using Map with Dataframes
- Using Apply with Dataframes
- Feature Engineering with Apply and Map
- Feature Engineering
- Calculated Fields
- Google API Keys
- Other Examples: Dealing with Addresses
Introduction to TensorFlow
- Deep Learning and Neural Network Introduction
- Classification or Regression
- Neurons and Layers
- Types of Neurons
- Input and Output Neurons
- Hidden Neurons
- Bias Neurons
- Context Neurons
- Other Neuron Types
- Why are Bias Neurons Needed?
- Modern Activation Functions
- Linear Activation Function
- Rectified Linear Units (ReLU)
- Softmax Activation Function
- Classic Activation Functions
- Step Activation Function
- Sigmoid Activation Function
- Hyperbolic Tangent Activation Function
- Why ReLU?
- Module Assignment
- Introduction to Tensorflow and Keras
- Why TensorFlow
- Deep Learning Tools
- Using TensorFlow Directly
- TensorFlow Linear Algebra Examples
- TensorFlow Mandelbrot Set Example
- Introduction to Keras
- Simple TensorFlow Regression: MPG
- Introduction to Neural Network Hyperparameters
- Controlling the Amount of Output
- Regression Prediction
- Simple TensorFlow Classification: Iris
- Saving and Loading a Keras Neural Network
- Early Stopping in Keras to Prevent Overfitting
- Early Stopping with Classification
- Early Stopping with Regression
- Extracting Weights and Manual Network Calculation
- Weight Initialization
- Manual Neural Network Calculation
Training for Tabular Data
- Encoding a Feature Vector for Keras Deep Learning
- Generate X and Y for a Classification Neural Network
- Generate X and Y for a Regression Neural Network
- Module Assignment
- Multiclass Classification with ROC and AUC
- Binary Classification and ROC Charts
- ROC Chart Example
- Multiclass Classification Error Metrics
- Calculate Classification Accuracy
- Calculate Classification Log Loss
- Keras Regression for Deep Neural Networks with RMSE
- Mean Square Error
- Root Mean Square Error
- Lift Chart
- Training Neural Networks
- Classic Backpropagation
- Momentum Backpropagation
- Batch and Online Backpropagation
- Stochastic Gradient Descent
- Other Techniques
- ADAM Update
- Methods Compared
- Specifying the Update Rule in Tensorflow
- Error Calculation from Scratch
- Regression
- Classification
Regularization and Dropout
- Introduction to Regularization: Ridge and Lasso
- L and L Regularization
- Linear Regression
- L (Lasso) Regularization
- L (Ridge) Regularization
- ElasticNet Regularization
- Using K-Fold Cross-validation with Keras
- Regression vs Classification K-Fold Cross-Validation
- Out-of-Sample Regression Predictions with K-Fold Cross-Validation
- Classification with Stratified K-Fold Cross-Validation
- Training with both a Cross-Validation and a Holdout Set
- L and L Regularization to Decrease Overfitting
- Drop Out for Keras to Decrease Overfitting
- Benchmarking Regularization Techniques
- Additional Reading on Hyperparameter Tuning
- Bootstrapping for Regression
- Bootstrapping for Classification
- Benchmarking
Convolutional Neural Networks (CNN) for Computer Vision
- Image Processing in Python
- Creating Images (from pixels) in Python
- Transform Images in Python (at the pixel level)
- Standardize Images
- Adding Noise to an Image
- Module Assignment
- Keras Neural Networks for Digits and Fashion MNIST
- Computer Vision
- Computer Vision Data Sets
- MNIST Digits Data Set
- MNIST Fashion Data Set
- CIFAR Data Set
- Other Resources
- Convolutional Neural Networks (CNNs)
- Convolution Layers
- Max Pooling Layers
- TensorFlow with CNNs
- Access to Data Sets – DIGITS
- Display the Digits
- Training/Fitting CNN – DIGITS
- Evaluate Accuracy – DIGITS
- MNIST Fashion
- Display the Apparel
- Training/Fitting CNN – Fashion
- Implementing a ResNet in Keras
- Keras Sequence vs Functional Model API
- CIFAR Dataset
- ResNet V
- CONTENTS ix
- ResNet V
- Using Your Own Images with Keras
- Recognizing Multiple Images with Darknet
- How Does DarkNet/YOLO Work?
- Using YOLO in Python
- Installing YoloV-TF
- Transfering Weights
- Running DarkFlow (YOLO)
- Module Assignment
Generative Adversarial Networks
- Introduction to GANS for Image and Data Generation
- Implementing DCGANs in Keras
- Face Generation with StyleGAN and Python
- Keras Sequence vs Functional Model API
- Generating High Rez GAN Faces with Google CoLab
- Run StyleGan From Command Line
- Run StyleGAN From Python Code
- Examining the Latent Vector
- GANS for Semi-Supervised Training in Keras
- Semi-Supervised Classification Training
- Semi-Supervised Regression Training
- Application of Semi-Supervised Regression
- An Overview of GAN Research
- Select Projects
Kaggle Data Sets
- Introduction to Kaggle
- Kaggle Ranks
- Typical Kaggle Competition
- How Kaggle Competition Scoring
- Preparing a Kaggle Submission
- Select Kaggle Competitions
- Module Assignment
- Building Ensembles with Scikit-Learn and Keras
- Evaluating Feature Importance
- Classification and Input Perturbation Ranking
- Regression and Input Perturbation Ranking
- Biological Response with Neural Network
- What Features/Columns are Important
- Neural Network Ensemble
- Architecting Network: Hyperparameters
- Number of Hidden Layers and Neuron Counts
- Activation Functions
- Advanced Activation Functions
- Regularization: L, L, Dropout
- Batch Normalization
- Training Parameters
- Experimenting with Hyperparameters
- x CONTENTS
- Bayesian Hyperparameter Optimization for Keras
- Current Semester’s Kaggle
- Iris as a Kaggle Competition
- MPG as a Kaggle Competition (Regression)
- Module Assignment
Transfer Learning
- Introduction to Keras Transfer Learning
- Transfer Learning Example
- Module Assignment
- Popular Pretrained Neural Networks for Keras
- DenseNet
- InceptionResNetV and InceptionV
- MobileNet
- MobileNetV
- NASNet
- ResNet, ResNetV, ResNeXt
- VGG and VGG
- Xception
- Transfer Learning for Computer Vision and Keras
- Transfer
- Transfer Learning for Languages and Keras
- Transfer Learning for Keras Feature Engineering
Time Series in Keras
- Time Series Data Encoding
- Module Assignment
- Programming LSTM with Keras and TensorFlow
- Understanding LSTM
- Simple TensorFlow LSTM Example
- Sun Spots Example
- Further Reading for LSTM
- Text Generation with LSTM
- Additional Information
- Character-Level Text Generation
- Image Captioning with Keras and TensorFlow
- Needed Data
- Running Locally
- Clean/Build Dataset From Flickrk
- Choosing a Computer Vision Neural Network to Transfer
- Creating the Training Set
- Using a Data Generator
- Loading Glove Embeddings
- Building the Neural Network
- Train the Neural Network
- Generating Captions
- Evaluate Performance on Test Data from Flickerk
- Evaluate Performance on My Photos
- Module Assignment
- CONTENTS xi
- Temporal CNN in Keras and TensorFlow
- Sun Spots Example – CNN
Natural Language Processing and Speech Recognition
- Getting Started with Spacy in Python
- Installing Spacy
- Tokenization
- Sentence Diagramming
- Stop Words
- WordVec and Text Classification
- Suggested Software for WordVec
- What are Embedding Layers in Keras
- Simple Embedding Layer Example
- Transferring An Embedding
- Training an Embedding
- Natural Language Processing with Spacy and Keras
- Word-Level Text Generation
- Learning English from Scratch with Keras and TensorFlow
- Imports and Utility Functions
- Getting the Data
- Building the Vocabulary
- Building the Training and Test Data
- Compile the Neural Network
- Train the Neural Network
- Evaluate Accuracy
- Adhoc Query
Reinforcement Learning
- Introduction to the OpenAI Gym
- OpenAI Gym Leaderboard
- Looking at Gym Environments
- Render OpenAI Gym Environments from CoLab
- Introduction to Q-Learning
- Introducing the Mountain Car
- Programmed Car
- Reinforcement Learning
- Running and Observing the Agent
- Inspecting the Q-Table
- Keras Q-Learning in the OpenAI Gym
- DQN and the Cart-Pole Problem
- Hyperparameters
- Environment
- Agent
- Policies
- Metrics and Evaluation
- Replay Buffer
- Data Collection
- Training the agent
- Visualization
- KS-Statistic
- Detecting Drift between Training and Testing Datasets by Training
- Using a Keras Deep Neural Network with a Web Application
- Converting Keras to CoreML
- Creating an IOS CoreML Application
- More Reading
- Other Neural Network Techniques
- What is AutoML
- AutoML from your Local Computer
- AutoML from Google Cloud
- A Simple AutoML System
- Running My Sample AutoML Program
- Using Denoising AutoEncoders in Keras
- Function Approximation
- Multi-Output Regression
- Simple Autoencoder
- Autoencode (single image)
- Standardize Images
- Image Autoencoder (multi-image)
- Adding Noise to an Image
- Denoising Autoencoder
- Anomaly Detection in Keras
- Read in KDD Data Set
- Preprocessing
- Training the Autoencoder
- Detecting an Anomaly
- Training an Intrusion Detection System with KDD
- Read in Raw KDD- Dataset
- Analyzing a Dataset
- Encode the feature vector
- Train the Neural Network
- New Technologies
- Neural Structured Learning (NSL)
- Bert, AlBert, and Other NLP Technologies
- Explainability Frameworks
Advanced/Other Topics
- Flask and Deep Learning Web Services
- Flask Hello World
- MPG Flask
- Flask MPG Client
- Images and Web Services
- Part : Deploying a Model to AWS
- Train Model (optionally, outside of AWS)
- Next Step: Convert Model (must use AWS SageMaker Notebook)
- Step Set up
- Step Load the Keras model using the JSON and weights file
- Step Export the Keras model to the TensorFlow ProtoBuf format (must use AWS
- SageMaker Notebook)
- Step Convert TensorFlow model to a SageMaker readable format (must use AWS
- SageMaker Notebook)
- Tar the entire directory and upload to S
- Step Deploy the trained model (must use AWS SageMaker Notebook)
- Test Model Deployment (optionally, outside of AWS)
- Call the end point
- Additional Reading
- Using a Keras Deep Neural Network with a Web Application
- When to Retrain Your Neural Network
- Preprocessing the Sberbank Russian Housing Market Data
- KS-Statistic
- Detecting Drift between Training and Testing Datasets by Training
- Using a Keras Deep Neural Network with a Web Application
- Converting Keras to CoreML
- Creating an IOS CoreML Application
- More Reading
- Other Neural Network Techniques
- What is AutoML
- AutoML from your Local Computer
- AutoML from Google Cloud
- A Simple AutoML System
- Running My Sample AutoML Program
- Using Denoising AutoEncoders in Keras
- Function Approximation
- Multi-Output Regression
- Simple Autoencoder
- Autoencode (single image)
- Standardize Images
- Image Autoencoder (multi-image)
- Adding Noise to an Image
- Denoising Autoencoder
- Anomaly Detection in Keras
- Read in KDD Data Set
- Preprocessing
- Training the Autoencoder
- Detecting an Anomaly
- Training an Intrusion Detection System with KDD
- Read in Raw KDD- Dataset
- Analyzing a Dataset
- Encode the feature vector
- Train the Neural Network
- New Technologies
- Neural Structured Learning (NSL)
- Bert, AlBert, and Other NLP Technologies
- Explainability Frameworks