Understanding Deep Neural Networks Training Course

Part 1 – Deep Learning and DNN Concepts

Introduction to AI, Machine Learning & Deep Learning

• Overview of the history, core concepts, and typical applications of artificial intelligence, distinguishing reality from common misconceptions in the field
• Collective Intelligence: the process of aggregating knowledge shared among multiple virtual agents
• Genetic algorithms: evolving populations of virtual agents through selection processes
• Definition of a standard Learning Machine
• Task categories: supervised learning, unsupervised learning, and reinforcement learning
• Action types: classification, regression, clustering, density estimation, and dimensionality reduction
• Examples of Machine Learning algorithms, including Linear regression, Naive Bayes, and Random Tree
• Machine Learning versus Deep Learning: identifying problems where Machine Learning still represents the state of the art (e.g., Random Forests & XGBoost)

Basic Concepts of a Neural Network (Application: multi-layer perceptron)

• Recap of essential mathematical foundations
• Definition of a neural network: classical architecture, activation functions,
• Weighting of previous activations and the concept of network depth
• Defining the learning process in neural networks: cost functions, back-propagation, Stochastic gradient descent, and maximum likelihood
• Modeling neural networks: configuring input and output data based on the problem type (regression, classification, etc.), and addressing the curse of dimensionality
• Distinguishing between multi-feature data and signals, and selecting appropriate cost functions based on data characteristics
• Function approximation by neural networks: theoretical presentation and practical examples
• Distribution approximation by neural networks: theoretical presentation and practical examples
• Data Augmentation: techniques for balancing datasets
• Generalizing the results produced by neural networks
• Initialization and regularization techniques: L1/L2 regularization and Batch Normalization
• Optimization and convergence algorithms

Standard ML / DL Tools

This section provides a comparative overview of key tools, highlighting their advantages, disadvantages, position within the ecosystem, and typical use cases.

• Data management tools: Apache Spark, Apache Hadoop Tools
• Machine Learning libraries: Numpy, Scipy, Sci-kit
• High-level Deep Learning frameworks: PyTorch, Keras, Lasagne
• Low-level Deep Learning frameworks: Theano, Torch, Caffe, TensorFlow

Convolutional Neural Networks (CNN).

• Introduction to CNNs: fundamental principles and key applications
• Basic operations of a CNN: convolutional layers, kernel usage,
• Padding & stride mechanisms, feature map generation, and pooling layers, including extensions to 1D, 2D, and 3D
• Review of CNN architectures that have established new standards in classification tasks
• Image-specific architectures: LeNet, VGG Networks, Network in Network, Inception, and ResNet, highlighting the innovations introduced by each and their broader applications (e.g., 1x1 convolution or residual connections)
• Implementation of attention models
• Application to common classification scenarios involving text or images
• Generative applications of CNNs: super-resolution and pixel-to-pixel segmentation, including a presentation of relevant techniques
• Primary strategies for enhancing feature maps for image generation

Recurrent Neural Networks (RNN).

• Introduction to RNNs: fundamental principles and applications
• Basic RNN operations: hidden activations, back propagation through time, and the unfolded representation
• Evolution towards Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTM)
• Overview of the various states and the evolutionary improvements brought by these architectures
• Challenges related to convergence and vanishing gradients
• Classical architectures for time-series prediction and classification
• RNN Encoder-Decoder architectures and the integration of attention models
• Natural Language Processing (NLP) applications: word/character encoding and translation
• Video applications: predicting the next generated image in a video sequence

Generative models: Variational AutoEncoder (VAE) and Generative Adversarial Networks (GAN).

• Introduction to generative models and their relationship with CNNs
• Auto-encoders: dimensionality reduction and basic generation capabilities
• Variational Auto-encoders: generative models for distribution approximation, including the definition and use of latent space, the reparameterization trick, observed applications, and limitations
• Generative Adversarial Networks: core fundamentals
• Dual Network Architecture (Generator and Discriminator), alternating learning processes, and available cost functions
• Convergence behaviors of GANs and common difficulties encountered
• Techniques for improved convergence, such as Wasserstein GAN and Began, including the concept of Earth Mover's Distance
• Applications in image/photograph generation, text generation, and super-resolution

Deep Reinforcement Learning.

• Introduction to reinforcement learning: controlling an agent within a defined environment
• Defining states and possible actions
• Utilizing neural networks to approximate state functions
• Deep Q-Learning: experience replay techniques and their application to controlling video games
• Policy optimization: on-policy and off-policy methods, Actor-Critic architecture, and A3C
• Practical applications: controlling single video games or digital systems

Part 2 – Theano for Deep Learning

Theano Basics

• Introduction to the library
• Installation and configuration processes

TheanoFunctions

• Management of inputs, outputs, updates, and givens

Training and Optimization of a Neural Network Using Theano

• Neural Network Modeling techniques
• Implementing Logistic Regression
• Managing Hidden Layers
• Training a network
• Computing and Classification tasks
• Optimization strategies
• Understanding Log Loss

Testing the Model

Part 3 – DNN Using TensorFlow

TensorFlow Basics

• Creating, initializing, saving, and restoring TensorFlow variables
• Feeding, reading, and preloading data in TensorFlow
• Leveraging TensorFlow infrastructure for large-scale model training
• Visualizing and evaluating models using TensorBoard

TensorFlow Mechanics

• Data preparation
• Data downloading procedures
• Handling Inputs and Placeholders
• Building Graphs
  • Inference components
  • Loss calculation
  • Training operations
• Training the Model
  • Graph structure
  • Session management
  • Training loops
• Evaluating the Model
  • Constructing the Evaluation Graph
  • Processing Evaluation Output

The Perceptron

• Various activation functions
• The perceptron learning algorithm
• Performing binary classification using the perceptron
• Document classification using the perceptron
• Identifying the limitations of the perceptron

From the Perceptron to Support Vector Machines

• Kernels and the kernel trick
• Maximum margin classification and the role of support vectors

Artificial Neural Networks

• Handling nonlinear decision boundaries
• Differences between feedforward and feedback artificial neural networks
• Multilayer perceptrons
• Techniques for minimizing the cost function
• Forward propagation mechanisms
• Back propagation processes
• Strategies for improving neural network learning

Convolutional Neural Networks

• Defining project goals
• Designing the Model Architecture
• Understanding underlying principles
• Organizing code structure
• Launching and Training the Model
• Evaluating Model performance

Basic Introductions to the Following Modules (Brief overviews provided based on available time):

TensorFlow - Advanced Usage

• Managing Threading and Queues
• Implementing Distributed TensorFlow
• Writing Documentation and Sharing Models
• Customizing Data Readers
• Manipulating TensorFlow Model Files

TensorFlow Serving

• Introduction to the service
• Basic Serving Tutorial
• Advanced Serving Tutorial
• Tutorial on Serving the Inception Model