• Mastering Statistics with R
  • Welcome!
    • Preface
      • Acknowledgement
      • Progress of this book
    • How to use this book?
  • Part I: Foundations
  • 1 Probability Concept
    • 1.1 Introduction to Probability
      • 1.1.1 What is probability ?
      • 1.1.2 Basic Mathematic
      • 1.1.3 History of probability
      • 1.1.4 Definitions of Probability
      • 1.1.5 Conditional Probability and Independence
      • 1.1.6 Bayes’ Theorem
    • 1.2 Random Variables
      • 1.2.1 Random variables and probability functions
      • 1.2.2 Expected values and Variance
      • 1.2.3 Transformation of random variables
    • 1.3 Families of Distributions
      • 1.3.1 Discrete probability distributions
      • 1.3.2 Continuous probability distributions
      • 1.3.3 Distributions derive from Normal
    • 1.4 Multivariate Random Variables
      • 1.4.1 Joint distributions
      • 1.4.2 Change of variables
      • 1.4.3 Families of multivariate distributions
    • 1.5 System of Moments
    • 1.6 Limit Theorem
      • 1.6.1 Central Limit Theorem
      • 1.6.2 Probability inequality
      • 1.6.3 Law of large numbers
  • 2 Elementary Statistics
    • 2.1 Descriptive Statistics
      • 2.1.1 Frequency distribution
      • 2.1.2 Measures of statistical characteristics
      • 2.1.3 Exploratory data analysis (EDA)
    • 2.2 Sampling
      • 2.2.1 Random sampling methods (probability sampling techniques)
      • 2.2.2 Nonrandom sampling methods (non-probability sampling techniques)
      • 2.2.3 Other sampling methods
    • 2.3 Estimation
      • 2.3.1 Point Estimation
      • 2.3.2 Interval Estimation
    • 2.4 Testing Hypotheses
      • 2.4.1 Null hypothesis vs. alternative hypothesis
      • 2.4.2 The Neyman-Pearson Lemma
    • 2.5 Some statistical test
      • 2.5.1 Parametric statistical test
      • 2.5.2 Non-parametric test
    • 2.6 Analysis of Variance (ANOVA)
      • 2.6.1 Levene’s test
      • 2.6.2 Bartlett’s test
      • 2.6.3 One-way ANOVA
      • 2.6.4 Welch’s ANOVA
      • 2.6.5 Kruskal–Wallis test
      • 2.6.6 Friedman test
      • 2.6.7 Normality Test
    • 2.7 Correlation Analysis and Linear Regression
  • 3 Mathematical Statistics
    • 3.1 Properties of estimators
      • 3.1.1 Uniformly Minimum Variance Unbiased Estimator (UMVUE)
    • 3.2 Limiting Distributions
      • 3.2.1 Converge in probability
      • 3.2.2 Converge in distribution
    • 3.3 Asymptotic Theory
    • 3.4 Hypothesis Testing Theory
      • 3.4.1 MP test and UMP test
      • 3.4.2 monotone likelihood ratio (MLR)
      • 3.4.3 LR-test, GLRT
      • 3.4.4 sequential probability ratio test (SPRT)
    • 3.5 Decision Theory
      • 3.5.1 Regret
    • 3.6 Bayesian Tests
  • Part II: Methodology - Beginner
  • 4 Probability Models
    • 4.1 Review of Probability Computation
    • 4.2 Stochastic Process
      • 4.2.1 Discrete-time stochastic process
      • 4.2.2 Continuous time stochastic process
      • 4.2.3 Random Walk
      • 4.2.4 Poisson Process
      • 4.2.5 Markov Process
      • 4.2.6 Wiener Process
      • 4.2.7 Lévy Process
    • 4.3 Discrete Time Markov Chain
    • 4.4 Continueous Time Markov Chain
      • 4.4.1 Semi-Markov Chain
      • 4.4.2 Hidden Markov Models
      • 4.4.3 Mover-Stayer Models
    • 4.5 Ergodic Theory
    • 4.6 Degradation data
    • 4.7 Renewal Theory
    • 4.8 Ruin Theory
    • 4.9 Extreme Value Theory
    • 4.10 Change Detection
    • 4.11 Stochastic Calculus
      • 4.11.1 Ito’s Lemma
  • 5 Regression Analysis
    • 5.1 Ordinary Least Squares (OLS)
      • 5.1.1 Variable Selection
      • 5.1.2 Model Selection
    • 5.2 Fixed Effect and Random Effect
    • 5.3 Analysis of Covariance (ANCOVA)
    • 5.4 Logistic Regression
    • 5.5 Fractional Model
    • 5.6 Nonlinear Regression
  • 6 Categorical Data Analysis
    • 6.1 Partial least squares regression (PLS)
  • 7 Multivariate Analysis
    • 7.1 Multivariate distributions
    • 7.2 General Linear Model
    • 7.3 Multivariate Analysis of Variance (MANOVA)
    • 7.4 Multivariate Analysis of Covariance (MANCOVA)
    • 7.5 Structural Equation Modeling (SEM)
    • 7.6 Statistical distance
    • 7.7 Dimension Reduction Method
      • 7.7.1 Random Projection
      • 7.7.2 Discriminant Analysis (LDA)
      • 7.7.3 SVD (Singular Value Decomposition)
      • 7.7.4 Principal Component Analysis (PCA)
      • 7.7.5 Nonnegative Matrix Factorization (NMF)
      • 7.7.6 t-SNE
      • 7.7.7 Locally Linear Embedding
      • 7.7.8 Independent Component Analysis (ICA)
      • 7.7.9 Autoencoders
      • 7.7.10 Laplacian Eigenmaps
      • 7.7.11 ISOMAP
      • 7.7.12 Uniform Manifold Approximation and Projection (UMAP)
      • 7.7.13 Self-organizing map
      • 7.7.14 Dynamic mode decomposition
    • 7.8 Factor Analysis
      • 7.8.1 Kaiser–Meyer–Olkin test
      • 7.8.2 Questionnaire
    • 7.9 Multidimensional Scaling (MDS)
    • 7.10 Canonical-correlation Analysis (CCA)
    • 7.11 Analysis of Similarities (ANOSIM)
  • 8 Time Series Analysis
    • 8.1 Time Series Decomposition
    • 8.2 ACF and PACF
    • 8.3 White Noise
    • 8.4 Autoregressive (AR)
    • 8.5 Moving Average (MA)
    • 8.6 Kalman Filter and Savitzky–Golay filter
    • 8.7 ARMA, ARIMA, SARIMA, SARFIMA
    • 8.8 Granger causality
    • 8.9 Nonlinear Time Series
      • 8.9.1 Threshold Autoregressive (TAR) Model
      • 8.9.2 GARCH
      • 8.9.3 Smooth Transition Autoregressive (STAR) Model
      • 8.9.4 Non-linear Moving Average (NMA) Model
      • 8.9.5 Polynomial and Exponential Model
    • 8.10 Hierarchical Time Series
    • 8.11 Multivariate Time Series
      • 8.11.1 VAR
      • 8.11.2 Factor Model
    • 8.12 Some Advanced Topics
      • 8.12.1 Lag regression
      • 8.12.2 Mixed-frequency data
      • 8.12.3 Time-varying Coefficients
  • Part III: Methodology - Advanced
  • 9 Generalized Linear Models
    • 9.1 Weighted Least Square (WLS) and Generalized Least Square (GLS)
      • 9.1.1 Rootogram
    • 9.2 Complex Linear Model
    • 9.3 Generalized Estimating Equation (GEE)
    • 9.4 Hierarchical Linear Model
      • 9.4.1 Instrumental variable
    • 9.5 Multilevel Model
  • 10 Spatial Statistics
    • 10.1 Point-referenced Data
      • 10.1.1 Gaussian Process
      • 10.1.2 Exploratory data analysis
      • 10.1.3 Models for spatial dependence
      • 10.1.4 Kriging (Spatial prediction)
    • 10.2 Areal/Lattice Data
      • 10.2.1 Spatial autocorrelation
      • 10.2.2 Conditionally auto-regressive (CAR) and Simultaneously auto-regressive (SAR) models
    • 10.3 Point Pattern Data
      • 10.3.1 Poisson processes
      • 10.3.2 Cox processes
      • 10.3.3 K-functions
    • 10.4 Other Topics
      • 10.4.1 Spatio-temporal models
      • 10.4.2 Frequency domain methods
      • 10.4.3 Deep Kriging
  • 11 Functional Data Analysis
  • 12 Bayesian Analysis
    • 12.1 Laplace Approximation and BIC
  • 13 High Dimensional Data Analysis
    • 13.1 Curse of Dimension
  • Part IV: Methodology - Others
  • 14 Nonparametric Method
    • 14.1 Measure of Centrality
    • 14.2 Nonparametric tests
    • 14.3 Quantile Regression
    • 14.4 Local Regression
    • 14.5 Isotonic regression
    • 14.6 Convex Regression
    • 14.7 Curve estimation
      • 14.7.1 Kernel
    • 14.8 Shape-Constrained Inference
  • 15 Directional Statistics
    • 15.1 Circular Distribution
    • 15.2 Circular Regression
  • 16 Geometric and Topological Data Analysis
    • 16.1 Compositional data
      • 16.1.1 Correlation analysis for compositional data
    • 16.2 Geodesic Regression
  • Part VI: Application - Biostatistics
  • 17 Biostatistical Data Analysis
    • 17.1 p-value correction
      • 17.1.1 Bofferoni
      • 17.1.2 Tukey’s HSD
      • 17.1.3 Fisher
      • 17.1.4 False Discovery Rate (FDR)
      • 17.1.5 Q-value
      • 17.1.6 E-value
    • 17.2 Trend Tests
      • 17.2.1 Cochran-Armitage test
      • 17.2.2 Jonckheere’s trend test
    • 17.3 Permutational multivariate analysis of variance (PERMANOVA)
      • 17.3.1 PERMDISP
    • 17.4 Propensity score
    • 17.5 PLINK
    • 17.6 Polygenic Risk Score
    • 17.7 RNA-seq Analysis
    • 17.8 Metabolomics Analysis
      • 17.8.1 SMART
      • 17.8.2 pareto normalization
      • 17.8.3 Bio-diversity
    • 17.9 Case Study
  • 18 Clinical Trials
    • 18.1 Phase I
    • 18.2 Phase II
    • 18.3 Phase III
    • 18.4 \(\alpha\) spending function
  • 19 Survial Analysis
    • 19.1 Unobserved data
    • 19.2 Survival Function and Hazard Function
    • 19.3 Kaplan–Meier Estimator
    • 19.4 Log-rank Test
    • 19.5 Proportional Hazards Model
    • 19.6 Accelerated Failure Time (AFT) Model
    • 19.7 Nelson–Aalen Estimator
    • 19.8 Turnbull-Frydman Estimator
    • 19.9 Restricted Median Survival Time (RMST)
    • 19.10 Firth’s penalized logistic regression
    • 19.11 Competing Risks
  • 20 Causal Inference
    • 20.1 Prerequisite Knowledge
    • 20.2 Causal diagrams
    • 20.3 Counterfactual model
    • 20.4 Identification and causal assumptions
    • 20.5 Estimation and modeling
    • 20.6 mediation analysis
    • 20.7 Moderation, Effect Measurement Modification, and Interaction
    • 20.8 Time-varying system
  • Part V: Application - Industrial Statistics
  • 21 Quality Control
    • 21.1 History
    • 21.2 7 tools
    • 21.3 ARL
    • 21.4 \(R\) chart
    • 21.5 \(s\) chart
    • 21.6 \(\bar{X}\) chart
    • 21.7 \(p\) chart
    • 21.8 CUSUM
    • 21.9 EWMA
    • 21.10 Sequential probability ratio test
  • 22 Reliability Analysis
  • 23 Design of Experiments
    • 23.1 Basic concept
      • 23.1.1 Latin hypercube
    • 23.2 Factorial Experiments
      • 23.2.1 Repeated measure design
      • 23.2.2 Two-way ANOVA
      • 23.2.3 mixed-design ANOVA
      • 23.2.4 Three-way ANOVA
    • 23.3 Sequential design
    • 23.4 Space-filling design
  • Part VII: Application - Others
  • 24 Operations Research
    • 24.1 Queueing Theory
    • 24.2 Optimization
      • 24.2.1 Linear Programming
      • 24.2.2 Convex Programming
      • 24.2.3 Non-linear Programming
      • 24.2.4 Integer Programming
    • 24.3 Project Management
  • 25 Financial Statistics
    • 25.1 Structural Vector Autoregressive (SVAR) Model
    • 25.2 Vector Error Correction Model (VECM)
  • 26 Social Statistics
    • 26.1 Human Behaviour
    • 26.2 Sport
    • 26.3 Psychometrics
      • 26.3.1 Item Response Theory
  • Part VIII: Computational Statistics
  • 27 Statistical Learning
    • 27.1 Root finding
      • 27.1.1 Newton’s method (Newton–Raphson algorithm)
      • 27.1.2 Gauss–Newton algorithm
      • 27.1.3 Gradient Descent
      • 27.1.4 Conjugate gradient method
      • 27.1.5 quasi-Newton method
      • 27.1.6 Nelder–Mead method
      • 27.1.7 Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm
      • 27.1.8 Davidon–Fletcher–Powell (DFP) formula
      • 27.1.9 Berndt–Hall–Hall–Hausman algorithm
    • 27.2 Classification Method
      • 27.2.1 Regression
      • 27.2.2 KNN
      • 27.2.3 Naive Bayes
      • 27.2.4 LDA, QDA
      • 27.2.5 Desicion Tree and Random Forest
      • 27.2.6 Bagging
      • 27.2.7 Boosting
      • 27.2.8 Neighborhood components analysis (NCA)
      • 27.2.9 Soft independent modelling of class analogies
    • 27.3 Regression Method
      • 27.3.1 SVR
    • 27.4 Clustering Method
      • 27.4.1 K-means, K-medoids
      • 27.4.2 Fuzzy C-mean
      • 27.4.3 Mean Shift
      • 27.4.4 Hierarchical Clustering
      • 27.4.5 DBSCAN
      • 27.4.6 Principal Co-ordinates Analysis (PCoA)
      • 27.4.7 Isolation Forest
      • 27.4.8 Self-organizing map (SOM)
      • 27.4.9 Spectral clustering
      • 27.4.10 Quantum clustering
    • 27.5 Model Selection
      • 27.5.1 Information Criteria
  • 28 Statistical Computing
    • 28.1 Generate random variables
      • 28.1.1 Inverse transform method
      • 28.1.2 Accept-Rejection method
      • 28.1.3 Importance Sampling
    • 28.2 Variance reduction
    • 28.3 Gibbs sampling
    • 28.4 Metropolis-Hastings
    • 28.5 Monte-Carlo and Markov chain (MCMC)
    • 28.6 Statistics Algorithm
      • 28.6.1 Smoothing
      • 28.6.2 Simulated annealing
      • 28.6.3 EM algorithm
      • 28.6.4 Back-fitting algorithm
    • 28.7 Evolutionary Algorithm
      • 28.7.1 Particle Swarm Optimization (PSO)
      • 28.7.2 Genetic Algorithm
      • 28.7.3 Approximate Bayesian computation
    • 28.8 Numerical linear algebra
  • 29 Advanced Machine Learning
    • 29.1 Double Machine Learning
    • 29.2 Adversarial machine learning (AML)
    • 29.3 Reinforcement Learning
    • 29.4 Curriculum learning
    • 29.5 Rule-based machine learning
    • 29.6 Online machine learning
    • 29.7 Few-Shot Learning
    • 29.8 Automated machine learning (AutoML)
    • 29.9 Active learning (Optimal experimental design)
    • 29.10 Online machine learning
    • 29.11 Computational Learning Theory
      • 29.11.1 Inductive Bias
      • 29.11.2 Probably Approximately Correct (PAC)
    • 29.12 Explainable AI (XAI)
  • 30 Deep Learning
    • 30.1 Basic concept
    • 30.2 DNN
    • 30.3 CNN
    • 30.4 RNN
      • 30.4.1 Long Short-Term Memory (LSTM)
      • 30.4.2 Gated Recurrent Unit (GRU)
      • 30.4.3 Temporal Convolutional Network (TCN)
    • 30.5 Autoencoders & Variational Autoencoders (VAEs)
    • 30.6 Generative adversarial networks (GAN)
    • 30.7 Transformer Networks
    • 30.8 Graph Neural Networks (GNNs)
    • 30.9 Physics-informed neural networks (PINNs)
    • 30.10 Deep Q-Networks (DQNs)
    • 30.11 Quantum neural network (QNN)
    • 30.12 Some famous models
      • 30.12.1 LeNet、AlexNet、VGG、NiN
      • 30.12.2 GoogLeNet
      • 30.12.3 ResNet
      • 30.12.4 DenseNet
      • 30.12.5 U-Net
      • 30.12.6 YOLO
    • 30.13 Knowledge Distillation
    • 30.14 Modern NN models
      • 30.14.1 Deep Operator Network
      • 30.14.2 Liquid Neural Network (LNN)
      • 30.14.3 Kolmogorov-Arnold Networks (KAN)
      • 30.14.4 Large Language Model (LLM)
  • Part IX: Computer Science Skills
  • 31 Data Structure and Algorithm
    • 31.1 Data Structure
      • 31.1.1 Linked list
      • 31.1.2 Satck
      • 31.1.3 Queue
      • 31.1.4 Tree
    • 31.2 Algorithm
      • 31.2.1 Graph and tree traversal algorithms
      • 31.2.2 Dynamic Programming
      • 31.2.3 Mathematical algorithm
  • 32 Information Theory
    • 32.1 Entropy
    • 32.2 Data compression
  • 33 Big Data Analytics Techniques
    • 33.1 Visualization
    • 33.2 Statistical method for big data
    • 33.3 Hadoop
    • 33.4 Spark
  • 34 Quantum Computing
    • 34.1 Basic Concept
    • 34.2 Quantum Algorithm
    • 34.3 Quantum Machine Learning
  • Part X: Data Communication
  • 35 Data Processing
    • 35.1 Data Collection
    • 35.2 Data Preprocessing
      • 35.2.1 Data Cleaning
      • 35.2.2 Handling Missing Data
      • 35.2.3 Normalization & Standardization
      • 35.2.4 Feature Engineering
  • 36 Data Visualization
    • 36.1 Why we need DataVis?
    • 36.2 Visual Analytics
    • 36.3 Estimation statistics
      • 36.3.1 Gardner–Altman plot
    • 36.4 Sina plot
    • 36.5 Cumming plot
    • 36.6 Radar chart
    • 36.7 Parallel coordinates
    • 36.8 Streamplots
    • 36.9 Andrews plot
    • 36.10 Spaghetti plot
    • 36.11 Fish plot
    • 36.12 Volcano plot
    • 36.13 Circle Packing Chart
    • 36.14 Chord diagram (information visualization)
    • 36.15 Climate spiral and Warming stripes
    • 36.16 Bland–Altman plot
    • 36.17 Cherry Blossom Front
    • 36.18 Symbolic data analysis (SDA)
    • 36.19 Cartogram
  • 37 Data Mining
    • 37.1 Structured Data
    • 37.2 Semi-Structured and Unstructured Data
    • 37.3 SQL
    • 37.4 NoSQL
    • 37.5 Association rule learning
      • 37.5.1 Apriori Algorithm
      • 37.5.2 ECLAT Algorithm
      • 37.5.3 FP-growth algorithm
    • 37.6 Anomaly detection
  • 38 Data Ethics and Philosophy
    • 38.1 Benford’s Law
    • 38.2 Differential Privacy
      • 38.2.1 DP-SGD
    • 38.3 Machine Unlearning
    • 38.4 Threat modeling
    • 38.5 Attack
  • Part XI: Modern Data Analysis
  • 39 Network Analysis
    • 39.1 Random Graph
      • 39.1.1 Urchin Tracking Module (UTM)
  • 40 Semantic Analysis
  • 41 Audio Analysis
  • 42 Image and Video Analysis
  • Part XII: Data Workflow
  • 43 Data Management and Integration
    • 43.1 Database
    • 43.2 Data Compression
    • 43.3 Data Integration
    • 43.4 Data Quality
    • 43.5 DataOps
  • 44 Multimodal Data Analysis
    • 44.1 Meta Analysis
    • 44.2 Federated Learning
    • 44.3 Data Fusion
  • 45 Statistical Consulting
    • 45.1 Garbage in, garbage out
  • Part XIII: Statistic Theory
  • 46 Statistical Inference
    • 46.1 Frequentist inference
      • 46.1.1 Estimatation
    • 46.2 Robust Statistics
    • 46.3 Fiducial Inference
  • 47 Probability Theory
    • 47.1 Basics from Measure Theory
    • 47.2 Limit of the sets
    • 47.3 Probability Inequalities
    • 47.4 Bertrand Paradox
    • 47.5 Stochastic ordering
    • 47.6 Malliavin Calculus
    • 47.7 CLT
    • 47.8 Regular conditional probability
      • 47.8.1 Markov kernel
    • 47.9 Martingale
      • 47.9.1 Reverse martingale
  • 48 Algebraic Statistics
    • 48.1 Free Probability Theory
  • Part XIV: Miscellaneous
  • 49 Statistical Education
    • 49.1 Stories
      • 49.1.1 Gambler’s fallacy
      • 49.1.2 Gambler’s ruin
      • 49.1.3 Buffon’s needle problem
      • 49.1.4 Simpson’s paradox
      • 49.1.5 Berkson’s paradox
      • 49.1.6 Lindley’s paradox
      • 49.1.7 Freedman’s paradox
      • 49.1.8 Texas sharpshooter fallacy
      • 49.1.9 Survivorship bias
      • 49.1.10 All models are wrong
      • 49.1.11 Stein’s phenomenon
      • 49.1.12 German tank problem
      • 49.1.13 Lindy effect
      • 49.1.14 Doomsday argument
      • 49.1.15 Stanine
  • 50 R Advanced programming
    • 50.1 Technique for Basic operator
    • 50.2 Special operator
      • 50.2.1 Inner function
      • 50.2.2 Super assignment <<-
    • 50.3 Pipe operator
      • 50.3.1 User define pipe operator
    • 50.4 Non-standard Evaluation (NSE)
      • 50.4.1 Tidy evaluation
    • 50.5 Functional programming
      • 50.5.1 Helper function
    • 50.6 Progress bar
    • 50.7 Parallel computing
  • Appendix
  • A Matrix calculus
  • Published with bookdown

Mastering Statistics with R

38.2 Differential Privacy

38.2.1 DP-SGD

https://arxiv.org/abs/1607.00133 https://www.tensorflow.org/responsible_ai/privacy/tutorials/classification_privacy https://medium.com/pytorch/differential-privacy-series-part-1-dp-sgd-algorithm-explained-12512c3959a3