New feature

• A new BigDL document website online https://bigdl-project.github.io/, which replace the original BigDL wiki
• Added New Models & Layers
  • TreeLSTM and examples for sentiment analytics
  • convLSTM layer
  • 1D convolution layer
  • Mean Absolute Error (MAE) metrics
  • TimeDistributed Layer
  • VolumetricConvolution(3D convolution)
  • VolumetricMaxPooling
  • RoiPooling layer
  • DiceCoefficient loss
  • bi-recurrent layers
• API change
  • Allow user to set regularization per layer
  • Allow user to set learning rate per layer
  • Add predictClass API for python
  • Add DLEstimator for Spark ML pipeline
  • Add Functional API for model definition
  • Add movie length dataset API
  • Add 4d normalize support
  • Add evaluator API to simplify model test
• Install & Deploy
  • Allow user to install BigDL from pip
  • Support win64 platform
  • A new script to auto pack/distribute python dependency on yarn cluster mode
• Model Save/Load
  • Allow user to save BigDL model as Caffe model file
  • Allow user to load/save some Tensorflow model(cover tensorflow slim APIs)
  • Support save/load model file from/to s3/hdfs
• Optimization
  • Add plateau learning rate schedule
  • Allow user to adjust optimization process based on loss and score
  • Add Exponential learning rate decay
  • Add natural exp decay learning rate schedule
  • Add multistep learning rate policy

Enhancement

1. Optimization method API refactor
2. Allow user to load a Caffe model without pre-defining a BigDL model
3. Optimize Recurrent Layers performance
4. Refine the ML pipeline related API, and add more examples
5. Optimize JoinTable layer performance
6. Allow user to use nio blockmanager on Spark 1.5
7. Refine layer parameter initialization algorithm API
8. Refine Sample class to save memory usage when cache train/test dataset as tensor format
9. Refine MiniBatch API to support padding and multiple tensors
10. Remove bigdl.sh. BigDL will set MKL behavior through MKL Java API, and user can control this via Java properties
11. Allow user to remove Spark log in redirecting log file
12. Allow user create a SpatialConvultion layer without bias
13. Refine validation metrics API
14. Refine smoothL1Criterion and reduce tensor storage usage
15. Use reflection to handle difference of Spark2 platforms, and user need not to recompile BigDL for different Spark2 platform
16. Optimize FlattenTable performance
17. Use maven package instead of script to copy dist artifacts together

Bug Fix

1. Fix some error in Text-classifier document
2. Fix a bug when call JoinTable after clearState()
3. Fix a bug in Concat layer when the dimension concatenated along is larger than 2
4. Fix a bug in MapTable layer
5. Fix some multi-thread error not catch issue
6. Fix maven artifact dependency issue
7. Fix model save method won’t close the stream issue
8. Fix a bug in BCECriterion
9. Fix some ConcatTable don’t clear gradInput buffer
10. Fix SpatialDilatedConvolution not clear gradInput content