Title: Machine Learning in Elixir: Chapter 1
Example: Flower classification - final accuracy on the test data is inconsistent.
When completing chapter 1 and evaluating our trained model on the test data I find the results can be inconsistent based on the shuffle of the original dataset. My first pass got 0% accuracy on the test data, but 96% accuracy when I evaluated using the training data. My second attempt (after re-running the Livebook cell that shuffled the training set) my model got 40% accuracy on the test data. I’ve tried various iterations of set sizes, iterations, epochs, etc but I cannot get a trained model to be more than 40% accurate on the test data. Is there some way to make the results more consistent?
I’m hitting the same issue, very much seems to depend on the random train vs test split, but most of the time my trained model accuracy is >96% but my test evaluation is < 40%
I’ve been comparing to a kaggle notebook I found exploring the same iris dataset that seems to get 100% accuracy on both training and evaluation sets, it has 3 main differences
- it uses normalization instead of standardization
- it uses the :adam optimizer instead of :sgd
- it has multiple dense layers (and a dropout layer) in its NN
I’ve tried to make the Elixir/Axon example more closely match that Kaggle/Keras/Tensorflow example but still getting very low test evaluation…
So I’m actually not much help here 
just reporting my experience so far and looking forward to seeing if this conversation expands.
(and generally enjoying the book!)
… also just noticed a similar forum post that has a potential answer from Sean (I haven’t tried it yet)
1 Like
Yes, having now tried it I can confirm Sean’s answer in the other forum post DOES resolve the issue for me and I now consistently get > 96% accuracy in test evaluation no matter the splits.
Thanks Sean!
1 Like
Oh excellent! Thanks for pointing me to that other post.
1 Like
