Machine Learning in Elixir: Chapter 1, Poor Accuracy when following the code in book

Community PragProg Customers
,
1

Hey @seanmor5,

I’m having great training accuracy but poor evaluation accuracy for the example in Chapter 1 when following the code in the book.

Epoch: 9, Batch: 450, accuracy: 0.9750040 loss: 0.2519934

Batch: 0, accuracy: 0.0000000
%{
  0 => %{
    "accuracy" => #Nx.Tensor<
      f32
      0.0
    >
  }
}

After some trouble, I realised that the book’s code diverges from the accompanying livebooks from PragProg. Here’s the code in question.

Book, Page 16:

...
train_categories =
  train_df["species"]
  |> Explorer.Series.cast(:category)

y_train =
  train_categories
  |> Nx.stack(axis: -1)
  |> Nx.equal(Nx.iota({1, 3}, axis: -1))

x_test = Nx.stack(test_df[feature_columns], axis: 1)

test_categories =
  test_df["species"]
  |> Explorer.Series.cast(:category)

y_test =
  test_categories
  |> Nx.stack(axis: -1)
  |> Nx.equal(Nx.iota({1, 3}, axis: -1))

Accompanying Livebook:

...
y_train =
  train_df
  |> DF.pull(label_column)
  |> Explorer.Series.to_list()
  |> Enum.map(fn
    "Iris-setosa" -> 0
    "Iris-versicolor" -> 1
    "Iris-virginica" -> 2
  end)
  |> Nx.tensor(type: :u8)
  |> Nx.new_axis(-1)
  |> Nx.equal(Nx.iota({1, 3}, axis: -1))
...
y_test =
  test_df
  |> DF.pull(label_column)
  |> Explorer.Series.to_list()
  |> Enum.map(fn
    "Iris-setosa" -> 0
    "Iris-versicolor" -> 1
    "Iris-virginica" -> 2
  end)
  |> Nx.tensor(type: :u8)
  |> Nx.new_axis(-1)
  |> Nx.equal(Nx.iota({1, 3}, axis: -1))

Seems like it had to do with the ordering of the categories and how it maps when doing the one-hot encoding.

There’s a thread started on Elixirforum where @grossvogel more succintly explains what’s happening & with some alternative code.

2

Thanks for pointing this out, I’ve updated the code and language for the next beta. The approach is to cast the entire dataframe to a categorical variable before splitting/shuffling, this ensures that the one-hot encoding across train/test sets are consistent :slight_smile:

2 Likes