Обучение нейронной сети теореме Пифагора

#python #нейронные_сети #машинное_обучение #keras #обучение_с_подкреплением

Хочу обучить НС теореме Пифагора. Всё вроде правильно сделал, нормализовал данные,
модель правильная, а вот в чём ошибка не понятно..

import numpy as np

from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.utils import np_utils

np.random.seed()

NB_EPOCH = 500
VERBOSE = 1

X_in = [[ 0 , 44 ], [ 0 , 18 ], [ 38 , 0 ], [ 48 , 14 ], [ 0 , 36 ], [ 14 , 0 ],
[ 34 , 0 ], [ 0 , 0 ], [ 0 , 38 ], [ 32 , 0 ], [ 28 , 0 ], [ 36 , 0 ], [ 20 , 48 ],
[ 0 , 6 ], [ 0 , 20 ], [ 0 , 42 ], [ 0 , 8 ], [ 24 , 32 ], [ 4 , 0 ], [ 6 , 8 ], [
24 , 10 ], [ 0 , 22 ], [ 16 , 12 ], [ 30 , 40 ], [ 0 , 32 ], [ 0 , 32 ], [ 16 , 0 ],
[ 48 , 20 ], [ 0 , 8 ], [ 32 , 0 ], [ 0 , 46 ], [ 0 , 22 ], [ 0 , 8 ], [ 10 , 24 ],
[ 0 , 36 ], [ 14 , 0 ], [ 0 , 22 ], [ 42 , 0 ], [ 16 , 12 ], [ 40 , 30 ], [ 44 , 0
], [ 40 , 0 ], [ 34 , 0 ], [ 0 , 32 ], [ 40 , 30 ], [ 32 , 0 ], [ 0 , 30 ], [ 24 ,
18 ], [ 0 , 26 ], [ 22 , 0 ], [ 0 , 4 ], [ 16 , 0 ], [ 10 , 0 ], [ 0 , 32 ], [ 0 ,
42 ], [ 2 , 0 ], [ 0 , 38 ], [ 32 , 24 ], [ 48 , 0 ], [ 20 , 0 ], [ 0 , 18 ], [ 0 ,
38 ], [ 14 , 48 ], [ 40 , 42 ], [ 16 , 12 ], [ 26 , 0 ], [ 0 , 20 ], [ 40 , 30 ], [
16 , 30 ], [ 36 , 48 ], [ 36 , 0 ], [ 18 , 24 ], [ 34 , 0 ], [ 16 , 0 ], [ 0 , 24 ],
[ 0 , 24 ], [ 0 , 18 ], [ 38 , 0 ], [ 28 , 0 ], [ 0 , 34 ], [ 0 , 36 ], [ 24 , 32 ],
[ 16 , 30 ], [ 40 , 30 ], [ 24 , 0 ], [ 0 , 14 ], [ 8 , 6 ], [ 12 , 0 ], [ 16 , 0 ],
[ 16 , 30 ], [ 48 , 14 ], [ 0 , 30 ], [ 38 , 0 ], [ 38 , 0 ], [ 0 , 8 ], [ 36 , 48
], [ 0 , 32 ], [ 10 , 24 ], [ 46 , 0 ], [ 24 , 10 ], [ 30 , 0 ], [ 0 , 48 ], [ 40 ,
0 ], [ 42 , 0 ], [ 32 , 24 ], [ 32 , 0 ], [ 12 , 16 ], [ 0 , 4 ], [ 0 , 28 ], [ 32
, 0 ], [ 40 , 42 ], [ 46 , 0 ], [ 0 , 24 ], [ 30 , 16 ], [ 36 , 48 ], [ 40 , 0 ], [
24 , 0 ], [ 0 , 22 ], [ 40 , 42 ], [ 10 , 24 ], [ 0 , 16 ], [ 14 , 48 ], [ 22 , 0 ],
[ 0 , 22 ], [ 30 , 0 ], [ 0 , 2 ], [ 48 , 20 ], [ 6 , 0 ], [ 6 , 0 ], [ 28 , 0 ], [
20 , 0 ], [ 0 , 40 ], [ 42 , 0 ], [ 48 , 36 ], [ 14 , 0 ], [ 10 , 24 ], [ 0 , 30 ],
[ 48 , 20 ], [ 40 , 30 ], [ 0 , 0 ], [ 42 , 40 ], [ 0 , 48 ], [ 32 , 24 ]]
X_answer = [[44] ,[18] ,[38] ,[50] ,[36] ,[14] ,[34] ,[0] ,[38] ,[32] ,[28] ,[36]
,[52] ,[6] ,[20] ,[42] ,[8] ,[40] ,[4] ,[10] ,[26] ,[22] ,[20] ,[50] ,[32] ,[32] ,[16]
,[52] ,[8] ,[32] ,[46] ,[22] ,[8] ,[26] ,[36] ,[14] ,[22] ,[42] ,[20] ,[50] ,[44] ,[40]
,[34] ,[32] ,[50] ,[32] ,[30] ,[30] ,[26] ,[22] ,[4] ,[16] ,[10] ,[32] ,[42] ,[2] ,[38]
,[40] ,[48] ,[20] ,[18] ,[38] ,[50] ,[58] ,[20] ,[26] ,[20] ,[50] ,[34] ,[60] ,[36]
,[30] ,[34] ,[16] ,[24] ,[24] ,[18] ,[38] ,[28] ,[34] ,[36] ,[40] ,[34] ,[50] ,[24]
,[14] ,[10] ,[12] ,[16] ,[34] ,[50] ,[30] ,[38] ,[38] ,[8] ,[60] ,[32] ,[26] ,[46]
,[26] ,[30] ,[48] ,[40] ,[42] ,[40] ,[32] ,[20] ,[4] ,[28] ,[32] ,[58] ,[46] ,[24]
,[34] ,[60] ,[40] ,[24] ,[22] ,[58] ,[26] ,[16] ,[50] ,[22] ,[22] ,[30] ,[2] ,[52]
,[6] ,[6] ,[28] ,[20] ,[40] ,[42] ,[60] ,[14] ,[26] ,[30] ,[52] ,[50] ,[0] ,[58] ,[48]
,[40]]
X_in = np.asarray(X_in, dtype=np.float32)
X_answer = np.asarray(X_answer, dtype=np.float32)

X_in /= np.amax(X_in)
X_answer /= np.amax(X_answer)

model = Sequential()
model.add(Dense(10, input_dim = 2, activation='relu'))
model.add(Dense(10, activation='relu'))
model.add(Dense(1, activation='softmax'))

model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])

history = model.fit(X_in, X_answer, epochs=NB_EPOCH, verbose=VERBOSE)


Что до 100 эпох, что до 500 один и тот же результат:

Epoch 1/100
143/143 [==============================] - 0s 2ms/step - loss: 0.2966 - acc: 0.0280
Epoch 2/100
143/143 [==============================] - 0s 52us/step - loss: 0.2966 - acc: 0.0280
Epoch 3/100
143/143 [==============================] - 0s 52us/step - loss: 0.2966 - acc: 0.0280
Epoch 4/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 5/100
143/143 [==============================] - 0s 52us/step - loss: 0.2966 - acc: 0.0280
Epoch 6/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 7/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 8/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 9/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 10/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 11/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 12/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 13/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 14/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 15/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 16/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 17/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 18/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 19/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 20/100
143/143 [==============================] - 0s 35us/step - loss: 0.2966 - acc: 0.0280
Epoch 21/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 22/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 23/100
143/143 [==============================] - 0s 35us/step - loss: 0.2966 - acc: 0.0280
Epoch 24/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 25/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 26/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 27/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 28/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 29/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 30/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 31/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 32/100
143/143 [==============================] - 0s 52us/step - loss: 0.2966 - acc: 0.0280
Epoch 33/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 34/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 35/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 36/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 37/100
143/143 [==============================] - 0s 49us/step - loss: 0.2966 - acc: 0.0280
Epoch 38/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 39/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 40/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 41/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 42/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 43/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 44/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 45/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 46/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 47/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 48/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 49/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 50/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 51/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 52/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 53/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 54/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 55/100
143/143 [==============================] - 0s 49us/step - loss: 0.2966 - acc: 0.0280
Epoch 56/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 57/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 58/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 59/100
143/143 [==============================] - 0s 35us/step - loss: 0.2966 - acc: 0.0280
Epoch 60/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 61/100
143/143 [==============================] - 0s 35us/step - loss: 0.2966 - acc: 0.0280
Epoch 62/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 63/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 64/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 65/100
143/143 [==============================] - 0s 52us/step - loss: 0.2966 - acc: 0.0280
Epoch 66/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 67/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 68/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 69/100
143/143 [==============================] - 0s 52us/step - loss: 0.2966 - acc: 0.0280
Epoch 70/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 71/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 72/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 73/100
143/143 [==============================] - 0s 39us/step - loss: 0.2966 - acc: 0.0280
Epoch 74/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 75/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 76/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 77/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 78/100
143/143 [==============================] - 0s 49us/step - loss: 0.2966 - acc: 0.0280
Epoch 79/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 80/100
143/143 [==============================] - 0s 49us/step - loss: 0.2966 - acc: 0.0280
Epoch 81/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 82/100
143/143 [==============================] - 0s 35us/step - loss: 0.2966 - acc: 0.0280
Epoch 83/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 84/100
143/143 [==============================] - 0s 35us/step - loss: 0.2966 - acc: 0.0280
Epoch 85/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 86/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 87/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 88/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 89/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 90/100
143/143 [==============================] - 0s 49us/step - loss: 0.2966 - acc: 0.0280
Epoch 91/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 92/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 93/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280
Epoch 94/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 95/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 96/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 97/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 98/100
143/143 [==============================] - 0s 45us/step - loss: 0.2966 - acc: 0.0280
Epoch 99/100
143/143 [==============================] - 0s 42us/step - loss: 0.2966 - acc: 0.0280
Epoch 100/100
143/143 [==============================] - 0s 38us/step - loss: 0.2966 - acc: 0.0280

    

Ответы

Ответ 1

Как уже сказал @L.Murashov функция активации softmax применяется в задачах мультиклассовой
классификации - она помогает посчитать вероятности принадлежности образца каждому из
классов.

Есть еще несколько моментов, на которые стоит обратить внимание:


В обучающей выборке слишком много (около половины) нулевых значений - это вырожденный
случай для теоремы Пифагора, когда одна из сторон треугольника имеет нулевую длину.
Для того, чтобы лучше обучить модель можно выборку взять побольше - в вашем случае
не проблема сгенерировать столько данных сколько нужно.
Для данной (простой) задачи хватит одного скрытого слоя
В качестве функции активации выходного слоя можно использовать linear
В качестве функции потери и метрики можно выбрать mean_squared_error  




Пример:

import numpy as np
from keras.models import Sequential
from keras.layers.core import Dense, Activation

N = 5000
np.random.seed(1234)
X = np.random.randint(0, 50, size=(N,2))
y = np.linalg.norm(X, axis=1)

NB_EPOCHS = 100
VERBOSE = 1

model = Sequential()
model.add(Dense(20, input_dim = 2, activation='relu'))
model.add(Dense(1, activation='linear'))
model.compile(loss='mse', optimizer='adam', metrics=['mean_squared_error'])
model.fit(X, y, epochs=NB_EPOCHS, verbose=VERBOSE)


Обучение:

...
5000/5000 [==============================] - 0s 22us/step - loss: 0.0057 - mean_squared_error:
0.0057
Epoch 98/100
5000/5000 [==============================] - 0s 22us/step - loss: 0.0048 - mean_squared_error:
0.0048
Epoch 99/100
5000/5000 [==============================] - 0s 22us/step - loss: 0.0045 - mean_squared_error:
0.0045
Epoch 100/100
5000/5000 [==============================] - 0s 22us/step - loss: 0.0043 - mean_squared_error:
0.0043
Out[70]: 


Предсказание:

In [71]: model.predict(np.array([[3,4], [10,10], [5,6]]))
Out[71]:
array([[ 5.018393],
       [14.130004],
       [ 7.841759]], dtype=float32)


Ответ 2

У вас неправильная функция активации на последнем слое. Softmax ограничен [0..1]

model.add(Dense(1, activation='softmax'))


заменить на

# После слоя не применяется функция активации
model.add(Dense(1))


Еще accuracy здесь бесполезна т.к. задача регрессии.

metrics=['accuracy']