multipleoutputs - Keras Single Input Multiple Outputs - why losses are so high comparing with Single Output?

Question

I think something wrong with my Keras multiple outputs coding, which causes a high loss comparing with the Sequential model. Please help me out which part of it is wrong.

import os, random, string, pandas, math, numpy
import tensorflow as tf
from tensorflow import keras

Training data:

feature_data = [] # common feature data
label_data = [] # for multiple outputs
single_data = [] # for single output
size = 10000
features = ['x1', 'x2']
labels = ['y1', 'y2']
for i in range(size):
    a = random.random()
    b = random.random()
    c = math.sin(a)
    d = math.cos(b)
    feature = [a, b]
    label = [c, d]
    feature_data.append(feature)
    label_data.append(label)
    single_data.append(c)

This is my single output model, which is working well: loss < 2e-05

single = keras.Sequential([
    keras.layers.Dense(2, input_shape=(2,), activation=tf.nn.softmax),
    keras.layers.Dense(4, activation=tf.nn.softmax),
    keras.layers.Dense(1)])
optimizer = tf.optimizers.RMSprop(learning_rate=0.001)
single.compile(loss='mse', optimizer=optimizer, metrics=['mae'])
single.fit(x=feature_data, y=single_data, epochs=100, batch_size=100)

This should be the identical multiple output model, but the losses are really high: 0.1

def build_model():
    input_shape=(2, )
    inputs = keras.Input(shape=input_shape)
    outputs = []
    for label in labels:
        u = keras.layers.Dense(2, input_shape=input_shape, activation=tf.nn.softmax)(inputs)
        v = keras.layers.Dense(4, activation=tf.nn.softmax)(u)
        w = keras.layers.Dense(1, name=label)(v)
        outputs.append(w)
    model = keras.Model(inputs = inputs, outputs = outputs)
    optimizer = tf.optimizers.RMSprop(learning_rate=0.001)
    model.compile(loss='mse', optimizer=optimizer, metrics=['mae'])
    return model

model = build_model()
model.fit(x=feature_data, y=label_data, epochs=100, batch_size=100)

I guess that something wrong with the input layer, or the label data format, but still have no idea how to fix it. Please help.

question from:https://stackoverflow.com/questions/65906978/keras-single-input-multiple-outputs-why-losses-are-so-high-comparing-with-sing

Answer 1

[Ignore this] The second model is not the same as the first model: it uses a different Dense layer at the beginning.

This structure matches the first model:

def build_model():
    input_shape=(2, )
    inputs = keras.Input(shape=input_shape)
    outputs = []
    d = keras.layers.Dense(2, input_shape=input_shape, activation=tf.nn.softmax)
    for label in labels:
        u = d(inputs)
        v = keras.layers.Dense(4, activation=tf.nn.softmax)(u)
        w = keras.layers.Dense(1, name=label)(v)
        outputs.append(w)
    model = keras.Model(inputs = inputs, outputs = outputs)
    optimizer = tf.optimizers.RMSprop(learning_rate=0.001)
    model.compile(loss='mse', optimizer=optimizer, metrics=['mae'])
    return model

The structure matches the first model because it uses the same dense layer (the exact same weights) to interpret the input in both chains.

Try plotting the loss history from your second network. You might see dual oscillating values: one gets good, the other gets bad, and they trade places.