text:
Tf2 porting

@w4nderlust   This is a test of sending results of my work.  Initially I tried pushing my changes directly to uber/ludwig but it failed because I did not have write permission to uber/ludwig.
I'm now trying to submit my changes through a PR for branch tf2_porting branch in my forkjimthompson5802/ludwig.  The target for this PR is uber/ludwig' branch 'tf2_porting.  If this works, then I can just add commits to this PR.
The Docker image I use for my development environment is built with the updated requirements.txt on the tf2_porting branch, which contains
tensorflow==2.1
tensorflow-addons

With this initial set of commits, train completes and some, not all, of the data are saved for TensorBoard.  For my test, I'm using the Titanic Survivor example.  Here is the log from training.
tf2_sample_train_log.txt
Here is screenshot of TensorBoard for the data that was collected.

Let me know what you think.

I thought it would be good to establish a baseline for the unit tests.  In doing this, I noticed some strange behavior.  The initial run of the unit tests led to this result:
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /opt/project
plugins: typeguard-2.7.1
collected 81 items

../../tests/integration_tests/test_api.py .                              [  1%]
../../tests/integration_tests/test_contrib_wandb.py FE                   [  2%]
../../tests/integration_tests/test_experiment.py EEEEEEEEEEEEEEEEEEEEEEEE [ 17%]
EEEEEEEEEEEEEE                                                           [ 25%]
../../tests/integration_tests/test_kfold_cv.py EEEEEE                    [ 29%]
../../tests/integration_tests/test_server.py EE                          [ 30%]
../../tests/integration_tests/test_visualization.py EEEEEEEEEEEEEEEEEEEE [ 43%]
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE                                         [ 62%]
../../tests/integration_tests/test_visualization_api.py EEEEEEEEEEEEEEEE [ 72%]
EEEEEEEEEEEEEEEEEEEEEEEEEEEE                                             [ 90%]
../../tests/ludwig/models/modules/test_encoder.py EEEEEEEE               [ 95%]
../../tests/ludwig/utils/test_data_utils.py EE                           [ 96%]
../../tests/ludwig/utils/test_image_utils.py EEEE                        [ 98%]
../../tests/ludwig/utils/test_normalization.py EE                        [100%]

==================================== ERRORS ====================================

This is the complete log file
pytest_all_output.txt
All of the tests failed with this error:
____________ ERROR at teardown of test_experiment_sequence_combiner ____________

self = <contextlib._GeneratorContextManager object at 0x7fc6dc29f5f8>
type = None, value = None, traceback = None

    def __exit__(self, type, value, traceback):
        if type is None:
            try:
>               next(self.gen)
E               OSError: [Errno 29] Illegal seek

/usr/lib/python3.6/contextlib.py:88: OSError

Not clear, at least to me, how this may be related to TF2.  I'll note the travis-ci run encounters the same problem as above.
Have you encountered this kind of problem with test_contrib_wandb.py before?
This is the results of the pytest run if  test_contrib_wandb.py is removed.
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /opt/project
plugins: typeguard-2.7.1
collected 80 items

../../tests/integration_tests/test_api.py .                              [  1%]
../../tests/integration_tests/test_experiment.py ....FF..FFF..FF....     [ 25%]
../../tests/integration_tests/test_kfold_cv.py ...                       [ 28%]
../../tests/integration_tests/test_server.py F                           [ 30%]
../../tests/integration_tests/test_visualization.py .................... [ 55%]
......                                                                   [ 62%]
../../tests/integration_tests/test_visualization_api.py ................ [ 82%]
......                                                                   [ 90%]
../../tests/ludwig/models/modules/test_encoder.py FF..                   [ 95%]
../../tests/ludwig/utils/test_data_utils.py .                            [ 96%]
../../tests/ludwig/utils/test_image_utils.py ..                          [ 98%]
../../tests/ludwig/utils/test_normalization.py .                         [100%]

Here is the complete log file for this run:
pytest_output_initial.txt
Whatever the issue is with test_contrib_wandb.py it appears to corrupt the run-time environment for pytest.

Commit https://github.com/uber/ludwig/pull/639/commits/1dbfdfbd04dc1f9fca4344941870f30528c9f089  disables running tests/integration_tests/test_contrib_wandb.py  temporarily.  Once the issue with this test and TF2 is resolved, the test should be re-enabled.

@w4nderlust  a question...this code fragment in ludwig/models/modules/loss_module.py is the reference tf2.keras.... guidance to me to focus on getting these functions to work or is just an artifact of earlier testing and not specific guidance on the approach?
https://github.com/uber/ludwig/blob/21b5f296e5955130a8826a7597b1ffd1201db4f3/ludwig/models/modules/loss_modules.py#L324-L328

Have you encountered this kind of problem with test_contrib_wandb.py before?

No never. I think it's fine to disable the test temporarily and then to re-add it when the rest of the stuff is done. There are not so many tests failing, I expected worse to be honest, but there's still quite some work to do.
Regarding the regularizers, my understanding is that in TF2 all regularizers are actually keras regularizers. So I tried to replace the old ones with the keras ones, but it did not work. So I believe we can ignore the regularizers for now, port everything to TF2 and only later reintroduce them.

Commit https://github.com/uber/ludwig/pull/639/commits/6689168b5b8a23c814d79c915d5ddb6eb5f98ccd eliminates 5 out of 10 errors in the unit tests.  Local running of unit tests:
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /opt/project
plugins: typeguard-2.7.1
collected 80 items

../../tests/integration_tests/test_api.py .                              [  1%]
../../tests/integration_tests/test_experiment.py ....F...FFF..F.....     [ 25%]
../../tests/integration_tests/test_kfold_cv.py ...                       [ 28%]
../../tests/integration_tests/test_server.py .                           [ 30%]
../../tests/integration_tests/test_visualization.py .................... [ 55%]
......                                                                   [ 62%]
../../tests/integration_tests/test_visualization_api.py ................ [ 82%]
......                                                                   [ 90%]
../../tests/ludwig/models/modules/test_encoder.py ....                   [ 95%]
../../tests/ludwig/utils/test_data_utils.py .                            [ 96%]
../../tests/ludwig/utils/test_image_utils.py ..                          [ 98%]
../../tests/ludwig/utils/test_normalization.py .                         [100%]

=================================== FAILURES ===================================
<<<<< DELETED LINES >>>>>
=========== 5 failed, 75 passed, 7040 warnings in 294.51s (0:04:54) ============

test_encoder.py and test_server.py are now error free.  Elimiated two errors in test_experiment.py

Commit https://github.com/uber/ludwig/pull/639/commits/b2673eb9e0ef22743989a4b511f46c06b2ae202d fixes 1 of the 5 remaining errors in test_experiment.py
Local testing:
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /opt/project
plugins: typeguard-2.7.1
collected 80 items

../../tests/integration_tests/test_api.py .                              [  1%]
../../tests/integration_tests/test_experiment.py ....F...F.F..F.....     [ 25%]
../../tests/integration_tests/test_kfold_cv.py ...                       [ 28%]
../../tests/integration_tests/test_server.py .                           [ 30%]
../../tests/integration_tests/test_visualization.py .................... [ 55%]
......                                                                   [ 62%]
../../tests/integration_tests/test_visualization_api.py ................ [ 82%]
......                                                                   [ 90%]
../../tests/ludwig/models/modules/test_encoder.py ....                   [ 95%]
../../tests/ludwig/utils/test_data_utils.py .                            [ 96%]
../../tests/ludwig/utils/test_image_utils.py ..                          [ 98%]
../../tests/ludwig/utils/test_normalization.py .                         [100%]

=================================== FAILURES ===================================
<<<<< DELETED LINES >>>>>
=========== 4 failed, 76 passed, 6972 warnings in 297.38s (0:04:57) ============

Just a quick update...3 of the 4 remaining errors in the unit tests relate to sequence related models.  re: the seq2seq test, made some progress in the sense that errors occur later in the processing.  Slowly reverse engineering conversion from tf.contrib api semantics to tfa semantics.  Commit https://github.com/uber/ludwig/pull/639/commits/3994e2c67e8a274f285e513b700483e605c8f4ce shows work to date.  Let me know if I'm heading down the wrong path.

a quick update....I'm focused on fixing the multiple seq2seq unit test.  I'm assuming getting this to work will inform how to resolve the other failing tests.
From what I can tell I've I accounted for all "input parameters" and converted the deprecated tf.contrib.seq2seq.TrainingHelper to the TF2 addons equivalent and mapped them to the equivalent in TF2.
Although multiple seq2seq test fails, the unit test fail later in processing than when I first started working on it.  I think I'm making progress.  Current error appears to involve some dependency on the legacy tf.contrib.seq2seq.TrainingHelper class attributes/methods.
I've pushed all changes to-date to this PR.  Any feedback will be appreciated, especially on if the data structures are mapped correctly to the TF2 equivalent usage.

@w4nderlust   Re: an earlier comment

There are not so many tests failing, I expected worse to be honest, but there's still quite some work to do.

You may have noticed that the number of fails in the unit test took a turn for the worse in some of the commits (https://github.com/uber/ludwig/pull/639/commits/04470ae564b47695baa7da7b009bcd7730f3fbb8 https://github.com/uber/ludwig/pull/639/commits/7e4261cb06e471de170ed623aef587f7ae8b7ebb).  This occurred because I changed line of code https://github.com/uber/ludwig/blob/2f55f8bd4922890a738d89dc10d953298037e25d/ludwig/models/modules/recurrent_modules.py#L36 to use tf2.keras.layers.SimpleRNNCell.
For the moment I went back to tf.nn.rnn_cell.BasicRNNCell to not distract my focus on getting the remaining tests converted to use the tensorflow_addons package.  However, once that is taken care of and we start making other changes, the we'll see new types of failures in the unit tests.

I've been looking at the last set of errors and could use a second set of eyes.
The tests now fail with
      try:
        c_op = c_api.TF_FinishOperation(op_desc)
      except errors.InvalidArgumentError as e:
        # Convert to ValueError for backwards compatibility.
>       raise ValueError(str(e))
E       ValueError: Shape must be rank 1 but is rank 0 for 'sequence_5CF70/predictions_sequence_5CF70/rnn_decoder/decoder/concat_3' (op: 'ConcatV2') with input shapes: [1], [], [].

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1622: ValueError

As I understand the error, the code is expecting a vector but received instead a scalar value.  Likely cause is that I mis-mapped a data structure when I converted to the tfa api set.
Here is an example error stack trace
../../tests/integration_tests/test_experiment.py:83: in run_experiment
    exp_dir_name = full_experiment(**args)
../../ludwig/experiment.py:349: in full_experiment
    **kwargs
../../ludwig/experiment.py:124: in experiment
    **kwargs
../../ludwig/train.py:354: in full_train
    debug=debug
../../ludwig/train.py:501: in train
    debug=debug
../../ludwig/models/model.py:116: in __init__
    **kwargs
../../ludwig/models/model.py:192: in __build
    is_training=self.is_training
../../ludwig/models/outputs.py:45: in build_outputs
    **kwargs
../../ludwig/models/outputs.py:98: in build_single_output
    **kwargs
../../ludwig/features/base_feature.py:314: in concat_dependencies_and_build_output
    **kwargs
../../ludwig/features/sequence_feature.py:265: in build_output
    kwarg=kwargs
../../ludwig/features/sequence_feature.py:295: in build_sequence_output
    regularizer=regularizer
../../ludwig/features/sequence_feature.py:392: in sequence_predictions
    is_timeseries=is_timeseries
../../ludwig/models/modules/sequence_decoders.py:123: in __call__
    regularizer=regularizer
../../ludwig/models/modules/recurrent_modules.py:469: in recurrent_decoder
    inputs=encoder_outputs
../../ludwig/models/modules/recurrent_modules.py:430: in decode
    decoder_init_kwargs={'initial_state': initial_state}
/usr/local/lib/python3.6/dist-packages/tensorflow_addons/seq2seq/decoder.py:346: in dynamic_decode
    decoder.output_dtype,
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/nest.py:568: in map_structure
    structure[0], [func(*x) for x in entries],
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/nest.py:568: in <listcomp>
    structure[0], [func(*x) for x in entries],
/usr/local/lib/python3.6/dist-packages/tensorflow_addons/seq2seq/decoder.py:343: in <lambda>
    _prepend_batch(decoder.batch_size, shape), dtype=dtype
/usr/local/lib/python3.6/dist-packages/tensorflow_addons/seq2seq/decoder.py:534: in _prepend_batch
    return tf.concat(([batch_size], shape), axis=0)
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/dispatch.py:180: in wrapper
    return target(*args, **kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py:1517: in concat
    return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_array_ops.py:1126: in concat_v2
    "ConcatV2", values=values, axis=axis, name=name)
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py:742: in _apply_op_helper
    attrs=attr_protos, op_def=op_def)
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:3322: in _create_op_internal
    op_def=op_def)
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1786: in __init__
    control_input_ops)

The full pytest log file if needed.
pytest_output_log2.txt

First of all, thank you for this work, it's substantial progress. The generator decoder is most likely the most complicated part of the code in Ludwig's codebase, plus it was relying on those contrib functions that were really not well documented, as I believe also the new addons ones, which makes everything more complicated. But I believe that once that part is solved, everything else should be much simpler ;)
(the other slightly more complicated part maybe the sequence and concat_sequence combiners, but if tests are not failing for those already, that-s a great sign).
I went through all the commits. It looks like when I "translated" to tf2 addons the helpers to samplers I did a few mistakes, plus I was assuming the functionalities and parameters were the same, but your commits show that it was not the case. My commits had mostly the goal of making things run without import errors to be honest, so I didn't look in detail at the new addons package, but i can see on its github page that there are some examples about how to "translate" from contrib to addons https://github.com/tensorflow/addons/blob/master/tensorflow_addons/seq2seq/README.md . I guess this is what you are following, right?
Specifically to your last message about vectors vs scalars, I can't be sure by just looking at the trace *I should reproduce the error) but usually those kind of problems used to happen in some specific combinations of parameters when using the contrib package, and I believe this information may be helpful for you also for the addons package. Specifically the are 4 scenarios: greedy decoding vs beam decoding and attention vs no attention (their dot product).
Greedy decoding usually requires only one vector as input, while the beam search requires as many vectors as the size of the beam, as you can see here: https://github.com/uber/ludwig/blob/master/ludwig/models/modules/recurrent_modules.py#L401-L417 (the batches are tiled as many times as the beam width).
The other point is about attention. As you can see here: https://github.com/uber/ludwig/blob/master/ludwig/models/modules/recurrent_modules.py#L370-L380 the attention mechanism takes as input the encoder outputs, but for the attention mechanism to work, that tensor has to be of size [batch, length, hidden]. Ludwig, when encoding a input sequence, by default reduces it to a vector (you can specify how to do it, for instance for rnn by default you take the last element of the length dimension, for other encoders there's a pooling layer), so if you want to have your full non-reduced tensor, you have to specify reduce: null in your sequence input_feature. If you look at the model definition of those tests with attention you should see it. because of this, if attention is used, the initial state of the decoder is obtained by reducing the 3d input tensor before along the second dimension: https://github.com/uber/ludwig/blob/master/ludwig/models/modules/recurrent_modules.py#L390-L391
When there's attention and beam search, obviously things are even more complicated as both constraints need to be respected.
Because of those caveats and reduce and expected tensor dimensions, that code is quite messy and I was thinking about making big changes to it, but first I wanted to port it to tf2 to avoid having to redo all the cleanup again after the tf2 port. Anyway, hope this information would be useful to diagnose the problem. A final note, the Timeseries decoding never really worked entirely before, another thing I wanted to fix in the future, so don't worry if it doesn't work now.  Timeseries output features are not listed in the doc for this reason.
Let me know if this helps and if you have more questions. I'll try to reproduce the errors myself too to figure out in what conditions they emerge.

Thank you for the response.

github page that there are some examples about how to "translate" from contrib to addons https://github.com/tensorflow/addons/blob/master/tensorflow_addons/seq2seq/README.md . I guess this is what you are following, right?

In addition the README.md, I've had to study tfa's source code and its unit test to figure out the sequence of api calls.
I appreciate your insights into how problems have manifested in the past and what sections (TimeSeries) I can ignore for the moment.
I've been using this as my primary test driver.
run_seq2seq_example.txt
Just change the file extension from txt to py.  This should cause the latest failure if you run it on the tf2_porting branch from this PR.

Looking at the script, at the moment the reduce= None part si kinda useless. If you copied that from our current tests, it  means they will have to be updated.
Let me explain. As you have moltiple input features and a concat combiner, the output of that would be a single vector. for each element of the batch. Instead what would make a difference would be to have a seuqnece_concat combiner with the sequence input feature reduce = null. In that case the output would be a length x hidden matrix for each element of the batch. In that case you would need a reduce_input value in the decoder when you don't have attention, or use attention for the decoding. I'm not sure if this is clea, it if isn't I will write some code and some illustrations to explain it, as it is actually tricky, and, again, the most tricky part of the codebase.

re:

If you copied that from our current tests, it means they will have to be updated.

Yes, I did copy that section of code from the tests/integrtion_tests/test_experiment.py.  The function name in the code I sent is the same function name in the preceding file.
Sample code would be helpful.

Let me try to explain using some drawings.
b = batch
l = sequence lenght
h = hidden size
e = embedding size
First we have a regular sequence generator:

Independently of how the input is encoded, as long as it is of shape b x h, and as this is exactly the shape required by the generator, everything works fine.
An example model definition would look like:
input_features:
  -
    name: a
    type: whatever
output_features
  -
    name: b
    type: sequence
    decoder: generator

In the test code there are multiple inputs, and a combiner, like this:

The fact that there are multiple inputs with potentially different shapes doesn't matter becausethe concat combiner takes care of their dimensions, and in the end outputs a b x h matrix, exactly like if we had a single encoder, so the generator input dimensions are respected.
An example model definition would look like:
input_features:
  -
    name: a
    type: whatever
  -
    name: c
    type: whatever
combined:
  type: concat
output_features
  -
    name: b
    type: sequence
    decoder: generator

Now let's forget for a second about multiple inputs, let's see the effect of reduce_output and reduce_input:

reduce?output is a parameter that defines how the input of a sequential encoder should be reduced. Sequential encoders usually output a b x l x h tensor, and the reduce_output paramter tells what to do with it. If =sum then values are summed along the l dimension, if =last the last element of the l dimension is selected, etc. Either way, what is returned is a b x h matrix like in the first example. Setting reduce_output=null makes it so that no reduction is applied and the full bx l x h tensor is returned.
Now, because the sequence generator requires a b x h tensor as input, there's another parameter that can be used in the generator> reduce_input. It works exactly as -reduce_output and is applied to the inputs of the generator, is it is not null, reduces acrtoss the length dimension, so in this scare is you specify sum, you'll get a b x h matrix, that is the dimension the sequence generator expects. (if you see in the test case, the other non-sequential output features use this mechanism for reducing the input) dimensions).
An example model definition would look like:
input_features:
  -
    name: a
    type: sequence
    reduce_output: null
output_features
  -
    name: b
    type: sequence
    decoder: generator
    reduce_input: sum

.Now that the role of reduction is clear, let's look at the attention mechanism:

When a sequence generator has an attention mechanism (either value), it requires the inputs to be b x l x h instead of the usual b x h. In order to achieve that, both the reduce_output of the encoder and reduce_input of the decoder have to be null. Internally the tesnor is used as input to attention here: https://github.com/uber/ludwig/blob/master/ludwig/models/modules/recurrent_modules.py#L370-L380 and it also used as input to the rnn generator (as the state needs to be a b x h matrix, it is actually reduced here): https://github.com/uber/ludwig/blob/master/ludwig/models/modules/recurrent_modules.py#L390-L391
The bottom line is: when using attention you need the l dimension, and reduce set to null achieves it.
An example model definition would look like:
input_features:
  -
    name: a
    type: sequence
    reduce_output: null
output_features
  -
    name: b
    type: sequence
    decoder: generator
    reduce_input: null
    attention_mechanism: luong

Now let's get to the most complex case, when you have multiple outputs and want to use attention:

To make this work we need to use the sequence combiner. I won't go into the details of it, but suffices to say that it requires one of  the inputs to be b x l x h (so a sequence encoder with reduce output null will do), and any number of additional inputs either of size b x ? or b x l x ?, for instance the categorical encoder in the picture witch outputs a b x e matrix. The sequence concat combiner tiles the other inputs to be of the same size of the sequence one and concatenates them along the final simension, so in the case depicted you'll obtain a b x l x h+e tensor. This shape, as we have seen before, is good for the sequential generator with attention.
An example model definition would look like:
input_features:
  -
    name: a
    type: sequence
    reduce_output: null
  -
    name: c
    type: category
combiner:
  type: sequence_concat
output_features
  -
    name: b
    type: sequence
    decoder: generator
    reduce_input: null
    attention_mechanism: luong

One caveat is the beam search. When the decoder uses a beam search, the input it requires is beam_size x b x h, that's why we do this: https://github.com/uber/ludwig/blob/master/ludwig/models/modules/recurrent_modules.py#L403-L404
This works fine without attention, last time I checked it was working fine with attention too, but with the new addons package one has to be a bit extra carefull.
Ideally we would have tests to cover the following cases: regular generator, generator with attention, generator with beam > 1, generator with attention and beam > 1.
I guess this covers almost everything, let me know if you have further questions.

Thank your for very much for the impromptu tutorial.  I'll have to re-read a few times to digest the material.   If I have questions, I'll do follow-up.
In planning my next set of tests to help isolate the error, I was thinking of this earlier comment,

Looking at the script, at the moment the reduce= None part is kinda useless. If you copied that from our current tests, it means they will have to be updated.

I was planning to run the test in TF1 with a different value for value parameter based on the tutorial.  If it fails in TF1, then there may be a latent issue.  If it works, then I know it is my port that probably introduced the error or does not properly account for how tfa functions operate.

That's fair. Also if there is an issue on the tf1 version, I think it would be fair not to solve it as part of the porting process, and then open a PR for that specifically.

I tested with TF1, tried several different values for reduce and could not recreate the problem.  So I'm thinking this is unique to the TF2 port.  Focusing now on how I translated the the tf.contrib api calls to tfa calls.
In terms of the tfa conversion, I'm trying to reconcile the explanation https://github.com/uber/ludwig/pull/639#issuecomment-588501820 with what I see in the stack trace.
For the record, I'm using this model definition in the test.  This comes from this unit test: https://github.com/uber/ludwig/blob/aa29ad332922b10112eb97ae5494bb36284c7d57/tests/integration_tests/test_experiment.py#L187
Here are the input and output features for the model:
https://github.com/uber/ludwig/blob/aa29ad332922b10112eb97ae5494bb36284c7d57/tests/integration_tests/test_experiment.py#L189-L200
The rest of the model definition is
https://github.com/uber/ludwig/blob/aa29ad332922b10112eb97ae5494bb36284c7d57/tests/integration_tests/test_experiment.py#L63-L71
From the stack trace these are the relevant function invocations:
https://github.com/tensorflow/addons/blob/11e50d7da7493196756ea264e6b0b5b179fb92e4/tensorflow_addons/seq2seq/decoder.py#L341-L347
to
https://github.com/tensorflow/tensorflow/blob/e5bf8de410005de06a7ff5393fafdf832ef1d4ad/tensorflow/python/util/nest.py#L567-L569
to
https://github.com/tensorflow/addons/blob/6b202169c95fb2cafa111f6c47995bd0eaad1dd2/tensorflow_addons/seq2seq/decoder.py#L523-L535
This is where the tf.concat is invoked that results in the exception (line 534)
In the prior call, these prepare the data structures to be passed to tf.util.nest.pack_sequence_as() function.
https://github.com/tensorflow/tensorflow/blob/e5bf8de410005de06a7ff5393fafdf832ef1d4ad/tensorflow/python/util/nest.py#L564-L565
In looking at the structure tuple, it appears to be related to the Projection layer.  This based purely on some of the names seen in the structure tuple.

As noted earlier, I believe the tf.concat operation that fails is line 534 in this section of code: https://github.com/tensorflow/addons/blob/6b202169c95fb2cafa111f6c47995bd0eaad1dd2/tensorflow_addons/seq2seq/decoder.py#L523-L535
I still have to dig more into this but at first blush based on just the variable names, the variables may not contain the correct values:

Based just on the names, I would have expected shape to be the a tensor not just an integer.  I can see batch_size being a tensor with the value of the batch size.
With this in mind, I'll revisit the tf.contrib to tfa port to see what effect that may have had.

Let me look into the details, will do this evening or tomorrow morning, but the fact that there are no errors with TF1 is actually good news, it means the current implementation is solid even if we were not testing it 100% correctly. Adding your additional tests to the integration package would be great (although not urgent).

re: the tf.concat() (line 534) call in this section https://github.com/tensorflow/addons/blob/6b202169c95fb2cafa111f6c47995bd0eaad1dd2/tensorflow_addons/seq2seq/decoder.py#L523-L535
I was able to reproduce the ValueError exception that we are seeing with this code fragment:
import tensorflow as tf
import numpy as np

batch_size = tf.constant(5, name="my_constant")
print("batch_size", batch_size)

shape = 8
print(shape)

t3 = tf.concat(([batch_size], shape), axis=0, name='my_concat')
print(t3)

generated output
batch_size Tensor("my_constant:0", shape=(), dtype=int32)
8
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1607, in _create_c_op
    c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape must be rank 1 but is rank 0 for 'my_concat' (op: 'ConcatV2') with input shapes: [1], [], [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/project/sandbox/tensorflow_sandbox.py", line 10, in <module>
    t3 = tf.concat(([batch_size], shape), axis=0, name='my_concat')
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py", line 1420, in concat
    return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_array_ops.py", line 1257, in concat_v2
    "ConcatV2", values=values, axis=axis, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1770, in __init__
    control_input_ops)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1610, in _create_c_op
    raise ValueError(str(e))
ValueError: Shape must be rank 1 but is rank 0 for 'my_concat' (op: 'ConcatV2') with input shapes: [1], [], [].

Process finished with exit code 1

Changing this statement
shape = 8

to
shape = tf.constant([8], dtype=tf.int32)

The code fragment completes w/o error.
import tensorflow as tf
import numpy as np

batch_size = tf.constant(5, name="my_constant")
print("batch_size", batch_size)

shape = tf.constant([8], dtype=tf.int32)
print(shape)

t3 = tf.concat(([batch_size], shape), axis=0, name='my_concat')
print(t3)

generated output
batch_size Tensor("my_constant:0", shape=(), dtype=int32)
Tensor("Const:0", shape=(1,), dtype=int32)
Tensor("my_concat:0", shape=(2,), dtype=int32)

Process finished with exit code 0

So I'm going to focus to see how the shape variable is created by the tfa run-time.

This appears to be the problematic code in tfa.seq2seq.dynamic_decode() https://github.com/tensorflow/addons/blob/a8e04afeab32e6ffda13c3d0a63ce0a05b5d3dfe/tensorflow_addons/seq2seq/decoder.py#L341-L347.  Specifically thedecoder.output_size structure contains the int attribute instead of a tensor.

The attribute projection_input should be a tensor instead of the integer 8.
I'm focusing on the calling sequence to tfa.seq2seq.dynamic_decode() to make sure the arguments I'm passing are what is expected.

I see, it looks like this is either a problem in the tfa package or that we are passing the wrong parameters to it. Could it be the case that keras layers return a tensor where tf1 layers return a  integer for the shape and thatparameter inside the decoder is obtained that way?

I've been reviewing the legacy tf.contrib functions in an attempt  to understand the context for the tfa implementation.  The error we're seeing https://github.com/uber/ludwig/pull/639#issuecomment-589466674  seems involve the Projection object.  I've looked for documentation of Projection in the tf.contrib but no luck.  Is there a document or paper that provide background on this code fragment:
https://github.com/uber/ludwig/blob/a594ab5fd93337cbae3bdc4b8719509c1739e129/ludwig/models/modules/recurrent_modules.py#L60-L132
I'm trying to understand the relationship of the Decoder and Projection.

With hopes of better understanding how tfa works, I've been looking at their tutorial examples.
To do this, I've stripped down our failing test case to just this set of input and output features:
    input_features = [
        sequence_feature(vocab_size=10, max_len=10, encoder='embed')
    ]
    output_features = [
        sequence_feature(vocab_size=10, max_len=5),
    ]

This tfa example, which demonstrates language translation,  seems close to the stripped down test case: https://github.com/tensorflow/addons/blob/master/docs/tutorials/networks_seq2seq_nmt.ipynb
Although the tfa example involves the use of Attention and ours, for the moment, does not, I'm thinking it will be worthwhile to see if I can get the tfa example to work with our generated test data set.  If I can, then we'll have a code template that I'm hoping to help guide the conversion from tf.conrib to tfa.  And to answer questions such as

I see, it looks like this is either a problem in the tfa package or that we are passing the wrong parameters to it. Could it be the case that keras layers return a tensor where tf1 layers return a integer for the shape and thatparameter inside the decoder is obtained that way?

Make sense?

I was able to adapt the tfa NMT example to work with the Ludwig generate test data set.  This means I have a working example of setting up the encoder/decoder constructs with tfa.   Here is the relevant code fragment:
#ENCODER
class EncoderNetwork(tf.keras.Model):
    def __init__(self,input_vocab_size,embedding_dims, rnn_units ):
        super().__init__()
        self.encoder_embedding = tf.keras.layers.Embedding(input_dim=input_vocab_size,
                                                           output_dim=embedding_dims)
        self.encoder_rnnlayer = tf.keras.layers.LSTM(rnn_units,return_sequences=True,
                                                     return_state=True )

#DECODER
class DecoderNetwork(tf.keras.Model):
    def __init__(self,output_vocab_size, embedding_dims, rnn_units):
        super().__init__()
        self.decoder_embedding = tf.keras.layers.Embedding(input_dim=output_vocab_size,
                                                           output_dim=embedding_dims)
        self.dense_layer = tf.keras.layers.Dense(output_vocab_size)
        self.decoder_rnncell = tf.keras.layers.LSTMCell(rnn_units)
        # Sampler
        self.sampler = tfa.seq2seq.sampler.TrainingSampler()
        # Create attention mechanism with memory = None
        self.attention_mechanism = self.build_attention_mechanism(dense_units,None,BATCH_SIZE*[Tx])
        self.rnn_cell =  self.build_rnn_cell(BATCH_SIZE)
        self.decoder = tfa.seq2seq.BasicDecoder(self.rnn_cell, sampler= self.sampler,
                                                output_layer=self.dense_layer)

    def build_attention_mechanism(self, units,memory, memory_sequence_length):
        return tfa.seq2seq.LuongAttention(units, memory = memory,
                                          memory_sequence_length=memory_sequence_length)
        #return tfa.seq2seq.BahdanauAttention(units, memory = memory, memory_sequence_length=memory_sequence_length)

    # wrap decodernn cell
    def build_rnn_cell(self, batch_size ):
        rnn_cell = tfa.seq2seq.AttentionWrapper(self.decoder_rnncell, self.attention_mechanism,
                                                attention_layer_size=dense_units)
        return rnn_cell

    def build_decoder_initial_state(self, batch_size, encoder_state,Dtype):
        decoder_initial_state = self.rnn_cell.get_initial_state(batch_size = batch_size,
                                                                dtype = Dtype)
        decoder_initial_state = decoder_initial_state.clone(cell_state=encoder_state)
        return decoder_initial_state

encoderNetwork = EncoderNetwork(input_vocab_size,embedding_dims, rnn_units)
decoderNetwork = DecoderNetwork(output_vocab_size,embedding_dims, rnn_units)
optimizer = tf.keras.optimizers.Adam()

# ## **Initializing Training functions**

def loss_function(y_pred, y):

    #shape of y [batch_size, ty]
    #shape of y_pred [batch_size, Ty, output_vocab_size]
    sparsecategoricalcrossentropy = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True,
                                                                                reduction='none')
    loss = sparsecategoricalcrossentropy(y_true=y, y_pred=y_pred)
    mask = tf.logical_not(tf.math.equal(y,0))   #output 0 for y=0 else output 1
    mask = tf.cast(mask, dtype=loss.dtype)
    loss = mask* loss
    loss = tf.reduce_mean(loss)
    return loss

def train_step(input_batch, output_batch,encoder_initial_cell_state):
    #initialize loss = 0
    loss = 0
    with tf.GradientTape() as tape:
        encoder_emb_inp = encoderNetwork.encoder_embedding(input_batch)
        a, a_tx, c_tx = encoderNetwork.encoder_rnnlayer(encoder_emb_inp,
                                                        initial_state =encoder_initial_cell_state)

        #[last step activations,last memory_state] of encoder passed as input to decoder Network

        # Prepare correct Decoder input & output sequence data
        decoder_input = output_batch[:,:-1] # ignore <end>
        #compare logits with timestepped +1 version of decoder_input
        decoder_output = output_batch[:,1:] #ignore <start>

        # Decoder Embeddings
        decoder_emb_inp = decoderNetwork.decoder_embedding(decoder_input)

        #Setting up decoder memory from encoder output and Zero State for AttentionWrapperState
        decoderNetwork.attention_mechanism.setup_memory(a)
        decoder_initial_state = decoderNetwork.build_decoder_initial_state(BATCH_SIZE,
                                                                           encoder_state=[a_tx, c_tx],
                                                                           Dtype=tf.float32)

        #BasicDecoderOutput
        outputs, _, _ = decoderNetwork.decoder(decoder_emb_inp,initial_state=decoder_initial_state,
                                               sequence_length=BATCH_SIZE*[Ty-1])

        logits = outputs.rnn_output
        #Calculate loss

        loss = loss_function(logits, decoder_output)

    #Returns the list of all layer variables / weights.
    variables = encoderNetwork.trainable_variables + decoderNetwork.trainable_variables
    # differentiate loss wrt variables
    gradients = tape.gradient(loss, variables)

    #grads_and_vars – List of(gradient, variable) pairs.
    grads_and_vars = zip(gradients,variables)
    optimizer.apply_gradients(grads_and_vars)
    return loss

Using the Ludwig test with the above construct resulted in successfully translating Ludwig "English" to Ludwig "German" :-)
2020-02-23 17:25:10.952179: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-02-23 17:25:10.952436: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-02-23 17:25:10.952557: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Using TensorFlow backend.
2020-02-23 17:25:13.993325: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-02-23 17:25:13.993373: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-02-23 17:25:13.993398: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (fbd7b7ba565d): /proc/driver/nvidia/version does not exist
2020-02-23 17:25:13.993803: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-02-23 17:25:14.001836: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2791150000 Hz
2020-02-23 17:25:14.003845: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5c498a0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-02-23 17:25:14.003915: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
total loss: 2.4664602279663086 epoch 1 batch 5
total loss: 2.385169744491577 epoch 1 batch 10
total loss: 2.1820569038391113 epoch 2 batch 5
total loss: 2.0681848526000977 epoch 2 batch 10
a_tx : (1, 1024)
c_tx : (1, 1024)
decoder_initial_state = [a_tx, c_tx] : (2, 1, 1024)

Compared to simple encoder-decoder without attention, the decoder_initial_state  is an AttentionWrapperState object containing s_prev tensors and context and alignment vector

decoder initial state shape : (6,)
decoder_initial_state tensor
 AttentionWrapperState(cell_state=[<tf.Tensor: shape=(1, 1024), dtype=float32, numpy=
array([[-0.0064916 , -0.01640026, -0.00735155, ...,  0.00877857,
         0.01247721, -0.00202089]], dtype=float32)>, <tf.Tensor: shape=(1, 1024), dtype=float32, numpy=
array([[-0.01298414, -0.03293902, -0.01476123, ...,  0.01750642,
         0.02497621, -0.00402932]], dtype=float32)>], attention=<tf.Tensor: shape=(1, 1024), dtype=float32, numpy=array([[0., 0., 0., ..., 0., 0., 0.]], dtype=float32)>, time=<tf.Tensor: shape=(), dtype=int32, numpy=0>, alignments=<tf.Tensor: shape=(1, 12), dtype=float32, numpy=array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)>, alignment_history=(), attention_state=<tf.Tensor: shape=(1, 12), dtype=float32, numpy=array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)>)

first_inputs returns the same decoder_input i.e. embedding of  <start> : (1, 256)
start_index_emb_avg  tf.Tensor(0.3027622, shape=(), dtype=float32)
English Sentence:
fjjnvzmn qniqd spiik

German Translation:
gdgsyxp xcls xcls mwhhsx

Process finished with exit code 0

From a perspective of next steps, I'm thinking my initial approach of trying to work within the original tf.contrib design and class structure may not be viable.  I'm thinking of gutting and redoing some of the existing classes in ludwig/models/modules/recurrent_modules.py; specifically, I'm looking at these classes:
Projection
BasicDecodeOutput
BasicDecoder
TimesereisTrainingSampler
RecurrentStack
Depending on dependencies, other classes may come in scope.
Any concerns?  What Pandora box am I opening with this approach? :-)

From a perspective of next steps, I'm thinking my initial approach of trying to work within the original tf.contrib design and class structure may not be viable. I'm thinking of gutting and redoing some of the existing classes in ludwig/models/modules/recurrent_modules.py; specifically, I'm looking at these classes:
Projection
BasicDecodeOutput
BasicDecoder
TimesereisTrainingSampler
RecurrentStack
Depending on dependencies, other classes may come in scope.
Any concerns? What Pandora box am I opening with this approach? :-)

This is great progress, thank you!
Let me give you some context for those classes. Most of them are just there for timeseries decoding, which was attempted, made partially working, but never completed. So I would say most of those classes can go and dont need to be ported, the only important one is RecurrentStack as it is used in bot rnn based encoders and decoders.
Regarding your example with keras, there are few considerations: that code runs in eager mode, as you can see the calculations are defined in the train step, there's no session.run, which in general may be ok and a direction we may move towards in the future, but right now all the rest of the Ludwig code first defines the graph and then runs sess.run for training an testing. Because of that, as you can see in the original code, both operations for obtaining predictions (with the greedy sampler that avoids using teacher forcing) and training (which actually uses the teacher forcing, aka decoder_inputs in your code example) are provided so that, depending if we are training or the model or using it for predictions, different nodes are provided to sess.run as readout nodes. This means that that code for our purpose is incomplete, as it shows only how to train, but not how to predict (there's no way to predict without specifying the expected outputs). Another consideration is that this code basically assumes that the encoder provides these badly names outputs a, a_tc and c_tx. This means that the code assumes the encoder to be an RNN, while in Ludwig, for decoupling purposes, we make no assumption on the encoder, we only make assumptions on the shapes of the outputs it provides for better encapsulation.
All these considerations mean that that code example could be a starting point, but would need to be heavily adapted in order to work within Ludwig.
There is another option we could consider. Stop for a second the focus on the recurrent output, rewrite the maining step in eager mode as shown in your example code (with the tape etc.), make it work for all other cases first, and then fcus on the sequence decoder again, so that your example code and the final code that will be there match more closely.
This may require quite some changes, but could be beneficial. What do you think?

Good point about "eager mode" vs "graph mode".  I had been thinking about as well.
RE:

All these considerations mean that that code example could be a starting point, but would need to be heavily adapted in order to work within Ludwig.

My initial objective was to get something working w/in the Ludwig framework as prototype.  This would have been a starting point for adapting the code to comply with Ludwig's overall design objectives.
Re:

There is another option we could consider. Stop for a second the focus on the recurrent output, rewrite the maining step in eager mode as shown in your example code (with the tape etc.), make it work for all other cases first, and then fcus on the sequence decoder again, so that your example code and the final code that will be there match more closely.

I'm open to this direction.  Lacking an understanding of how Ludwig evolved, e.g. status of the various classes, I'm struggling right now to figure out the appropriate way to shim in the changes required for tfa.   So taking a fresh approach might be reasonable.
When you write, "...rewrite the maining step in eager mode as shown in your example code (with the tape etc.), make it work for all other cases first,..." I need more clarification.
Does "rewrite the maining step in eager mode" mean converting the "main training loop, including making predictions" to eager mode?
re: "make it work for all other test cases first" what test cases do these cover?

RE:

rewrite the maining step in eager mode as shown in your example code (with the tape etc.), make it work for all other cases first

In looking at this change, I'm planning to modify class Model in ludwig/models/model.py.  These methods will be updated:
def initialize_session(self, ...)
def train(self, ...)
def train_online(self, ...)
def batch_evaluation(self, ...)
def batch_collect_activations(self, ...)
def collect_weights(self,...)

The major in these methods will be replacing the session.run() construct with the equivalent TF2 eager mode apis.
Anything else I should be aware of?

Yeah sorry, "maining" is a typo, I mean training. Yes the changes you suggest make sense, bu there's a bigger one which is the model building one. Right now Ludwig creates the model by adding operations to a graph, and then that graph is used to start a session that is run in session.run calls. Things would have to change to become functions that compute prediction and probabilities, and those functions are called both when training and when predicting.
I'm actually thinking that the amount of change is too big for someone other than me to do it, probably it's a bit too much information and learning that is required. If that is the case let me know and we can find a different thing to work on entirely and I can can continue the porting on my own.

Your work has been great! I'm suggesting refocus just because it seems unfair to me to ask you to spend your time to get that deeper level of understanding. On the other hand, if you are willing to and you believe it's something beneficial to you, we can definitely keep on working on it. It's your call, just don't want you to feel obliged because I understand how much of a learning curve there is :)

I'm still willing to work on the TF1 to TF2 upgrade.  While it is a lot of effort and slow progress, I'm getting practical experience in making deep learning software work.  So if you're OK with the current pace of work and me bugging you with questions, I'd like to continue.
We can have this understanding.  If, for any reason, you want to take over the task, just let me know that you want to take over.  At the same time, if I feel overwhelmed, I'll reach out to you and pass the work back to you.
These can help me in my work.

Your insight to code segments, such as

Most of them are just there for timeseries decoding, which was attempted, made partially working, but never completed. So I would say most of those classes can go and dont need to be ported,

Or guidance such as this

Right now Ludwig creates the model by adding operations to a graph, and then that graph is used to start a session that is run in session.run calls. Things would have to change to become functions that compute prediction and probabilities, and those functions are called both when training and when predicting.

Basically setting boundaries for the work will help me focus in the right area.

I'm still willing to work on the TF1 to TF2 upgrade. While it is a lot of effort and slow progress, I'm getting practical experience in making deep learning software work. So if you're OK with the current pace of work and me bugging you with questions, I'd like to continue.
We can have this understanding. If, for any reason, you want to take over the task, just let me know that you want to take over. At the same time, if I feel overwhelmed, I'll reach out to you and pass the work back to you.

that's a perfect arrangement for me.

These can help me in my work.

Your insight to code segments, such as

Most of them are just there for timeseries decoding, which was attempted, made partially working, but never completed. So I would say most of those classes can go and dont need to be ported,

Let me do a pass on the current recurrent_modules in the PR to remove everything related to timeseries, that way things would be much cleaner to begin with.

Or guidance such as this

Right now Ludwig creates the model by adding operations to a graph, and then that graph is used to start a session that is run in session.run calls. Things would have to change to become functions that compute prediction and probabilities, and those functions are called both when training and when predicting.

Basically setting boundaries for the work will help me focus in the right area.

makes sense. Part of the reason it is difficult is because you expect some things to work in a certain way in TF2, that you just need to replace calls with corresponsing ones, but than we discover it's not the case, so one needs to adjust. But that's fine, it's an iterative process.
Regarding the graph building part, the function you should look at is model.__build(). That function reads the model definiton and depending on its contents build a computation graph. It does it in modular way, for inputs, outputs and combiners it calls the respective functions, and then at the end it collects the tensors that need to be provided to sess.run for both training and prediction.
Each of these modules gets the model definition part they need and adds the operations to the graph. For isntance the inputs part cicles over the inputs and depending on the type of the input icalls the build_input function of the specific feature. For isntance if you specify an input sequence feature, build_inputs() is called https://github.com/uber/ludwig/blob/master/ludwig/models/model.py#L162 which in turn cycles through the input features and calls build_single_input https://github.com/uber/ludwig/blob/master/ludwig/models/inputs.py#L38 which calls the build_input function of the feature depending on its type https://github.com/uber/ludwig/blob/master/ludwig/models/inputs.py#L61-L69 . In this case, the build_input function of a sequence encoder would be used https://github.com/uber/ludwig/blob/master/ludwig/features/sequence_feature.py#L152 . Then within that function in msot cases an encoder would be selected, like a parallen_cnn or an rnn etc. depending on what the user specifies. You can follow the implementation of build_input to get an idea. Then the same concept is applied to combiners and outputs. Outputs are a bit more complicated, because they have nodes both for prediction and for training (like a placeholder for the ground truth and nodes for losses and measures that are important at training time, at test time but not at training time). check out the categorical output feature class as a comprehensive example and the binary one as a simple example. What i suggest you to do is to put a breakpoint at the beginning of model.__build and follow all the nested steps up until the end of that function, that will give you a great example of how things work. Once you have that understanding, you'll be much better equipped to then figure out how to make the whole thing eager.

Thank you for the guidance.  As a proof-of-concept for the above guidance, I'm planning to demonstrate eager execution on this model:
    input_features = [
        {'name': 'x1', 'type': 'numerical'},
        {'name': 'x2', 'type': 'numerical'},
        {'name': 'x3', 'type': 'numerical'}
    ]
    output_features = [
        {'name': 'y', 'type': 'numerical'}
    ]

    model_definition = {
        'input_features': input_features,
        'output_features': output_features,
        'combiner': {
            'type': 'concat',  # 'concat',
            'num_fc_layers': 2,
            'fc_size': 64
        },
        'training': {'epochs': 20}
    }

This way I only have to focus on one type of feature and a simple fully connected layer.  If I can get this to work for train, validation and test functions, this would form the foundation for the rest of the work.
I'll provide updates as the work progresses

Kind of "Daily stand-up" :-)
What was accomplished
This code and resulting output is a proof-of-concept showing creation and use
the relevant TF2 functions and confirmation on the network structure.
from functools import reduce

import numpy as np
import pandas as pd

from sklearn.model_selection import train_test_split

import tensorflow as tf
from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.models import Model

raw_df = pd.read_csv('./data/train.csv')
raw_X = raw_df.loc[:,'x1':'x3']
raw_y = raw_df['y']
train_X, test_X, train_y, test_y = train_test_split(raw_X, raw_y, test_size=0.2)
print(train_X.shape, test_X.shape, train_y.shape, test_y.shape)

#
# after the input features are combined
#
inputs = Input(shape=(train_X.shape[1],))

###
# For fully connected layer add code here:
# two alternative methods
###
# Alternative 1: standard out-of-the-book method.  One drawback is keeping track of
# argument, e.g., "x" as the functions are created
# x = Dense(64, activation='relu')(inputs)
# x = Dense(64, activation='relu')(x)
#predictions = Dense(1, activation='linear')(x)

# Alternative 2:  just append functions to a list and use reduce() to combine them
# this has the advantage of not having to explicitly track the argument "x"
# 'function_layers' will be added as an instance variable to 'Model' class
function_layers = []
function_layers.append(inputs)
function_layers.append(Dense(64, activation='relu'))
function_layers.append(Dense(64, activation='relu'))
function_layers.append(Dense(1, activation='linear'))
predictions = reduce(lambda x, y: y(x), function_layers)

# create model structure
model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer='adam',
              loss='MSE',
              metrics=['mse'])

# train model
print("\ntraining")
model.fit(train_X, train_y, batch_size=5, epochs=10)  # starts training

# evalutate model on test set
print("\nrunning evlaution")
eval = model.evaluate(test_X, test_y)
print(eval)

print(model.summary())

<<<<<<<< Output >>>>>>>>>>
e012dd85d84b:python -u /opt/project/sandbox/tf2_port/sandbox_model_keras.py
2020-02-27 11:57:44.979921: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-02-27 11:57:44.980095: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-02-27 11:57:44.980117: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
(799, 3) (200, 3) (799,) (200,)
2020-02-27 11:57:45.714441: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-02-27 11:57:45.714487: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-02-27 11:57:45.714511: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (e012dd85d84b): /proc/driver/nvidia/version does not exist
2020-02-27 11:57:45.714726: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-02-27 11:57:45.721055: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2791110000 Hz
2020-02-27 11:57:45.722479: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5fd0b10 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-02-27 11:57:45.722523: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version

training
Train on 799 samples
Epoch 1/10
799/799 [==============================] - 1s 650us/sample - loss: 0.1142 - mse: 0.1142
Epoch 2/10
799/799 [==============================] - 0s 332us/sample - loss: 0.0877 - mse: 0.0877
Epoch 3/10
799/799 [==============================] - 0s 316us/sample - loss: 0.0869 - mse: 0.0869
Epoch 4/10
799/799 [==============================] - 0s 582us/sample - loss: 0.0850 - mse: 0.0850
Epoch 5/10
799/799 [==============================] - 0s 336us/sample - loss: 0.0867 - mse: 0.0867
Epoch 6/10
799/799 [==============================] - 0s 345us/sample - loss: 0.0849 - mse: 0.0849
Epoch 7/10
799/799 [==============================] - 0s 333us/sample - loss: 0.0847 - mse: 0.0847
Epoch 8/10
799/799 [==============================] - 0s 331us/sample - loss: 0.0852 - mse: 0.0852
Epoch 9/10
799/799 [==============================] - 1s 845us/sample - loss: 0.0845 - mse: 0.0845
Epoch 10/10
799/799 [==============================] - 0s 321us/sample - loss: 0.0826 - mse: 0.0826

running evlaution
200/200 [==============================] - 0s 299us/sample - loss: 0.1009 - mse: 0.1009
[0.1008630508184433, 0.100863054]

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         [(None, 3)]               0
_________________________________________________________________
dense (Dense)                (None, 64)                256
_________________________________________________________________
dense_1 (Dense)              (None, 64)                4160
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 65
=================================================================
Total params: 4,481
Trainable params: 4,481
Non-trainable params: 0
_________________________________________________________________
None

Process finished with exit code 0

My current thinking where the above code fragments are inserted in the Ludwig code structure
To fit in the above code fragments, I'm expecting to remove/replace surrounding code in the cited locations.
This fragment
inputs = Input(shape=(train_X.shape[1],))

is added in this region
https://github.com/uber/ludwig/blob/1fda07c555f3f3d3a9f989ab63441c8021a50391/ludwig/models/combiners.py#L83-L104
I'm leaning toward Alternative 2.  This fragment
# Alternative 2:  just append functions to a list and use reduce() to combine them
# this has the advantage of not having to explicitly track the argument "x"
# 'function_layers' will be added as an instance variable to 'Model' class
function_layers = []
function_layers.append(inputs)
function_layers.append(Dense(64, activation='relu'))
function_layers.append(Dense(64, activation='relu'))
function_layers.append(Dense(1, activation='linear'))
predictions = reduce(lambda x, y: y(x), function_layers)

is added in this region https://github.com/uber/ludwig/blob/1fda07c555f3f3d3a9f989ab63441c8021a50391/ludwig/models/modules/fully_connected_modules.py#L122-L137
This fragment
model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer='adam',
              loss='MSE',
              metrics=['mse'])

is added in this region https://github.com/uber/ludwig/blob/1fda07c555f3f3d3a9f989ab63441c8021a50391/ludwig/models/model.py#L203-L222
This fragment
print("\ntraining")
model.fit(train_X, train_y, batch_size=5, epochs=10)  # starts training

is added in this region https://github.com/uber/ludwig/blob/1fda07c555f3f3d3a9f989ab63441c8021a50391/ludwig/models/model.py#L531-L566
This fragment
print("\nrunning evlaution")
eval = model.evaluate(test_X, test_y)

is added in this region https://github.com/uber/ludwig/blob/1fda07c555f3f3d3a9f989ab63441c8021a50391/ludwig/models/model.py#L857-L893
Next steps

Make necessary modification to the Ludwig code base to incorporate the above.
Demonstrate successfully training and evaluation of this simple Ludwig model

input_features = [
    {'name': 'x1', 'type': 'numerical'},
    {'name': 'x2', 'type': 'numerical'},
    {'name': 'x3', 'type': 'numerical'}
]
output_features = [
    {'name': 'y', 'type': 'numerical'}
]

model_definition = {
    'input_features': input_features,
    'output_features': output_features,
    'combiner': {
        'type': 'concat',  # 'concat',
        'num_fc_layers': 2,
        'fc_size': 64
    },
    'training': {'epochs': 20}
}

Blockers
None

A few comments on this.

The starting point should be this isntead: https://www.tensorflow.org/tutorials/quickstart/advanced The reason is that this matches much better with Ludwig's training look, it uses functions for obtaining predictions and other functions to compute losses and objects to define the optimization steps. Kera compile function hides all of this, and those things are needed in Ludwig
The first option, the functional/eager approach is what we want to lean towards, and the example that i sent you only coversa that one.
Looking at your case, you have 3 numerical features. In Ludwig they are treated separately not as a single vector, then the combinar concatenates them. The input part in your code maps to numerical_inputFeatures._get_placeholder(). It will be called 3 times, and the 3 outputs would be given as inputs to the combiner, which will contain the dense layers, so it will be inside combiners.ConcatCombiner.
model.fit would not be there, if you look at the example I psoted there's a more direct mapping with Ludwig's training loop
also the evaluation part will not be a single call, but a bunch of them, but still within the model.evaluate function.
Let me know if this is not clear.

Oh, weird, I added a commit to remove the timeseries stuff from recurrent modules, and i pushed the suaul way but it merge it in the tf2_porting branch. I guess we can continue from there

I understand this branch was accidentally merged.  When you write, "I guess we can continue from there," are you referring to me to continue pushing changes to this branch jimthompson5802:tf2_porting?

I think so. Anyway, it was merged in the tf2_porting branch not in master, which is ok.

I believe I have an understanding of how the custom eager mode custom training loop works.  Using the advanced tutorial you pointed out, I worked out a native TF2 custom training example.  This code will serve as guide for future work.  My next step is begin the work to modify Ludwig for eager mode.
Note:  While working on the sample program, it appeared that training was not converging.  While researching this, I found this issue.  Not clear if this is relevant or not.  Any thoughts about this?
This is the native TF2 sample program

import pandas as pd

from sklearn.model_selection import train_test_split

import tensorflow as tf
from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.models import Model

raw_df = pd.read_csv('./data/train.csv')

raw_X = raw_df.loc[:,'x1':'x3']
raw_y = raw_df['y']

train_X, test_X, train_y, test_y = train_test_split(raw_X, raw_y, test_size=0.2)
print(train_X.shape, test_X.shape, train_y.shape, test_y.shape)

dataset_train = tf.data.Dataset.from_tensor_slices(
    (
        tf.cast(train_X.values, tf.float32),
        tf.cast(train_y.values, tf.float32)
    )
)

dataset_test = tf.data.Dataset.from_tensor_slices(
    (
        tf.cast(test_X.values, tf.float32),
        tf.cast(test_y.values, tf.float32)
    )
)

print("<<<<<< CUSTOM TRAINING LOOP>>>>>>>>>>")
class MyModel(Model):
    def __init__(self):
        super().__init__()

        self.d1 = Dense(64, activation='relu')
        self.d2 = Dense(64, activation='relu')
        self.prediction = Dense(1, activation='linear')

    def call(self, inputs):
        x = self.d1(inputs)
        x = self.d2(x)
        return self.prediction(x)

    def add_layer(self, name, layer):
        self.model_layers[name] = layer

model = MyModel()

loss_object = tf.keras.losses.MeanSquaredError()
optimizer = tf.keras.optimizers.Adam()

train_loss = tf.keras.metrics.Mean(name='train_loss')
train_metric = tf.keras.metrics.MeanSquaredError(name='train_metric')

test_loss = tf.keras.metrics.Mean(name='test_loss')
test_metric = tf.keras.metrics.MeanSquaredError(name='test_metric')

tf.config.experimental_run_functions_eagerly(True)

@tf.function
def train_step(model, optimizer, loss_object, x, y):
    #  issue?: https://github.com/tensorflow/tensorflow/issues/28901
    #y = y[:, tf.newaxis]
    with tf.GradientTape() as tape:
        y_hat = model(x, training=True)
        # print("in training", y.shape, y_hat.shape)
        loss = loss_object(y, y_hat)
        # print("training lost:", loss.numpy())
    gradients = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))

    train_loss(loss)
    train_metric(y, y_hat)

@tf.function
def test_step(model, loss_object, x, y):
    #y = y[:, tf.newaxis]
    y_hat = model(x, training=False)
    # print("in testing", y.shape, y_hat.shape)
    t_loss = loss_object(y, y_hat)

    test_loss(t_loss)
    test_metric(y, y_hat)

EPOCHS = 20

for epoch in range(EPOCHS):
    # Reset the metrics at the start of the next epoch
    train_loss.reset_states()
    train_metric.reset_states()
    test_loss.reset_states()
    test_metric.reset_states()

    for X, y in dataset_train.batch(16):
        #print(X.shape)
        train_step(model, optimizer, loss_object, X, y)

    for X, y in dataset_test.batch(16):
        test_step(model, loss_object, X, y)

    template = 'Epoch {}, train Loss: {}, : train metric {}, Test Loss: {}, Test Metric: {}'
    print(template.format(epoch+1,
                        train_loss.result(),
                        train_metric.result(),
                        test_loss.result(),
                        test_metric.result()))

model.summary()

print('<<<<<<<<<<<<<<< SIMPLE MODEL >>>>>>>>>>>>>>>')
# Compare results with simple model
inputs = Input(shape=(3,))
x = Dense(64, activation='relu')(inputs)
x = Dense(64, activation='relu')(x)
prediction = Dense(1)(x)

model = Model(inputs, prediction)

model.compile(optimizer='adam',
              loss='MSE',
              metrics=['mse'])
print("\ntraining")
model.fit(dataset_train.batch(16), epochs=20)  # starts training
print("\nrunning evlaution")
eval = model.evaluate(test_X, test_y)
print(eval)

print(model.summary())

--------------------------------------------------------
2e19d158f757:python -u /opt/project/sandbox/tf2_port/sandbox_model_keras.py
2020-02-29 01:58:05.937796: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-02-29 01:58:05.937935: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-02-29 01:58:05.937956: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
(799, 3) (200, 3) (799,) (200,)
<<<<<< CUSTOM TRAINING LOOP>>>>>>>>>>
2020-02-29 01:58:06.569722: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-02-29 01:58:06.569745: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-02-29 01:58:06.569773: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (2e19d158f757): /proc/driver/nvidia/version does not exist
2020-02-29 01:58:06.570113: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-02-29 01:58:06.575861: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2791535000 Hz
2020-02-29 01:58:06.576537: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4599350 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-02-29 01:58:06.576557: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
Epoch 1, train Loss: 0.12295467406511307, : train metric 0.12295467406511307, Test Loss: 0.09914495050907135, Test Metric: 0.09914495050907135
Epoch 2, train Loss: 0.09224193543195724, : train metric 0.09224193543195724, Test Loss: 0.0926184132695198, Test Metric: 0.0926184132695198
Epoch 3, train Loss: 0.08829674869775772, : train metric 0.08829674869775772, Test Loss: 0.09075924009084702, Test Metric: 0.09075924009084702
Epoch 4, train Loss: 0.08678507059812546, : train metric 0.08678507059812546, Test Loss: 0.09065888822078705, Test Metric: 0.09065888822078705
Epoch 5, train Loss: 0.0861264318227768, : train metric 0.0861264318227768, Test Loss: 0.0906728133559227, Test Metric: 0.0906728133559227
Epoch 6, train Loss: 0.08584615588188171, : train metric 0.08584615588188171, Test Loss: 0.09036322683095932, Test Metric: 0.09036322683095932
Epoch 7, train Loss: 0.085653156042099, : train metric 0.085653156042099, Test Loss: 0.09012754261493683, Test Metric: 0.09012754261493683
Epoch 8, train Loss: 0.08558208495378494, : train metric 0.08558208495378494, Test Loss: 0.09015573561191559, Test Metric: 0.09015573561191559
Epoch 9, train Loss: 0.0854942575097084, : train metric 0.0854942575097084, Test Loss: 0.09005601704120636, Test Metric: 0.09005601704120636
Epoch 10, train Loss: 0.08544553816318512, : train metric 0.08544553816318512, Test Loss: 0.09012728184461594, Test Metric: 0.09012728184461594
Epoch 11, train Loss: 0.08539620041847229, : train metric 0.08539620041847229, Test Loss: 0.09005728363990784, Test Metric: 0.09005728363990784
Epoch 12, train Loss: 0.08536931127309799, : train metric 0.08536931127309799, Test Loss: 0.0901104137301445, Test Metric: 0.0901104137301445
Epoch 13, train Loss: 0.08532935380935669, : train metric 0.08532935380935669, Test Loss: 0.09014303237199783, Test Metric: 0.09014303237199783
Epoch 14, train Loss: 0.0852506086230278, : train metric 0.0852506086230278, Test Loss: 0.09023386985063553, Test Metric: 0.09023386985063553
Epoch 15, train Loss: 0.08521439880132675, : train metric 0.08521439880132675, Test Loss: 0.09024890512228012, Test Metric: 0.09024890512228012
Epoch 16, train Loss: 0.08518398553133011, : train metric 0.08518398553133011, Test Loss: 0.09023744612932205, Test Metric: 0.09023744612932205
Epoch 17, train Loss: 0.08514454215765, : train metric 0.08514454215765, Test Loss: 0.09019779413938522, Test Metric: 0.09019779413938522
Epoch 18, train Loss: 0.0850943773984909, : train metric 0.0850943773984909, Test Loss: 0.09019617736339569, Test Metric: 0.09019617736339569
Epoch 19, train Loss: 0.08504881709814072, : train metric 0.08504881709814072, Test Loss: 0.09022630006074905, Test Metric: 0.09022630006074905
Epoch 20, train Loss: 0.08499737083911896, : train metric 0.08499737083911896, Test Loss: 0.0901600793004036, Test Metric: 0.0901600793004036
Model: "my_model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense (Dense)                multiple                  256
_________________________________________________________________
dense_1 (Dense)              multiple                  4160
_________________________________________________________________
dense_2 (Dense)              multiple                  65
=================================================================
Total params: 4,481
Trainable params: 4,481
Non-trainable params: 0
_________________________________________________________________
<<<<<<<<<<<<<<< SIMPLE MODEL >>>>>>>>>>>>>>>

training
Train for 50 steps
Epoch 1/20
50/50 [==============================] - 0s 8ms/step - loss: 0.1314 - mse: 0.1315
Epoch 2/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0936 - mse: 0.0936
Epoch 3/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0892 - mse: 0.0892
Epoch 4/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0875 - mse: 0.0875
Epoch 5/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0866 - mse: 0.0866
Epoch 6/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0862 - mse: 0.0862
Epoch 7/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0859 - mse: 0.0859
Epoch 8/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0857 - mse: 0.0857
Epoch 9/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0856 - mse: 0.0856
Epoch 10/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0855 - mse: 0.0855
Epoch 11/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0854 - mse: 0.0854
Epoch 12/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0853 - mse: 0.0853
Epoch 13/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0853 - mse: 0.0853
Epoch 14/20
50/50 [==============================] - 0s 10ms/step - loss: 0.0852 - mse: 0.0852
Epoch 15/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0852 - mse: 0.0852
Epoch 16/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0851 - mse: 0.0852
Epoch 17/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0851 - mse: 0.0851
Epoch 18/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0851 - mse: 0.0851
Epoch 19/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0850 - mse: 0.0850
Epoch 20/20
50/50 [==============================] - 0s 8ms/step - loss: 0.0850 - mse: 0.0850

running evlaution
200/200 [==============================] - 0s 143us/sample - loss: 0.0902 - mse: 0.0902
[0.09019238114356995, 0.09019238]
Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         [(None, 3)]               0
_________________________________________________________________
dense_3 (Dense)              (None, 64)                256
_________________________________________________________________
dense_4 (Dense)              (None, 64)                4160
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 65
=================================================================
Total params: 4,481
Trainable params: 4,481
Non-trainable params: 0
_________________________________________________________________
None

Process finished with exit code 0

Following this comment you may want to check the shape of y and y_hat.

I'm now looking at how to adapt Ludwig data pipeline to support the design described in the previous post.  From what I can tell, the current design has training data converted primarily into a dictionary structure, which is fed into the current tf1 training loop.
For the TF2 eager execution, I have to convert the pre-processed training data to a tf.data.Dataset() structure. I'm looking at doing the conversation in this section of code.  By doing the conversion here,  I leave the current ludwig data preprocessing intact.  This seems the place that allows for minimal changes. https://github.com/uber/ludwig/blob/4039f93e605ed929e4f4a75c725bb70e148df92b/ludwig/data/dataset.py#L21-L37
Does this sound like a reasonable approach?
Are there other factors I need to be aware of?  For example is the ludwig.data.dataset.Dataset class used for purposes beyond training, evaluation and prediction?

I'm not sure you actually need to do the conversion to a tf dataset. I believe the inputs to MyModel.call() in your example will gradly accept numpy ndarrays, which is what Ludwig batcher provide. Actually Ludwig batcer provides a disctionary with the column names and ndarrays as values, so internally in the overall model call the different columns ndarrays need to be dispatched to the right subparts of the model (the different input encoders). The way to do it is by using the name of the column. If my model contains a dictionary of input_feature_name -> encoder function (let's call it self.input_feature_ecoders), when a new input batch arrives you just use the name of the column and obtain the output of the encode, and you do the same for the decoders, something like:
def __call__(self, inputs):
  encoder_outputs = []
  for input_feature_name, input_values in inputs:
    encoder_output = self.input_feature_ecoders[input_feature_name](input_values)
    encoder_outputs.append(encoder_output)
  combiner_outputs = self.combiner(ncoder_outputs)
  output_tensors = {}
  for output_feature_name, decoder in self.output_feature_decoders:
    decoder_output = decoder(combiner_outputs)
    output_tensors[output_feature_name] = decoder_output
  return output_tensors

does this make sense?

OK..as I understand the guidance, I should add these two dictionary attributes to MyModel class:

input_feature_encoders
output_feature_encoders

The dictionary key values will be the input/output column names, i.e., 'x1', 'x2', 'y'.    The dictionary values will be the functions that encode the data type of the column.  Depending on the answer to the next question, I may have follow-up question on the encoding functions.
re:

numpy ndarrays, which is what Ludwig batcher provide

Please point me to the relevant code section for the batcher.
The values returned by the "Ludwig batcher" have these already been pre-processed?   For example, if the column is a numerical data type, has it been "z-scored" if that was requested and missing values filled in.  Or does this function encoder_output = self.input_feature_ecoders[input_feature_name](input_values) have to take care of these operations?  I'm trying to understand how functionality is partitioned in Ludwig.

I see now how Batcher function works.  So no need to point out code section.
I'm still curious to know if the values returned by
batch = batcher.next_batch()

have already been pre-processed?

In thinking more about the recent guidance, I realize I may have misunderstood what you meant.
Initially, when I read the term "encoder" I thought about preparing the data for training.  That is why I had those questions about pre-processing.
However, now I think when you used the term "encoder" and "decoder", this is what you had in mind.  Is this correct?

With some experimentation and use of the debugger, I believe I have the answer to my question,   "Has the data been pre-processed?" at this point in the code
batch = batcher.next_batch()

From what I can tell, the answer is "Yes."  If this is true, then some of the changes I thought needed to happen is no longer relevant.
Again, thank you for the guidance.

I'm still curious to know if the values returned by the batcher have already been pre-processed?

Yes data is already preprocessed when it is provided as a batch. There are other PRs for refactoring that preprocessing code to make it mode generic, byt they could be completely independent form this I believe.
Ans yes the image is an example of an autoencoder with an encoder and a decoder, In Ludwig you have one different encoder for each input feature, a different decoder for each output feature and a combiner. Maybe the presentations here https://w4nderlu.st/projects/ludwig ot Ludwig's whitepaper can help you getting a better understanding about this notion.
Let me know if you have other questions.

Commit 1dbfdfb disables running tests/integration_tests/test_contrib_wandb.py temporarily. Once the issue with this test and TF2 is resolved, the test should be re-enabled.

I had a similar issue due to the fact that wandb uses the context manager which conflicts with pytest. It was solved by using a fixture such as with wandb.init(…) as run: return run.
In my case wandb.init() was called outside of the callback which has some pros (more flexibility to the user) but would be a breaking change.
This may not be the problem here since it is working in master branch.
In any case let me know if you need any help with this test.

@borisdayma thank you for offering to help! I this branch has many thing that are not working yet, so this is just one of the many, but when we get to the point where most things work, then we'll start work on the contribs, at that point your experience will likely be valuable. Please monitor the open PR (this one was merged by error) https://github.com/uber/ludwig/pull/646

I'm stuck with converting tf.contrib.seq2seq.TrainingHelper to tf2.2, please help me.

I'm stuck with converting tf.contrib.seq2seq.TrainingHelper to tf2.2, please help me.

use tensorflow_addons instead

thanks @w4nderlust , i fixed it, but know I have a problem with
training_logits, _, _ = tfa.seq2seq.decoder.dynamic_decode(decoder=training_decoder,
                                                                   maximum_iterations=max_target_length,
                                                                   swap_memory=True,
                                                                   scope="Training_Decoder",
                                                                   decoder_init_input=dec_embed_input,
                                                                   decoder_init_kwargs={
                                                                       'initial_state': initial_state
                                                                   })

It always return TypeError: call() got an unexpected keyword argument 'training' although I followed the tutorial on stackoverflow. Can you explain for me why I got this issue?

It's very difficult for me to help out without further context unfortuantely. Plus, this is not really Ludwig related :) What I suggest you to do is to look at Ludwig's Sequence decoders module and comapre with your implementation. Hopefully it helps.

repo: uber/ludwig


text:
Bag input feature tf2 porting

Code Pull Requests
Please provide the following:

a clear explanation of what your code dose
if applicable, a reference to an issue
a reproducible test for your PR (code, model definition and data sample)

Documentation Pull Requests
Note that the documentation HTML files are in docs/ while the Markdown sources are in mkdocs/docs.
If you are proposing a modification to the documentation you should change only the Markdown files.
api.md is automatically generated from the docstrings in the code, so if you want to change something in that file, first modify ludwig/api.py docstring, then run mkdocs/code_docs_autogen.py, which will create mkdocs/docs/api.md .

Left a couple comments, minor stuff.
Other than that, does this pass the tests? If yes I'll merge once the comments are addressed.

@w4nderlust test_experiment_various_feature_types which uses bag feature as input passes locally.

source: gharchive/pull-request

license_type: permissive

license_source: bigquery


text:
Don't fail on first error

When profiling a long lived multithreaded python process, sometimes the stack can't be read correctly.
This change simply handles such errors by logging the failure to stderr, and continues sampling.
Doesn't solve the underlying issue from #129, but it works around such failures (under the assumption it's ok to lose a handful of samples out of a few thousand)

Apologies for the delay on looking at this. In general I am kind of hesitant about these types of changes because it can lead to a situation where you just fail in a tight loop. Is this really just a rare failure, or does it keep happening?

I am able to reproduce the issue on a regular basis. It typically happens every few minutes when sampling over a long period of time. I agree that if the first cycle fails it probably indicates a real issue, so perhaps some logic to fail loudly for that case?
A better approach would be to include these as "failed" samples in the profiling data.

In that case your suggestion of reporting them as failed makes sense, I would accept that. Should be pretty easy to do, see how idle is reported. I also updated Travis last night so the tests should pass if you get the latest changes in master.

I was thinking about doing a small release (v1.6.4) to include #144, as I've gotten a number of complaints about the default sample rate setting. I will hold off on that for a bit so we can get this change in as well.

I set it up to log both to the summary as (failed), and when dumping with timestamps as (failed). I don't use chrome's cpu profiler, so I'd prefer to leave it to someone else to sort out that integration.

authors: eklitzke stevenkaras

repo: uber/pyflame

license_source: github-api


text:
When passing invalid thrift to tcurl it does not print line number

I get output like:
$ tcurl -t ./benchmarks/benchmark.thrift -p 127.0.0.1:21300 a b

undefined:6350
      throw peg$buildException(null, peg$maxFailExpected, peg$maxFailPos);
            ^
SyntaxError: Expected "/*", "0x", "=", "xsd_attributes", "xsd_nillable", "xsd_optional", "}", [+\-], [0-9], comment, end of line, list separator or whitespace but "." found.

it tells me the line number of the pegjs grammar but not the line number in the thrift file.
For this concrete error I left a ... in a struct definition to remind myself to add more things.

PEG SyntaxError instances have that information. We’ll surface it.

source: gharchive/issue

authors: Raynos kriskowal

repo: uber/tcurl

license: mit


text:
Use kwargs

This doesn't follow the suggestion in PEP 0249 (https://www.python.org/dev/peps/pep-0249/#id40) for the connect method to utilize keyword arguments. This may seem pedantic, but in fact this will raise errors when trying to use a generic connect method. For example, you can't use a generic method to connect using MySQLdb or vertica-python.

Agreed good change.

thanks for the diff

authors: Bennyelg alexjikim lemeryfertitta

repo: uber/vertica-python


text:
Support adding class name(s) to the input element

Any interest in adding this in? Would be handy

Could be helpfull, yes. Would be setting an additional class on the container enough? Or do you want to change the classname at all?

Either or... the latter being a little more useful.

Normally it should be enough to add a class to the container as the rest is self-contained in the module. Or is there a case where you’ll need CSS access to the list or suggested items other than the default classes?

Bump ;)

Released in 1.4.0. You can now add an additional className. See the Readme: https://github.com/ubilabs/react-geosuggest#classname

authors: nathanmarks ro-ka

repo: ubilabs/react-geosuggest


text:
Do we need to keep the library file in a folder?

Online sources (such as https://packaging.python.org/en/latest/tutorials/packaging-projects/) tell me that when creating a python library, you need to create a folder with the python files you wish to include and an init.py file. However, for a single file this leads to the slightly redundant behavior I noted in this issue https://github.com/ubsuny/23-Homework6G1/issues/17.
My point is, do we need to do the whole folder setup, or are we able to just use a single file in this case?

I don't think it is the necessity but if you want to you can.

What did you choose to do @WildJimmy

authors: WildJimmy reshnashrestha


text:
failed to create pod Error response from daemon: error creating aufs mount

Events from kubectl describe pods
Warning  FailedCreatePodSandBox  0s (x17 over 3m)  kubelet, ubuntu    (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "database-5946f77db9-gs7dh": Error response from daemon: error creating aufs mount to /var/snap/microk8s/common/var/lib/docker/aufs/mnt/5421fe090830b6e497d4704f991254785692c500470d1c9d8035e6c732528908-init: invalid argument

Thank you for reporting this @chandu0101 ,
I would like to see the output of sudo snap list so as to see the microk8s revision you are on. This kind of errors are usually associated with docker. Can I see the out put of sudo journalctl -u snap.microk8s.daemon-docker and sudo journalctl -u snap.microk8s.daemon-kubelet? Can you also make sure you do not have any other docker installed on your system (eg docker.io, docker-ce, snapped docker)?
Is there a way for me to reproduce the error you are seeing? Which linux distribution are you on? Does this error appear on a specific workload?
Thanks

output of sudo snap list
core                  16-2.33.1  4917  stable    canonical  core
gnome-3-26-1604       3.26.0     70    stable/…  canonical  -
gnome-calculator      3.28.2     180   stable/…  canonical  -
gnome-characters      3.28.2     103   stable/…  canonical  -
gnome-logs            3.28.2     37    stable/…  canonical  -
gnome-system-monitor  3.28.2     51    stable/…  canonical  -
gtk-common-themes     0.1        319   stable/…  canonical  -
microk8s              v1.11.1    149   beta      canonical  classic```

output of `sudo journalctl -u snap.microk8s.daemon-docker`

```-- Logs begin at Mon 2018-09-10 10:35:41 UTC, end at Tue 2018-09-11 06:41:08 UTC
Sep 11 00:13:25 ubuntu systemd[1]: Started Service for snap application microk8s
Sep 11 00:13:25 ubuntu microk8s.daemon-docker[6331]: Using a docker-default temp
Sep 11 00:13:25 ubuntu microk8s.daemon-docker[6331]: Reloading AppArmor profiles
Sep 11 00:13:25 ubuntu microk8s.daemon-docker[6331]: apparmor.service is not act
Sep 11 00:13:25 ubuntu systemd[1]: snap.microk8s.daemon-docker.service: Main pro
Sep 11 00:13:25 ubuntu systemd[1]: snap.microk8s.daemon-docker.service: Failed w
Sep 11 00:13:25 ubuntu systemd[1]: snap.microk8s.daemon-docker.service: Service
Sep 11 00:13:25 ubuntu systemd[1]: snap.microk8s.daemon-docker.service: Schedule
Sep 11 00:13:25 ubuntu systemd[1]: Stopped Service for snap application microk8s
Sep 11 00:13:25 ubuntu systemd[1]: Started Service for snap application microk8s
Sep 11 00:13:25 ubuntu microk8s.daemon-docker[6403]: Docker default profile alre
Sep 11 00:13:31 ubuntu microk8s.daemon-docker[6403]: time="2018-09-11T00:13:31.8
Sep 11 00:13:32 ubuntu microk8s.daemon-docker[6403]: time="2018-09-11T00:13:32.8
Sep 11 00:13:33 ubuntu microk8s.daemon-docker[6403]: time="2018-09-11T00:13:33.2

output of sudo journalctl -u snap.microk8s.daemon-kubelet
Sep 11 00:13:25 ubuntu systemd[1]: Started Service for snap application microk8s
Sep 11 00:13:34 ubuntu microk8s.daemon-kubelet[6339]: Flag --fail-swap-on has be
Sep 11 00:13:34 ubuntu microk8s.daemon-kubelet[6339]: Flag --feature-gates has b
Sep 11 00:13:34 ubuntu microk8s.daemon-kubelet[6339]: Flag --pod-cidr has been d
Sep 11 00:13:34 ubuntu microk8s.daemon-kubelet[6339]: Flag --non-masquerade-cidr
Sep 11 00:13:34 ubuntu microk8s.daemon-kubelet[6339]: Flag --eviction-hard has b
Sep 11 00:13:35 ubuntu microk8s.daemon-kubelet[6339]: I0911 00:13:35.378504    6```

>Can you also make sure you do not have any other docker installed on your system (eg docker.io, docker-ce, snapped docker)?
no docker installed , fresh ubuntu vm

@chandu0101 the formatting of your comment hides most of the info I was trying to gather.
I see above that snap.microk8s.daemon-docker is failing to start but the reason is truncated. The logs read: Sep 11 00:13:25 ubuntu systemd[1]: snap.microk8s.daemon-docker.service: Failed w. The missing part should give us a  hint of what is wrong.

Ok, so here is what you can do. journalctl -u snap.microk8s.daemon-docker > /tmp/docker.logs then open /tmp/docker.logs with your favorite editor that wraps lines and show us the error. I am mostly interested in the last part of the log.
You can always tell journalctl to wrap the text. Whatever is easier for you.

thanks for reply here is the full log
Sep 11 00:13:25 ubuntu systemd[1]: Started Service for snap application microk8s.daemon-docker.
Sep 11 00:13:25 ubuntu microk8s.daemon-docker[6331]: Using a docker-default template
Sep 11 00:13:25 ubuntu microk8s.daemon-docker[6331]: Reloading AppArmor profiles
Sep 11 00:13:25 ubuntu microk8s.daemon-docker[6331]: apparmor.service is not active, cannot reload.
Sep 11 00:13:25 ubuntu systemd[1]: snap.microk8s.daemon-docker.service: Main process exited, code=exited, sta$
Sep 11 00:13:25 ubuntu systemd[1]: snap.microk8s.daemon-docker.service: Failed with result 'exit-code'.
Sep 11 00:13:25 ubuntu systemd[1]: snap.microk8s.daemon-docker.service: Service hold-off time over, schedulin$
Sep 11 00:13:25 ubuntu systemd[1]: snap.microk8s.daemon-docker.service: Scheduled restart job, restart counte$
Sep 11 00:13:25 ubuntu systemd[1]: Stopped Service for snap application microk8s.daemon-docker.
Sep 11 00:13:25 ubuntu systemd[1]: Started Service for snap application microk8s.daemon-docker.
Sep 11 00:13:25 ubuntu microk8s.daemon-docker[6403]: Docker default profile already patched
Sep 11 00:13:31 ubuntu microk8s.daemon-docker[6403]: time="2018-09-11T00:13:31.873792814Z" level=info msg="li$
Sep 11 00:13:32 ubuntu microk8s.daemon-docker[6403]: time="2018-09-11T00:13:32.871584586Z" level=warning msg=$
Sep 11 00:13:33 ubuntu microk8s.daemon-docker[6403]: time="2018-09-11T00:13:33.277941867Z" level=info msg="Gr$
Sep 11 00:13:33 ubuntu microk8s.daemon-docker[6403]: time="2018-09-11T00:13:33.296444479Z" level=warning msg=$
Sep 11 00:13:33 ubuntu microk8s.daemon-docker[6403]: time="2018-09-11T00:13:33.306501052Z" level=warning msg=$
Sep 11 00:13:33 ubuntu microk8s.daemon-docker[6403]: time="2018-09-11T00:13:33.306780452Z" level=warning msg=$
Sep 11 00:13:33 ubuntu microk8s.daemon-docker[6403]: time="2018-09-11T00:13:33.307676514Z" level=info msg="Lo$```

@chandu0101 The end of the log lines are still truncated. Try this instead please:
sudo journalctl -u snap.microk8s.daemon-docker --no-pager

authors: chandu0101 ktsakalozos tvansteenburgh


text:
can i change endpoint name

Can I set another url for api like http://%s/simple-example/somethin instead http://%s/simple-example/predict?

Hi, @Archelunch.
There is no easy way yet. The clipper source must be modified manually.
https://github.com/ucbrise/clipper/blob/develop/src/frontends/src/query_frontend.hpp#L405

Ok, maybe I''try it

authors: Archelunch withsmilo

repo: ucbrise/clipper