text:
Error when an error occurs in some soda check

When all tests pass, no error occurs
Some libs used
prefect==2.4.2
prefect_shell==0.1.3
prefect-soda-core[postgres]
prefect-soda-core[athena]
soda-core
soda-core-postgres
soda-core-athena
MarkupSafe==2.0.1
PyAthena
[17:01:56]       c20 nulos [PASSED]
[17:01:56]       C20 deve ser maior ou igual a 0 [PASSED]
[17:01:56]       C40 nulos [PASSED]
[17:01:56]       C40 deve ser maior ou igual a 0 [PASSED]
[17:01:56]     db_maritimo [Duplicated] in cabotagem
[17:01:56]       CEs duplicados [PASSED]
[17:01:56]     db_maritimo [Today] in cabotagem
[17:01:56]       Sem registros criados na data de hoje [PASSED]
[17:01:56]       Data Operação null [PASSED]
05:01:56 PM
soda_scan_execute-1578bf38-0
[17:01:56] 2/30 checks WARNED:
05:01:56 PM
soda_scan_execute-1578bf38-0
[17:01:56]     db_maritimo [Five] in cabotagem
[17:01:56]       tipo_fcl nulos [WARNED]
[17:01:56]         check_value: 1.03
[17:01:56]         row_count: 7749
[17:01:56]         missing_count: 80
[17:01:56]     db_maritimo [Duplicated] in cabotagem
[17:01:56]       Embarque duplicados (id_porto_carga,id_porto_origem,nrblconhecimento,lista_fcl iguais) [WARNED]
[17:01:56]         check_value: 2368
05:01:56 PM
soda_scan_execute-1578bf38-0
[17:01:56] Only 2 warnings. 0 failures. 0 errors. 28 pass.
[17:01:56] Sending results to Soda Cloud
05:01:56 PM
soda_scan_execute-1578bf38-0
[17:02:03] Soda Cloud Trace: 7899457579506349419
05:02:03 PM
soda_scan_execute-1578bf38-0
Saving scan results to 2023-06-30T20:00:37.181228+00:00--soda_scan_execute-1578bf38-0.json
05:02:03 PM
soda_scan_execute-1578bf38-0
Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 1214, in orchestrate_task_run
    result = await task.fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect_soda_core/tasks.py", line 142, in soda_scan_execute
    raise e
  File "/usr/local/lib/python3.8/site-packages/prefect_soda_core/tasks.py", line 134, in soda_scan_execute
    soda_logs = await shell_run_command.fn(
  File "/usr/local/lib/python3.8/site-packages/prefect_shell/commands.py", line 103, in shell_run_command
    raise RuntimeError(msg)
RuntimeError: Command failed with exit code 1:
Saving scan results to 2023-06-30T20:00:37.181228+00:00--soda_scan_execute-1578bf38-0.json
05:02:08 PM
soda_scan_execute-1578bf38-0
Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 596, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 117, in run_sync_in_interruptible_worker_thread
    tg.start_soon(
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 597, in __aexit__
    raise exceptions[0]
  File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 97, in capture_worker_thread_and_result
    result = __fn(*args, **kwargs)
  File "flows/Governanca/soda/soda_cabotagem_flow.py", line 26, in soda_cabotagem_captura_flow
    run_soda_scan('psql',
  File "/tmp/tmpsv7sxzh_prefect/flows/lib_classes/libsoda.py", line 30, in run_soda_scan
    return soda_scan_execute(
  File "/usr/local/lib/python3.8/site-packages/prefect/tasks.py", line 295, in __call__
    return enter_task_run_engine(
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 736, in enter_task_run_engine
    return run_async_from_worker_thread(begin_run)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 137, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/usr/local/lib/python3.8/site-packages/anyio/from_thread.py", line 47, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 906, in run_async_from_thread
    return f.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 874, in get_task_call_return_value
    return await future._result()
  File "/usr/local/lib/python3.8/site-packages/prefect/futures.py", line 237, in _result
    return final_state.result(raise_on_failure=raise_on_failure)
  File "/usr/local/lib/python3.8/site-packages/prefect/orion/schemas/states.py", line 145, in result
    raise data
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 1214, in orchestrate_task_run
    result = await task.fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect_soda_core/tasks.py", line 142, in soda_scan_execute
    raise e
  File "/usr/local/lib/python3.8/site-packages/prefect_soda_core/tasks.py", line 134, in soda_scan_execute
    soda_logs = await shell_run_command.fn(
  File "/usr/local/lib/python3.8/site-packages/prefect_shell/commands.py", line 103, in shell_run_command
    raise RuntimeError(msg)
RuntimeError: Command failed with exit code 1:

Hey @carlosnizolli 👋
It seems the real cause of the issue is not reported in the log.
Can you try running Soda outside Prefect?
Do you see the same error?
Any additional debugging log would be useful

Hi @AlessandroLollo,
Outside the prefect no error occurs.
I'll try to get some more logs and send them later.

From the tests I've done, only when soda returns a warning does the flow break in this soda_logs variable

Hey @carlosnizolli 👋
Thanks for the update!
Would you mind providing a reproducible example of such behavior?
Otherwise it's a bit hard to investigate.
Thanks!

I'm running on a machine with these libs
scikit-learn
astroid==2.4.2
awswrangler
backcall==0.2.0
beautifulsoup4==4.9.2
boto3
botocore
bs4==0.0.1
cachetools==4.2.0
certifi==2020.12.5
cffi==1.14.3
chardet==3.0.4
click==8.1.2
cloudpickle==2.0
contextvars==2.4
croniter==1.0.12
cryptography==36.0.1
cycler==0.10.0
dask==2.25.0
decorator==4.4.2
distributed==2.25.0
elastic-transport==8.4.0
elasticsearch==7.13.4
future==0.18.2
google-api-core==1.24.1
google-auth==1.24.0
grpcio==1.34.0
HeapDict==1.0.1
idna==2.10
immutables==0.14
ipfn
ipykernel==5.3.4
ipython-genutils==0.2.0
ipython==7.16.1
isodate==0.6.0
isort==5.4.2
jedi==0.17.2
jellyfish
Jinja2==2.11.2
jmespath==0.10.0
joblib==0.16.0
kiwisolver==1.2.0
lazy-object-proxy==1.4.3
lxml==4.5.2
MarkupSafe==2.0.1
marshmallow-oneofschema==2.0.1
marshmallow==3.7.1
matplotlib
mccabe==0.6.1
Metaphone==0.6
msgpack==1.0.0
msrest==0.6.19
mypy-extensions==0.4.3
natsort==7.0.1
nltk
numpy
numpyencoder==0.3.0
oauthlib==3.1.0
oscrypto==1.2.1
pandas
parso==0.7.1
pendulum==2.1.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.2.0
prefect==2.4.2
prefect_aws
prompt-toolkit==3.0.6
proto-plus==1.13.0
protobuf
psutil==5.7.2
psycopg2-binary==2.9.3
ptyprocess==0.6.0
pyarrow
pyasn1-modules==0.2.8
pyasn1==0.4.8
PyAthena==2.2.0
pycparser==2.20
pycrypto==2.6.1
pycryptodomex==3.9.8
pydrive==1.2.1
Pygments==2.6.1
PyJWT==1.7.1
pylint==2.6.0
pyOpenSSL==19.1.0
pyparsing==2.4.7
python-dateutil==2.8.2
python-decouple==3.3
python-slugify==5.0.1
pytz==2021.1
pytzdata==2020.1
PyYAML==5.4.1
pyzmq==19.0.2
rarfile==4.0
regex==2020.7.14
requests-aws4auth==1.2.2
requests-oauthlib==1.3.0
requests
rsa==4.6
s3fs
s3transfer
scipy==1.5.2
seaborn==0.10.1
setuptools==49.6.0
simple-crypt==4.1.7
six==1.15.0
##sklearn
## The 'sklearn' PyPI package is deprecated use scikit-learn
##scikit-learn
#####
sortedcontainers==2.2.2
soupsieve==2.0.1
sparse_dot_topn
tabulate==0.8.7
tblib==1.7.0
tenacity==6.2.0
text-unidecode==1.3
threadpoolctl==2.1.0
toml==0.10.1
toolz==0.10.0
tornado==6.0.4
tqdm==4.48.2
traitlets==4.3.3
transformers
typed-ast==1.4.1
urllib3
wcwidth==0.2.5
websocket-client==0.57.0
wheel==0.35.1
wrapt==1.12.1
zict==2.0.0
prefect_shell==0.1.3
## prefect_shell current versions do not work (0.1.5 and 0.1.4)
soda-core
soda-core-postgres
soda-core-athena
prefect-soda-core[postgres]
prefect-soda-core[athena]

My code is like this
from decouple import config
from prefect_soda_core.soda_configuration import SodaConfiguration
from prefect_soda_core.sodacl_check import SodaCLCheck
from prefect_soda_core.tasks import soda_scan_execute

ENV = config("ENV")
labels = [ENV]

PATH = config('SODA_CORE_DIR')

def run_soda_scan(database, config_file, checks, data_source, variables_dict):
    if database == 'athena':
        variables_dict['ACCESS_KEY_ID'] = config('ATHENA_AWS_ACCESS_KEY_ID')
        variables_dict['SECRET_KEY'] = config('ATHENA_AWS_SECRET_ACCESS_KEY')
    else:
        variables_dict['POSTGRES_USERNAME'] = config('DB_QA_SODA_USER')
        variables_dict['POSTGRES_PASSWORD'] = config('DB_QA_SODA_PASSWORD')

    variables_dict['API_PUBLIC'] = config('SODA_CLOUD_API_KEY_ID')
    variables_dict['API_PRIVATE'] = config('SODA_CLOUD_API_SECRET')

    soda_configuration_block = SodaConfiguration(
        configuration_yaml_path=PATH + config_file
    )
    soda_check_block = SodaCLCheck(
        sodacl_yaml_path=PATH + checks
    )

    return soda_scan_execute(
        data_source_name=data_source,
        configuration=soda_configuration_block,
        checks=soda_check_block,
        variables=variables_dict,
        return_scan_result_file_content=True,
        verbose=False
    )

My flow is like this
from prefect import flow
from decouple import config
from flows.lib_classes.libprefect import BasePrefectService
from flows.lib_classes.libsoda import run_soda_scan

ENV = config("ENV")
labels = [ENV]

@flow(timeout_seconds=600)
def soda_cabotagem_athena_flow():

    run_soda_scan('athena',
                  'cabotagem/config_athena.yml',
                  'cabotagem/cabotagem_athena.yml',
                  'cabotage_apresentation',
                  {})

@flow(timeout_seconds=1800)
def soda_cabotagem_captura_flow():
    soda_variables = {"DATA_SOURCE": "cabotagem",
                      "HOST": config('DB_SISTEMA_REPLICA_HOST')}

    run_soda_scan('psql',
                  'cabotagem/config_psql.yml',
                  'cabotagem/db_maritimo.yml',
                  'cabotagem',
                  soda_variables)

@flow(timeout_seconds=3600)
def soda_cabotagem_intermediaria_flow():
    soda_variables = {"DATA_SOURCE": "cabotagem_intermediaria",
                      "HOST": config('DB_SISTEMA_REPLICA_HOST')}

    run_soda_scan('psql',
                  'cabotagem/configuration.yml',
                  'cabotagem/db_maritimo_intermediaria.yml',
                  'cabotagem_intermediaria',
                  soda_variables)

flowName = "soda_cabotagem_flow"

PrefectFlow = BasePrefectService(flowName, labels)

@flow(name=flowName)
def soda_cabotagem_flow():
    soda_cabotagem_athena_flow()
    soda_cabotagem_captura_flow()
    # soda_cabotagem_intermediaria_flow()

databases = [
    'ATHENA QA', 'DB SISTEMA REPLICA'
]

PrefectFlow.register(soda_cabotagem_flow, "SODA", databases)

Can you try setting verbose=True on soda_scan_execute?
I wonder if we can get some more information to debug the issue.

`
[11:18:34] 1/28 checks WARNED:
11:18:34 AM
soda_scan_execute-1578bf38-0
[11:18:34]     db_maritimo [Five] in cabotagem
[11:18:34]       Terminal de Embarque nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [WARNED]
[11:18:34]         check_value: 4.28
[11:18:34]         row_count: 6933
[11:18:34]         missing_count: 297
[11:18:34] Only 1 warning. 0 failure. 0 errors. 27 pass.
11:18:34 AM
soda_scan_execute-1578bf38-0
[11:18:34] Sending results to Soda Cloud
11:18:34 AM
soda_scan_execute-1578bf38-0
{
"definitionName": "Soda Core CLI",
"defaultDataSource": "cabotagem",
"dataTimestamp": "2023-07-03T14:11:57+00:00",
"scanStartTimestamp": "2023-07-03T14:11:57+00:00",
"scanEndTimestamp": "2023-07-03T14:18:34+00:00",
"hasErrors": false,
"hasWarnings": true,
"hasFailures": false,
"metrics": [
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmempresanavegacao-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-tipoconhecimento-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-tipo_carga-invalid_count-d413226a",
"metricName": "invalid_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-cdempresanavegacao-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-c40-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmconsignatario-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-c20-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-c20-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-tipo_carga-missing_count",
"metricName": "missing_count",
"value": 3
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmembarcacao-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-c40-min",
"metricName": "min",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-id-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nrcemercante-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-tipo_carga-missing_percent",
"metricName": "missing_percent",
"value": 0.04
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nrviagem-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-c40-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-c20-min",
"metricName": "min",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmterminalportuariocarregamento-missing_percent",
"metricName": "missing_percent",
"value": 4.28
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmembarcacao-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-id-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-teus-min",
"metricName": "min",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmterminalportuariocarregamento-missing_count",
"metricName": "missing_count",
"value": 297
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nrviagem-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-teus-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmportoorigem-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-row_count",
"metricName": "row_count",
"value": 6933
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-cdconsignatario-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmportoorigem-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-cdembarcacao-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Today-dtoperacao-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-cdshipper-missing_percent",
"metricName": "missing_percent",
"value": 0.36
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-id-duplicate_count",
"metricName": "duplicate_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Today-dtoperacao-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmportodestino-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-teus-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmempresanavegacao-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nrcemercante-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-cdembarcacao-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmconsignatario-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-cdconsignatario-missing_count",
"metricName": "missing_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nrcemercante-duplicate_count",
"metricName": "duplicate_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-tipoconhecimento-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Duplicated-nrcemercante-duplicate_count",
"metricName": "duplicate_count",
"value": 0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-cdempresanavegacao-missing_percent",
"metricName": "missing_percent",
"value": 0.0
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-cdshipper-missing_count",
"metricName": "missing_count",
"value": 25
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Today-row_count",
"metricName": "row_count",
"value": 1
},
{
"identity": "metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmportodestino-missing_count",
"metricName": "missing_count",
"value": 0
}
],
"checks": [
{
"identity": "01e5a581",
"identities": {
"v1": "0139be45",
"v2": "f701d16d",
"v3": "01e5a581"
},
"name": "Quantidade de embarques ultimos 5 dias",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - row_count > 0:\n      name: Quantidade de embarques ultimos 5 dias\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 33,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": null,
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-row_count"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 6933,
"fail": {
"lessThanOrEqual": 0.0
}
},
"source": "soda-core"
},
{
"identity": "4b94c36e",
"identities": {
"v1": "82e25c7a",
"v2": "12d58d07",
"v3": "4b94c36e"
},
"name": "IDs duplicados ultimos 5 dias",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - duplicate_count(id) = 0:\n      name: IDs duplicados ultimos 5 dias\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 36,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "id",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-id-duplicate_count"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0,
"fail": {
"greaterThan": 0.0,
"lessThan": 0.0
}
},
"source": "soda-core"
},
{
"identity": "e0510e56",
"identities": {
"v1": "c5ba812e",
"v2": "fc7ca72b",
"v3": "e0510e56"
},
"name": "ID nulo",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(id) = 0:\n      name: ID nulo\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 39,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "id",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-id-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 0.0,
"lessThan": 0.0
}
},
"source": "soda-core"
},
{
"identity": "19cd9673",
"identities": {
"v1": "f8df3636",
"v2": "7ef96a48",
"v3": "19cd9673"
},
"name": "CEs duplicados ultimos 5 dias",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - duplicate_count(nrcemercante) = 0:\n      name: CEs duplicados ultimos 5 dias\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 42,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "nrcemercante",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-nrcemercante-duplicate_count"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0,
"fail": {
"greaterThan": 0.0,
"lessThan": 0.0
}
},
"source": "soda-core"
},
{
"identity": "286e93dc",
"identities": {
"v1": "0cb36631",
"v2": "74b217da",
"v3": "286e93dc"
},
"name": "N\u00famero CE nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(nrcemercante) = 0:\n      name: N\u00famero CE nulos\n\n# Armador\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 45,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "nrcemercante",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-nrcemercante-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 0.0,
"lessThan": 0.0
}
},
"source": "soda-core"
},
{
"identity": "5ae431d2",
"identities": {
"v1": "dc355533",
"v2": "3a2dc2a5",
"v3": "5ae431d2"
},
"name": "Armador nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(nmempresanavegacao):\n      warn: when between 1 and 2\n      fail: when > 2\n      name: Armador nulos\n\n# Cod Armador\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 49,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "nmempresanavegacao",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmempresanavegacao-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 2.0
},
"warn": {
"greaterThanOrEqual": 1.0,
"lessThanOrEqual": 2.0
}
},
"source": "soda-core"
},
{
"identity": "c21b1657",
"identities": {
"v1": "6110eaf4",
"v2": "d8086196",
"v3": "c21b1657"
},
"name": "Codigo Armador nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(cdempresanavegacao):\n      warn: when between 1 and 5\n      fail: when > 5\n      name: Codigo Armador nulos\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 57,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "cdempresanavegacao",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-cdempresanavegacao-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 5.0
},
"warn": {
"greaterThanOrEqual": 1.0,
"lessThanOrEqual": 5.0
}
},
"source": "soda-core"
},
{
"identity": "3e24b78a",
"identities": {
"v1": "b659ac51",
"v2": "b58f5a02",
"v3": "3e24b78a"
},
"name": "Tipo Carga nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(tipo_carga):\n      warn: when between 0.5 and 1\n      fail: when > 1\n      name: Tipo Carga nulos\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 64,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "tipo_carga",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-tipo_carga-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [
{
"type": "failedRowsAnalysis",
"title": "Failed Rows Analysis",
"file": {
"columns": [
{
"name": "id",
"type": "int4"
},
{
"name": "ide",
"type": "int4"
},
{
"name": "idu",
"type": "int4"
},
{
"name": "data_registro",
"type": "timestamp"
},
{
"name": "nrmanifesto",
"type": "varchar"
},
{
"name": "dtemissao",
"type": "date"
},
{
"name": "dtoperacao",
"type": "date"
},
{
"name": "dtemissaoce",
"type": "date"
},
{
"name": "dtemissaocemaster",
"type": "date"
},
{
"name": "nrviagem",
"type": "varchar"
},
{
"name": "cdportocarregamento",
"type": "varchar"
},
{
"name": "nmportocarregamento",
"type": "varchar"
},
{
"name": "cdportodescarregamento",
"type": "varchar"
},
{
"name": "nmportodescarregamento",
"type": "varchar"
},
{
"name": "tptrafego",
"type": "varchar"
},
{
"name": "cdembarcacao",
"type": "varchar"
},
{
"name": "nmembarcacao",
"type": "varchar"
},
{
"name": "cdempresanavegacao",
"type": "varchar"
},
{
"name": "nmempresanavegacao",
"type": "varchar"
},
{
"name": "cdagencianavegacao",
"type": "varchar"
},
{
"name": "nmagencianavegacao",
"type": "varchar"
},
{
"name": "qtconhecimentosinformados",
"type": "int4"
},
{
"name": "qtconhecimentosexistentes",
"type": "int4"
},
{
"name": "qtconhecimentosassociado",
"type": "int4"
},
{
"name": "ckshipsconvenience",
"type": "varchar"
},
{
"name": "bloqueio_conhecimento",
"type": "varchar"
},
{
"name": "ret_pendente_analise",
"type": "varchar"
},
{
"name": "revisao_ou_pendencia_afrmm",
"type": "varchar"
},
{
"name": "pendencia_transito_maritimo",
"type": "varchar"
},
{
"name": "situacao_endosso",
"type": "varchar"
},
{
"name": "nrcemercante",
"type": "int8"
},
{
"name": "nrblconhecimento",
"type": "varchar"
},
{
"name": "vlcubagem",
"type": "numeric"
},
{
"name": "vlpesobruto",
"type": "numeric"
},
{
"name": "cdportoorigem",
"type": "varchar"
},
{
"name": "nmportoorigem",
"type": "varchar"
},
{
"name": "cdportodestino",
"type": "varchar"
},
{
"name": "nmportodestino",
"type": "varchar"
},
{
"name": "cdportooriginal",
"type": "varchar"
},
{
"name": "nmportooriginal",
"type": "varchar"
},
{
"name": "tipoconhecimento",
"type": "varchar"
},
{
"name": "categoriacarga",
"type": "varchar"
},
{
"name": "cdconsignatario",
"type": "varchar"
},
{
"name": "txshipper",
"type": "varchar"
},
{
"name": "txmercadoria",
"type": "varchar"
},
{
"name": "txobservacoes",
"type": "varchar"
},
{
"name": "txnotifypart",
"type": "varchar"
},
{
"name": "cdnotifypart",
"type": "varchar"
},
{
"name": "nmconsignatario",
"type": "varchar"
},
{
"name": "nrpassaporteconsignatario",
"type": "varchar"
},
{
"name": "txconsignatario",
"type": "varchar"
},
{
"name": "cdpracaentexterior",
"type": "varchar"
},
{
"name": "nmpracaentexterior",
"type": "varchar"
},
{
"name": "indblordem",
"type": "varchar"
},
{
"name": "cdmoedafrete",
"type": "varchar"
},
{
"name": "nmmoedafrete",
"type": "varchar"
},
{
"name": "oprecolhimentofrete",
"type": "varchar"
},
{
"name": "indmodalidadefrete",
"type": "varchar"
},
{
"name": "dtceorigem",
"type": "varchar"
},
{
"name": "nrceorigem",
"type": "varchar"
},
{
"name": "nmnavioorigem",
"type": "varchar"
},
{
"name": "transbordosexterior",
"type": "varchar"
},
{
"name": "ckblservico",
"type": "varchar"
},
{
"name": "nrceoriginal",
"type": "varchar"
},
{
"name": "cdterminalportuario",
"type": "varchar"
},
{
"name": "nmterminalportuario",
"type": "varchar"
},
{
"name": "indufdestino",
"type": "varchar"
},
{
"name": "nmufdestino",
"type": "varchar"
},
{
"name": "cdpaisprocedencia",
"type": "varchar"
},
{
"name": "nmpaisprocedencia",
"type": "varchar"
},
{
"name": "responsavel_cpf_conhec",
"type": "varchar"
},
{
"name": "responsavel_nome_conhec",
"type": "varchar"
},
{
"name": "responsavel_tipo_conhec",
"type": "varchar"
},
{
"name": "responsavel_ip_conhec",
"type": "varchar"
},
{
"name": "responsavel_data_conhec",
"type": "varchar"
},
{
"name": "itens",
"type": "varchar"
},
{
"name": "nrcemaster",
"type": "varchar"
},
{
"name": "qthouseinformadomaster",
"type": "varchar"
},
{
"name": "qthousedeclaradomaster",
"type": "varchar"
},
{
"name": "cdempnavegdesconmaster",
"type": "varchar"
},
{
"name": "nmempnavegdesconmaster",
"type": "varchar"
},
{
"name": "cdagtnavegmaster",
"type": "varchar"
},
{
"name": "nmagtnavegmaster",
"type": "varchar"
},
{
"name": "itemcarga_nritem",
"type": "varchar"
},
{
"name": "nrdde",
"type": "varchar"
},
{
"name": "indedse",
"type": "varchar"
},
{
"name": "cdpaisdestfinal",
"type": "varchar"
},
{
"name": "nmpaisdestfinal",
"type": "varchar"
},
{
"name": "cdshipper",
"type": "varchar"
},
{
"name": "qtveiculostransportadores",
"type": "varchar"
},
{
"name": "nova",
"type": "varchar"
},
{
"name": "opconsignatario",
"type": "varchar"
},
{
"name": "vlfretetotal",
"type": "numeric"
},
{
"name": "vlfrete",
"type": "numeric"
},
{
"name": "componentesfrete",
"type": "varchar"
},
{
"name": "cdterminalportuariocarregamento",
"type": "varchar"
},
{
"name": "nmterminalportuariocarregamento",
"type": "varchar"
},
{
"name": "vlfretetotalbld",
"type": "varchar"
},
{
"name": "dt_itens",
"type": "timestamp"
},
{
"name": "cdmoedafretetotalbld",
"type": "varchar"
},
{
"name": "ncm",
"type": "text"
},
{
"name": "ncm_desc",
"type": "text"
},
{
"name": "lista_fcl",
"type": "varchar"
},
{
"name": "tipo_fcl",
"type": "varchar"
},
{
"name": "tipo_carga",
"type": "varchar"
},
{
"name": "c20",
"type": "int4"
},
{
"name": "c40",
"type": "int4"
},
{
"name": "teus",
"type": "int4"
},
{
"name": "feus",
"type": "int4"
},
{
"name": "total_fcl",
"type": "int4"
},
{
"name": "cnpj_num",
"type": "numeric"
},
{
"name": "tpitem",
"type": "varchar"
},
{
"name": "itemcarga_vlpesobruto",
"type": "numeric"
},
{
"name": "itemcarga_vlcubagem",
"type": "numeric"
},
{
"name": "itemcarga_txmarca",
"type": "varchar"
},
{
"name": "itemcarga_txcontramarca",
"type": "varchar"
},
{
"name": "itemcarga_vlquantidade",
"type": "varchar"
},
{
"name": "itemcarga_cdindicadormercadoriaperigosa",
"type": "varchar"
},
{
"name": "itemcarga_cdclassemercadoriaperigosa",
"type": "varchar"
},
{
"name": "itemcarga_cdtipoembalagem",
"type": "varchar"
},
{
"name": "itemcarga_descricao_embalagem",
"type": "varchar"
},
{
"name": "itemcarga_cdncms",
"type": "varchar"
},
{
"name": "temcarga_cdncms",
"type": "varchar"
},
{
"name": "responsavel_cpf",
"type": "varchar"
},
{
"name": "responsavel_tipo",
"type": "varchar"
},
{
"name": "responsavel_ip",
"type": "varchar"
},
{
"name": "responsavel_data",
"type": "varchar"
},
{
"name": "responsavel_hora",
"type": "varchar"
},
{
"name": "itemcarga_noncms",
"type": "varchar"
},
{
"name": "itemcarga_tipocontainer",
"type": "varchar"
},
{
"name": "usoparcial",
"type": "varchar"
},
{
"name": "itemcarga_cdtipogranel",
"type": "varchar"
},
{
"name": "itemcarga_descricao_tipogranel",
"type": "varchar"
},
{
"name": "itemcarga_descricao",
"type": "varchar"
},
{
"name": "nmmoedafretetotalbld",
"type": "varchar"
},
{
"name": "dt_consulta_master",
"type": "timestamp"
},
{
"name": "incodacao",
"type": "varchar"
},
{
"name": "codigoncmitemcomplementar",
"type": "varchar"
},
{
"name": "descncmitemcomplementar",
"type": "varchar"
},
{
"name": "link",
"type": "varchar"
},
{
"name": "itemcarga_condncms",
"type": "varchar"
},
{
"name": "nomeconsiglce",
"type": "varchar"
},
{
"name": "enderecoconsiglce",
"type": "varchar"
},
{
"name": "nrruc",
"type": "varchar"
},
{
"name": "enviar",
"type": "varchar"
},
{
"name": "limpar",
"type": "varchar"
},
{
"name": "id_porto_origem",
"type": "int4"
},
{
"name": "id_porto_carga",
"type": "int4"
},
{
"name": "id_porto_descarga",
"type": "int4"
},
{
"name": "id_porto_destino",
"type": "int4"
},
{
"name": "id_armador",
"type": "int4"
},
{
"name": "id_agencia",
"type": "int4"
},
{
"name": "id_shipper",
"type": "int8"
},
{
"name": "id_consig",
"type": "int8"
},
{
"name": "id_shipconv",
"type": "int4"
},
{
"name": "id_notify",
"type": "int4"
},
{
"name": "id_agente",
"type": "int4"
},
{
"name": "id_agente_inter",
"type": "int4"
},
{
"name": "id_terminal_descarga",
"type": "int4"
},
{
"name": "id_terminal_carga",
"type": "int4"
},
{
"name": "id_moeda_frete",
"type": "int4"
},
{
"name": "id_pais_origem",
"type": "int4"
},
{
"name": "id_fcl",
"type": "int4"
},
{
"name": "id_pais_destino",
"type": "int4"
},
{
"name": "dt_cabotagem",
"type": "timestamp"
},
{
"name": "id_nvocc",
"type": "int4"
},
{
"name": "dt_update_agente",
"type": "timestamp"
},
{
"name": "tipoconhecimentooriginal",
"type": "varchar"
},
{
"name": "deleted_at",
"type": "timestamp"
},
{
"name": "teus_parcial",
"type": "numeric"
},
{
"name": "checked_at",
"type": "timestamp"
},
{
"name": "available_at",
"type": "timestamp"
},
{
"name": "processed_at",
"type": "timestamp"
},
{
"name": "checked_error_at",
"type": "timestamp"
},
{
"name": "obs_migr",
"type": "text"
},
{
"name": "id_recinto_aduaneiro",
"type": "int4"
},
{
"name": "cdncm_compl",
"type": "text"
},
{
"name": "lista_hscode",
"type": "text"
},
{
"name": "id_original",
"type": "int8"
},
{
"name": "desc_hscode",
"type": "text"
},
{
"name": "desc_ncm",
"type": "text"
},
{
"name": "verified_at",
"type": "timestamp"
},
{
"name": "ce_verified_at",
"type": "timestamp"
},
{
"name": "updated_at",
"type": "timestamptz"
},
{
"name": "id_embarcador_comp",
"type": "int8"
},
{
"name": "created_at",
"type": "timestamp"
},
{
"name": "cancelled_at",
"type": "timestamp"
},
{
"name": "possivel_data_entrega",
"type": "timestamp"
},
{
"name": "last_update_serpro_verified_at",
"type": "timestamp"
},
{
"name": "ce_dtoperacao_updated_at",
"type": "timestamp"
}
],
"totalRowCount": 3,
"storedRowCount": 3,
"reference": {
"type": "sodaCloudStorage",
"fileId": "5d9a0705-3e38-4b0f-a80b-fc419f3f8ec1"
}
},
"failingRowsQueryName": "cabotagem.db_maritimo[Five].tipo_carga.failed_rows[missing_count].failing_sql",
"passingRowsQueryName": "cabotagem.db_maritimo[Five].tipo_carga.failed_rows[missing_count].passing_sql",
"totalFailingRows": 3,
"sampleRowCount": 3
}
],
"value": 0.04,
"fail": {
"greaterThan": 1.0
},
"warn": {
"greaterThanOrEqual": 0.5,
"lessThanOrEqual": 1.0
}
},
"source": "soda-core"
},
{
"identity": "e8d2c9e5",
"identities": {
"v1": "1d3e5408",
"v2": "ac96be1e",
"v3": "e8d2c9e5"
},
"name": "Tipo de Carga com valor diferente de 'CONTEINER','SOLTA','GRANEL LIQUIDO', 'SOLIDOS', 'VEICULOS', 'LIQUIDOS', 'GASOSOS' e'OUTROS'",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - invalid_count(tipo_carga):\n      valid values: ['CONTEINER', 'SOLTA', 'GRANEL LIQUIDO', 'SOLIDOS', 'VEICULOS',\n        'LIQUIDOS', 'GASOSOS', 'OUTROS']\n      warn: when between 1 and 10\n      fail: when > 10\n      name: Tipo de Carga com valor diferente de 'CONTEINER','SOLTA','GRANEL LIQUIDO',\n        'SOLIDOS', 'VEICULOS', 'LIQUIDOS', 'GASOSOS' e'OUTROS'\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 71,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "tipo_carga",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-tipo_carga-invalid_count-d413226a"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0,
"fail": {
"greaterThan": 10.0
},
"warn": {
"greaterThanOrEqual": 1.0,
"lessThanOrEqual": 10.0
}
},
"source": "soda-core"
},
{
"identity": "7a63cb24",
"identities": {
"v1": "4a3709e7",
"v2": "759d56fc",
"v3": "7a63cb24"
},
"name": "Terminal de Embarque nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(nmterminalportuariocarregamento):\n      warn: when between 4 and 10\n      fail: when > 10\n      name: Terminal de Embarque nulos\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 79,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "nmterminalportuariocarregamento",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmterminalportuariocarregamento-missing_percent"
],
"outcome": "warn",
"diagnostics": {
"blocks": [
{
"type": "failedRowsAnalysis",
"title": "Failed Rows Analysis",
"file": {
"columns": [
{
"name": "id",
"type": "int4"
},
{
"name": "ide",
"type": "int4"
},
{
"name": "idu",
"type": "int4"
},
{
"name": "data_registro",
"type": "timestamp"
},
{
"name": "nrmanifesto",
"type": "varchar"
},
{
"name": "dtemissao",
"type": "date"
},
{
"name": "dtoperacao",
"type": "date"
},
{
"name": "dtemissaoce",
"type": "date"
},
{
"name": "dtemissaocemaster",
"type": "date"
},
{
"name": "nrviagem",
"type": "varchar"
},
{
"name": "cdportocarregamento",
"type": "varchar"
},
{
"name": "nmportocarregamento",
"type": "varchar"
},
{
"name": "cdportodescarregamento",
"type": "varchar"
},
{
"name": "nmportodescarregamento",
"type": "varchar"
},
{
"name": "tptrafego",
"type": "varchar"
},
{
"name": "cdembarcacao",
"type": "varchar"
},
{
"name": "nmembarcacao",
"type": "varchar"
},
{
"name": "cdempresanavegacao",
"type": "varchar"
},
{
"name": "nmempresanavegacao",
"type": "varchar"
},
{
"name": "cdagencianavegacao",
"type": "varchar"
},
{
"name": "nmagencianavegacao",
"type": "varchar"
},
{
"name": "qtconhecimentosinformados",
"type": "int4"
},
{
"name": "qtconhecimentosexistentes",
"type": "int4"
},
{
"name": "qtconhecimentosassociado",
"type": "int4"
},
{
"name": "ckshipsconvenience",
"type": "varchar"
},
{
"name": "bloqueio_conhecimento",
"type": "varchar"
},
{
"name": "ret_pendente_analise",
"type": "varchar"
},
{
"name": "revisao_ou_pendencia_afrmm",
"type": "varchar"
},
{
"name": "pendencia_transito_maritimo",
"type": "varchar"
},
{
"name": "situacao_endosso",
"type": "varchar"
},
{
"name": "nrcemercante",
"type": "int8"
},
{
"name": "nrblconhecimento",
"type": "varchar"
},
{
"name": "vlcubagem",
"type": "numeric"
},
{
"name": "vlpesobruto",
"type": "numeric"
},
{
"name": "cdportoorigem",
"type": "varchar"
},
{
"name": "nmportoorigem",
"type": "varchar"
},
{
"name": "cdportodestino",
"type": "varchar"
},
{
"name": "nmportodestino",
"type": "varchar"
},
{
"name": "cdportooriginal",
"type": "varchar"
},
{
"name": "nmportooriginal",
"type": "varchar"
},
{
"name": "tipoconhecimento",
"type": "varchar"
},
{
"name": "categoriacarga",
"type": "varchar"
},
{
"name": "cdconsignatario",
"type": "varchar"
},
{
"name": "txshipper",
"type": "varchar"
},
{
"name": "txmercadoria",
"type": "varchar"
},
{
"name": "txobservacoes",
"type": "varchar"
},
{
"name": "txnotifypart",
"type": "varchar"
},
{
"name": "cdnotifypart",
"type": "varchar"
},
{
"name": "nmconsignatario",
"type": "varchar"
},
{
"name": "nrpassaporteconsignatario",
"type": "varchar"
},
{
"name": "txconsignatario",
"type": "varchar"
},
{
"name": "cdpracaentexterior",
"type": "varchar"
},
{
"name": "nmpracaentexterior",
"type": "varchar"
},
{
"name": "indblordem",
"type": "varchar"
},
{
"name": "cdmoedafrete",
"type": "varchar"
},
{
"name": "nmmoedafrete",
"type": "varchar"
},
{
"name": "oprecolhimentofrete",
"type": "varchar"
},
{
"name": "indmodalidadefrete",
"type": "varchar"
},
{
"name": "dtceorigem",
"type": "varchar"
},
{
"name": "nrceorigem",
"type": "varchar"
},
{
"name": "nmnavioorigem",
"type": "varchar"
},
{
"name": "transbordosexterior",
"type": "varchar"
},
{
"name": "ckblservico",
"type": "varchar"
},
{
"name": "nrceoriginal",
"type": "varchar"
},
{
"name": "cdterminalportuario",
"type": "varchar"
},
{
"name": "nmterminalportuario",
"type": "varchar"
},
{
"name": "indufdestino",
"type": "varchar"
},
{
"name": "nmufdestino",
"type": "varchar"
},
{
"name": "cdpaisprocedencia",
"type": "varchar"
},
{
"name": "nmpaisprocedencia",
"type": "varchar"
},
{
"name": "responsavel_cpf_conhec",
"type": "varchar"
},
{
"name": "responsavel_nome_conhec",
"type": "varchar"
},
{
"name": "responsavel_tipo_conhec",
"type": "varchar"
},
{
"name": "responsavel_ip_conhec",
"type": "varchar"
},
{
"name": "responsavel_data_conhec",
"type": "varchar"
},
{
"name": "itens",
"type": "varchar"
},
{
"name": "nrcemaster",
"type": "varchar"
},
{
"name": "qthouseinformadomaster",
"type": "varchar"
},
{
"name": "qthousedeclaradomaster",
"type": "varchar"
},
{
"name": "cdempnavegdesconmaster",
"type": "varchar"
},
{
"name": "nmempnavegdesconmaster",
"type": "varchar"
},
{
"name": "cdagtnavegmaster",
"type": "varchar"
},
{
"name": "nmagtnavegmaster",
"type": "varchar"
},
{
"name": "itemcarga_nritem",
"type": "varchar"
},
{
"name": "nrdde",
"type": "varchar"
},
{
"name": "indedse",
"type": "varchar"
},
{
"name": "cdpaisdestfinal",
"type": "varchar"
},
{
"name": "nmpaisdestfinal",
"type": "varchar"
},
{
"name": "cdshipper",
"type": "varchar"
},
{
"name": "qtveiculostransportadores",
"type": "varchar"
},
{
"name": "nova",
"type": "varchar"
},
{
"name": "opconsignatario",
"type": "varchar"
},
{
"name": "vlfretetotal",
"type": "numeric"
},
{
"name": "vlfrete",
"type": "numeric"
},
{
"name": "componentesfrete",
"type": "varchar"
},
{
"name": "cdterminalportuariocarregamento",
"type": "varchar"
},
{
"name": "nmterminalportuariocarregamento",
"type": "varchar"
},
{
"name": "vlfretetotalbld",
"type": "varchar"
},
{
"name": "dt_itens",
"type": "timestamp"
},
{
"name": "cdmoedafretetotalbld",
"type": "varchar"
},
{
"name": "ncm",
"type": "text"
},
{
"name": "ncm_desc",
"type": "text"
},
{
"name": "lista_fcl",
"type": "varchar"
},
{
"name": "tipo_fcl",
"type": "varchar"
},
{
"name": "tipo_carga",
"type": "varchar"
},
{
"name": "c20",
"type": "int4"
},
{
"name": "c40",
"type": "int4"
},
{
"name": "teus",
"type": "int4"
},
{
"name": "feus",
"type": "int4"
},
{
"name": "total_fcl",
"type": "int4"
},
{
"name": "cnpj_num",
"type": "numeric"
},
{
"name": "tpitem",
"type": "varchar"
},
{
"name": "itemcarga_vlpesobruto",
"type": "numeric"
},
{
"name": "itemcarga_vlcubagem",
"type": "numeric"
},
{
"name": "itemcarga_txmarca",
"type": "varchar"
},
{
"name": "itemcarga_txcontramarca",
"type": "varchar"
},
{
"name": "itemcarga_vlquantidade",
"type": "varchar"
},
{
"name": "itemcarga_cdindicadormercadoriaperigosa",
"type": "varchar"
},
{
"name": "itemcarga_cdclassemercadoriaperigosa",
"type": "varchar"
},
{
"name": "itemcarga_cdtipoembalagem",
"type": "varchar"
},
{
"name": "itemcarga_descricao_embalagem",
"type": "varchar"
},
{
"name": "itemcarga_cdncms",
"type": "varchar"
},
{
"name": "temcarga_cdncms",
"type": "varchar"
},
{
"name": "responsavel_cpf",
"type": "varchar"
},
{
"name": "responsavel_tipo",
"type": "varchar"
},
{
"name": "responsavel_ip",
"type": "varchar"
},
{
"name": "responsavel_data",
"type": "varchar"
},
{
"name": "responsavel_hora",
"type": "varchar"
},
{
"name": "itemcarga_noncms",
"type": "varchar"
},
{
"name": "itemcarga_tipocontainer",
"type": "varchar"
},
{
"name": "usoparcial",
"type": "varchar"
},
{
"name": "itemcarga_cdtipogranel",
"type": "varchar"
},
{
"name": "itemcarga_descricao_tipogranel",
"type": "varchar"
},
{
"name": "itemcarga_descricao",
"type": "varchar"
},
{
"name": "nmmoedafretetotalbld",
"type": "varchar"
},
{
"name": "dt_consulta_master",
"type": "timestamp"
},
{
"name": "incodacao",
"type": "varchar"
},
{
"name": "codigoncmitemcomplementar",
"type": "varchar"
},
{
"name": "descncmitemcomplementar",
"type": "varchar"
},
{
"name": "link",
"type": "varchar"
},
{
"name": "itemcarga_condncms",
"type": "varchar"
},
{
"name": "nomeconsiglce",
"type": "varchar"
},
{
"name": "enderecoconsiglce",
"type": "varchar"
},
{
"name": "nrruc",
"type": "varchar"
},
{
"name": "enviar",
"type": "varchar"
},
{
"name": "limpar",
"type": "varchar"
},
{
"name": "id_porto_origem",
"type": "int4"
},
{
"name": "id_porto_carga",
"type": "int4"
},
{
"name": "id_porto_descarga",
"type": "int4"
},
{
"name": "id_porto_destino",
"type": "int4"
},
{
"name": "id_armador",
"type": "int4"
},
{
"name": "id_agencia",
"type": "int4"
},
{
"name": "id_shipper",
"type": "int8"
},
{
"name": "id_consig",
"type": "int8"
},
{
"name": "id_shipconv",
"type": "int4"
},
{
"name": "id_notify",
"type": "int4"
},
{
"name": "id_agente",
"type": "int4"
},
{
"name": "id_agente_inter",
"type": "int4"
},
{
"name": "id_terminal_descarga",
"type": "int4"
},
{
"name": "id_terminal_carga",
"type": "int4"
},
{
"name": "id_moeda_frete",
"type": "int4"
},
{
"name": "id_pais_origem",
"type": "int4"
},
{
"name": "id_fcl",
"type": "int4"
},
{
"name": "id_pais_destino",
"type": "int4"
},
{
"name": "dt_cabotagem",
"type": "timestamp"
},
{
"name": "id_nvocc",
"type": "int4"
},
{
"name": "dt_update_agente",
"type": "timestamp"
},
{
"name": "tipoconhecimentooriginal",
"type": "varchar"
},
{
"name": "deleted_at",
"type": "timestamp"
},
{
"name": "teus_parcial",
"type": "numeric"
},
{
"name": "checked_at",
"type": "timestamp"
},
{
"name": "available_at",
"type": "timestamp"
},
{
"name": "processed_at",
"type": "timestamp"
},
{
"name": "checked_error_at",
"type": "timestamp"
},
{
"name": "obs_migr",
"type": "text"
},
{
"name": "id_recinto_aduaneiro",
"type": "int4"
},
{
"name": "cdncm_compl",
"type": "text"
},
{
"name": "lista_hscode",
"type": "text"
},
{
"name": "id_original",
"type": "int8"
},
{
"name": "desc_hscode",
"type": "text"
},
{
"name": "desc_ncm",
"type": "text"
},
{
"name": "verified_at",
"type": "timestamp"
},
{
"name": "ce_verified_at",
"type": "timestamp"
},
{
"name": "updated_at",
"type": "timestamptz"
},
{
"name": "id_embarcador_comp",
"type": "int8"
},
{
"name": "created_at",
"type": "timestamp"
},
{
"name": "cancelled_at",
"type": "timestamp"
},
{
"name": "possivel_data_entrega",
"type": "timestamp"
},
{
"name": "last_update_serpro_verified_at",
"type": "timestamp"
},
{
"name": "ce_dtoperacao_updated_at",
"type": "timestamp"
}
],
"totalRowCount": 100,
"storedRowCount": 100,
"reference": {
"type": "sodaCloudStorage",
"fileId": "79a265a9-1ced-4203-b2aa-df9965bee438"
}
},
"failingRowsQueryName": "cabotagem.db_maritimo[Five].nmterminalportuariocarregamento.failed_rows[missing_count].failing_sql",
"passingRowsQueryName": "cabotagem.db_maritimo[Five].nmterminalportuariocarregamento.failed_rows[missing_count].passing_sql",
"totalFailingRows": 297,
"sampleRowCount": 100
}
],
"value": 4.28,
"fail": {
"greaterThan": 10.0
},
"warn": {
"greaterThanOrEqual": 4.0,
"lessThanOrEqual": 10.0
}
},
"source": "soda-core"
},
{
"identity": "582dcc62",
"identities": {
"v1": "68526910",
"v2": "7a481163",
"v3": "582dcc62"
},
"name": "Porto de Origem nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(nmportoorigem) = 0:\n      name: Porto de Origem nulos\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 86,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "nmportoorigem",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmportoorigem-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 0.0,
"lessThan": 0.0
}
},
"source": "soda-c
11:18:35 AM
soda_scan_execute-1578bf38-0
ore"
},
{
"identity": "9fed7253",
"identities": {
"v1": "9315638e",
"v2": "46b3fb8c",
"v3": "9fed7253"
},
"name": "Porto de Destino nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(nmportodestino):\n      warn: when between 0.5 and 1\n      fail: when > 1\n      name: Porto de Destino nulos\n\n  # - missing_percent(tipo_fcl):\n  #     warn: \n  #       when between 1 and 2\n  #     fail: \n  #       when > 2\n  #     name: tipo_fcl nulos  \n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 89,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "nmportodestino",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmportodestino-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 1.0
},
"warn": {
"greaterThanOrEqual": 0.5,
"lessThanOrEqual": 1.0
}
},
"source": "soda-core"
},
{
"identity": "fda03b3a",
"identities": {
"v1": "753f4909",
"v2": "b0c5abd0",
"v3": "fda03b3a"
},
"name": "CRITICAL - Tipo Conhecimento nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(tipoconhecimento) = 0:\n      name: CRITICAL - Tipo Conhecimento nulos\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 103,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "tipoconhecimento",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-tipoconhecimento-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 0.0,
"lessThan": 0.0
}
},
"source": "soda-core"
},
{
"identity": "5c35b455",
"identities": {
"v1": "f43cb340",
"v2": "b3bf8366",
"v3": "5c35b455"
},
"name": "Cod Navio nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(cdembarcacao) = 0:\n      name: Cod Navio nulos\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 106,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "cdembarcacao",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-cdembarcacao-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 0.0,
"lessThan": 0.0
}
},
"source": "soda-core"
},
{
"identity": "f0902995",
"identities": {
"v1": "ae5126ce",
"v2": "b09d9722",
"v3": "f0902995"
},
"name": "Navio nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(nmembarcacao) = 0:\n      name: Navio nulos\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 109,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "nmembarcacao",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmembarcacao-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 0.0,
"lessThan": 0.0
}
},
"source": "soda-core"
},
{
"identity": "33571644",
"identities": {
"v1": "ec13d526",
"v2": "4c2c7619",
"v3": "33571644"
},
"name": "Nr Viagem nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(nrviagem) = 0:\n      name: Nr Viagem nulos\n\n#   - missing_percent(nmpaisprocedencia):\n#       name: (Acompanhamento) Pais de Proced\u00eancia nulos        \n#       warn: \n#         when between 5 and 35\n#       fail: \n#         when > 35\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 112,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "nrviagem",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-nrviagem-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 0.0,
"lessThan": 0.0
}
},
"source": "soda-core"
},
{
"identity": "61e66ac1",
"identities": {
"v1": "daedecde",
"v2": "0de650ff",
"v3": "61e66ac1"
},
"name": "CNPJ Destinat\u00e1rio nulo",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(cdconsignatario):\n      warn: when between 0.5 and 1\n      fail: when > 1\n      name: CNPJ Destinat\u00e1rio nulo\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 122,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "cdconsignatario",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-cdconsignatario-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 1.0
},
"warn": {
"greaterThanOrEqual": 0.5,
"lessThanOrEqual": 1.0
}
},
"source": "soda-core"
},
{
"identity": "53164985",
"identities": {
"v1": "ce4122dc",
"v2": "c5922154",
"v3": "53164985"
},
"name": "Nome Destinat\u00e1rio nulo",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(nmconsignatario):\n      warn: when between 0.5 and 1\n      fail: when > 1\n      name: Nome Destinat\u00e1rio nulo\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 129,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "nmconsignatario",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-nmconsignatario-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 1.0
},
"warn": {
"greaterThanOrEqual": 0.5,
"lessThanOrEqual": 1.0
}
},
"source": "soda-core"
},
{
"identity": "43ac0a61",
"identities": {
"v1": "0463d5b3",
"v2": "d26ed9cf",
"v3": "43ac0a61"
},
"name": "CNPJ Remetente nulo",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(cdshipper):\n      warn: when between 0.5 and 1\n      fail: when > 1\n      name: CNPJ Remetente nulo\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 136,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "cdshipper",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-cdshipper-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [
{
"type": "failedRowsAnalysis",
"title": "Failed Rows Analysis",
"file": {
"columns": [
{
"name": "id",
"type": "int4"
},
{
"name": "ide",
"type": "int4"
},
{
"name": "idu",
"type": "int4"
},
{
"name": "data_registro",
"type": "timestamp"
},
{
"name": "nrmanifesto",
"type": "varchar"
},
{
"name": "dtemissao",
"type": "date"
},
{
"name": "dtoperacao",
"type": "date"
},
{
"name": "dtemissaoce",
"type": "date"
},
{
"name": "dtemissaocemaster",
"type": "date"
},
{
"name": "nrviagem",
"type": "varchar"
},
{
"name": "cdportocarregamento",
"type": "varchar"
},
{
"name": "nmportocarregamento",
"type": "varchar"
},
{
"name": "cdportodescarregamento",
"type": "varchar"
},
{
"name": "nmportodescarregamento",
"type": "varchar"
},
{
"name": "tptrafego",
"type": "varchar"
},
{
"name": "cdembarcacao",
"type": "varchar"
},
{
"name": "nmembarcacao",
"type": "varchar"
},
{
"name": "cdempresanavegacao",
"type": "varchar"
},
{
"name": "nmempresanavegacao",
"type": "varchar"
},
{
"name": "cdagencianavegacao",
"type": "varchar"
},
{
"name": "nmagencianavegacao",
"type": "varchar"
},
{
"name": "qtconhecimentosinformados",
"type": "int4"
},
{
"name": "qtconhecimentosexistentes",
"type": "int4"
},
{
"name": "qtconhecimentosassociado",
"type": "int4"
},
{
"name": "ckshipsconvenience",
"type": "varchar"
},
{
"name": "bloqueio_conhecimento",
"type": "varchar"
},
{
"name": "ret_pendente_analise",
"type": "varchar"
},
{
"name": "revisao_ou_pendencia_afrmm",
"type": "varchar"
},
{
"name": "pendencia_transito_maritimo",
"type": "varchar"
},
{
"name": "situacao_endosso",
"type": "varchar"
},
{
"name": "nrcemercante",
"type": "int8"
},
{
"name": "nrblconhecimento",
"type": "varchar"
},
{
"name": "vlcubagem",
"type": "numeric"
},
{
"name": "vlpesobruto",
"type": "numeric"
},
{
"name": "cdportoorigem",
"type": "varchar"
},
{
"name": "nmportoorigem",
"type": "varchar"
},
{
"name": "cdportodestino",
"type": "varchar"
},
{
"name": "nmportodestino",
"type": "varchar"
},
{
"name": "cdportooriginal",
"type": "varchar"
},
{
"name": "nmportooriginal",
"type": "varchar"
},
{
"name": "tipoconhecimento",
"type": "varchar"
},
{
"name": "categoriacarga",
"type": "varchar"
},
{
"name": "cdconsignatario",
"type": "varchar"
},
{
"name": "txshipper",
"type": "varchar"
},
{
"name": "txmercadoria",
"type": "varchar"
},
{
"name": "txobservacoes",
"type": "varchar"
},
{
"name": "txnotifypart",
"type": "varchar"
},
{
"name": "cdnotifypart",
"type": "varchar"
},
{
"name": "nmconsignatario",
"type": "varchar"
},
{
"name": "nrpassaporteconsignatario",
"type": "varchar"
},
{
"name": "txconsignatario",
"type": "varchar"
},
{
"name": "cdpracaentexterior",
"type": "varchar"
},
{
"name": "nmpracaentexterior",
"type": "varchar"
},
{
"name": "indblordem",
"type": "varchar"
},
{
"name": "cdmoedafrete",
"type": "varchar"
},
{
"name": "nmmoedafrete",
"type": "varchar"
},
{
"name": "oprecolhimentofrete",
"type": "varchar"
},
{
"name": "indmodalidadefrete",
"type": "varchar"
},
{
"name": "dtceorigem",
"type": "varchar"
},
{
"name": "nrceorigem",
"type": "varchar"
},
{
"name": "nmnavioorigem",
"type": "varchar"
},
{
"name": "transbordosexterior",
"type": "varchar"
},
{
"name": "ckblservico",
"type": "varchar"
},
{
"name": "nrceoriginal",
"type": "varchar"
},
{
"name": "cdterminalportuario",
"type": "varchar"
},
{
"name": "nmterminalportuario",
"type": "varchar"
},
{
"name": "indufdestino",
"type": "varchar"
},
{
"name": "nmufdestino",
"type": "varchar"
},
{
"name": "cdpaisprocedencia",
"type": "varchar"
},
{
"name": "nmpaisprocedencia",
"type": "varchar"
},
{
"name": "responsavel_cpf_conhec",
"type": "varchar"
},
{
"name": "responsavel_nome_conhec",
"type": "varchar"
},
{
"name": "responsavel_tipo_conhec",
"type": "varchar"
},
{
"name": "responsavel_ip_conhec",
"type": "varchar"
},
{
"name": "responsavel_data_conhec",
"type": "varchar"
},
{
"name": "itens",
"type": "varchar"
},
{
"name": "nrcemaster",
"type": "varchar"
},
{
"name": "qthouseinformadomaster",
"type": "varchar"
},
{
"name": "qthousedeclaradomaster",
"type": "varchar"
},
{
"name": "cdempnavegdesconmaster",
"type": "varchar"
},
{
"name": "nmempnavegdesconmaster",
"type": "varchar"
},
{
"name": "cdagtnavegmaster",
"type": "varchar"
},
{
"name": "nmagtnavegmaster",
"type": "varchar"
},
{
"name": "itemcarga_nritem",
"type": "varchar"
},
{
"name": "nrdde",
"type": "varchar"
},
{
"name": "indedse",
"type": "varchar"
},
{
"name": "cdpaisdestfinal",
"type": "varchar"
},
{
"name": "nmpaisdestfinal",
"type": "varchar"
},
{
"name": "cdshipper",
"type": "varchar"
},
{
"name": "qtveiculostransportadores",
"type": "varchar"
},
{
"name": "nova",
"type": "varchar"
},
{
"name": "opconsignatario",
"type": "varchar"
},
{
"name": "vlfretetotal",
"type": "numeric"
},
{
"name": "vlfrete",
"type": "numeric"
},
{
"name": "componentesfrete",
"type": "varchar"
},
{
"name": "cdterminalportuariocarregamento",
"type": "varchar"
},
{
"name": "nmterminalportuariocarregamento",
"type": "varchar"
},
{
"name": "vlfretetotalbld",
"type": "varchar"
},
{
"name": "dt_itens",
"type": "timestamp"
},
{
"name": "cdmoedafretetotalbld",
"type": "varchar"
},
{
"name": "ncm",
"type": "text"
},
{
"name": "ncm_desc",
"type": "text"
},
{
"name": "lista_fcl",
"type": "varchar"
},
{
"name": "tipo_fcl",
"type": "varchar"
},
{
"name": "tipo_carga",
"type": "varchar"
},
{
"name": "c20",
"type": "int4"
},
{
"name": "c40",
"type": "int4"
},
{
"name": "teus",
"type": "int4"
},
{
"name": "feus",
"type": "int4"
},
{
"name": "total_fcl",
"type": "int4"
},
{
"name": "cnpj_num",
"type": "numeric"
},
{
"name": "tpitem",
"type": "varchar"
},
{
"name": "itemcarga_vlpesobruto",
"type": "numeric"
},
{
"name": "itemcarga_vlcubagem",
"type": "numeric"
},
{
"name": "itemcarga_txmarca",
"type": "varchar"
},
{
"name": "itemcarga_txcontramarca",
"type": "varchar"
},
{
"name": "itemcarga_vlquantidade",
"type": "varchar"
},
{
"name": "itemcarga_cdindicadormercadoriaperigosa",
"type": "varchar"
},
{
"name": "itemcarga_cdclassemercadoriaperigosa",
"type": "varchar"
},
{
"name": "itemcarga_cdtipoembalagem",
"type": "varchar"
},
{
"name": "itemcarga_descricao_embalagem",
"type": "varchar"
},
{
"name": "itemcarga_cdncms",
"type": "varchar"
},
{
"name": "temcarga_cdncms",
"type": "varchar"
},
{
"name": "responsavel_cpf",
"type": "varchar"
},
{
"name": "responsavel_tipo",
"type": "varchar"
},
{
"name": "responsavel_ip",
"type": "varchar"
},
{
"name": "responsavel_data",
"type": "varchar"
},
{
"name": "responsavel_hora",
"type": "varchar"
},
{
"name": "itemcarga_noncms",
"type": "varchar"
},
{
"name": "itemcarga_tipocontainer",
"type": "varchar"
},
{
"name": "usoparcial",
"type": "varchar"
},
{
"name": "itemcarga_cdtipogranel",
"type": "varchar"
},
{
"name": "itemcarga_descricao_tipogranel",
"type": "varchar"
},
{
"name": "itemcarga_descricao",
"type": "varchar"
},
{
"name": "nmmoedafretetotalbld",
"type": "varchar"
},
{
"name": "dt_consulta_master",
"type": "timestamp"
},
{
"name": "incodacao",
"type": "varchar"
},
{
"name": "codigoncmitemcomplementar",
"type": "varchar"
},
{
"name": "descncmitemcomplementar",
"type": "varchar"
},
{
"name": "link",
"type": "varchar"
},
{
"name": "itemcarga_condncms",
"type": "varchar"
},
{
"name": "nomeconsiglce",
"type": "varchar"
},
{
"name": "enderecoconsiglce",
"type": "varchar"
},
{
"name": "nrruc",
"type": "varchar"
},
{
"name": "enviar",
"type": "varchar"
},
{
"name": "limpar",
"type": "varchar"
},
{
"name": "id_porto_origem",
"type": "int4"
},
{
"name": "id_porto_carga",
"type": "int4"
},
{
"name": "id_porto_descarga",
"type": "int4"
},
{
"name": "id_porto_destino",
"type": "int4"
},
{
"name": "id_armador",
"type": "int4"
},
{
"name": "id_agencia",
"type": "int4"
},
{
"name": "id_shipper",
"type": "int8"
},
{
"name": "id_consig",
"type": "int8"
},
{
"name": "id_shipconv",
"type": "int4"
},
{
"name": "id_notify",
"type": "int4"
},
{
"name": "id_agente",
"type": "int4"
},
{
"name": "id_agente_inter",
"type": "int4"
},
{
"name": "id_terminal_descarga",
"type": "int4"
},
{
"name": "id_terminal_carga",
"type": "int4"
},
{
"name": "id_moeda_frete",
"type": "int4"
},
{
"name": "id_pais_origem",
"type": "int4"
},
{
"name": "id_fcl",
"type": "int4"
},
{
"name": "id_pais_destino",
"type": "int4"
},
{
"name": "dt_cabotagem",
"type": "timestamp"
},
{
"name": "id_nvocc",
"type": "int4"
},
{
"name": "dt_update_agente",
"type": "timestamp"
},
{
"name": "tipoconhecimentooriginal",
"type": "varchar"
},
{
"name": "deleted_at",
"type": "timestamp"
},
{
"name": "teus_parcial",
"type": "numeric"
},
{
"name": "checked_at",
"type": "timestamp"
},
{
"name": "available_at",
"type": "timestamp"
},
{
"name": "processed_at",
"type": "timestamp"
},
{
"name": "checked_error_at",
"type": "timestamp"
},
{
"name": "obs_migr",
"type": "text"
},
{
"name": "id_recinto_aduaneiro",
"type": "int4"
},
{
"name": "cdncm_compl",
"type": "text"
},
{
"name": "lista_hscode",
"type": "text"
},
{
"name": "id_original",
"type": "int8"
},
{
"name": "desc_hscode",
"type": "text"
},
{
"name": "desc_ncm",
"type": "text"
},
{
"name": "verified_at",
"type": "timestamp"
},
{
"name": "ce_verified_at",
"type": "timestamp"
},
{
"name": "updated_at",
"type": "timestamptz"
},
{
"name": "id_embarcador_comp",
"type": "int8"
},
{
"name": "created_at",
"type": "timestamp"
},
{
"name": "cancelled_at",
"type": "timestamp"
},
{
"name": "possivel_data_entrega",
"type": "timestamp"
},
{
"name": "last_update_serpro_verified_at",
"type": "timestamp"
},
{
"name": "ce_dtoperacao_updated_at",
"type": "timestamp"
}
],
"totalRowCount": 25,
"storedRowCount": 25,
"reference": {
"type": "sodaCloudStorage",
"fileId": "a8f4fbe0-6d32-443a-a0b1-edc775d94d44"
}
},
"failingRowsQueryName": "cabotagem.db_maritimo[Five].cdshipper.failed_rows[missing_count].failing_sql",
"passingRowsQueryName": "cabotagem.db_maritimo[Five].cdshipper.failed_rows[missing_count].passing_sql",
"totalFailingRows": 25,
"sampleRowCount": 25
}
],
"value": 0.36,
"fail": {
"greaterThan": 1.0
},
"warn": {
"greaterThanOrEqual": 0.5,
"lessThanOrEqual": 1.0
}
},
"source": "soda-core"
},
{
"identity": "2673c8f2",
"identities": {
"v1": "a33e0281",
"v2": "abec18d0",
"v3": "2673c8f2"
},
"name": "Teus nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(teus):\n      warn: when between 0.5 and 1\n      fail: when > 1\n      name: Teus nulos\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 143,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "teus",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-teus-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 1.0
},
"warn": {
"greaterThanOrEqual": 0.5,
"lessThanOrEqual": 1.0
}
},
"source": "soda-core"
},
{
"identity": "b13d676d",
"identities": {
"v1": "40d9c23c",
"v2": "8858899b",
"v3": "b13d676d"
},
"name": "Teus deve ser maior ou igual a 0",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - min(teus) >= 0:\n      name: Teus deve ser maior ou igual a 0\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 150,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "teus",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-teus-min"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0,
"fail": {
"lessThan": 0.0
}
},
"source": "soda-core"
},
{
"identity": "d32b8663",
"identities": {
"v1": "ac585634",
"v2": "c4cceb0a",
"v3": "d32b8663"
},
"name": "c20 nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(c20):\n      warn: when between 0.5 and 1\n      fail: when > 1\n      name: c20 nulos\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 153,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "c20",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-c20-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 1.0
},
"warn": {
"greaterThanOrEqual": 0.5,
"lessThanOrEqual": 1.0
}
},
"source": "soda-core"
},
{
"identity": "42e17b67",
"identities": {
"v1": "9e15e304",
"v2": "3dbef5d3",
"v3": "42e17b67"
},
"name": "C20 deve ser maior ou igual a 0",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - min(c20) >= 0:\n      name: C20 deve ser maior ou igual a 0\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 160,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "c20",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-c20-min"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0,
"fail": {
"lessThan": 0.0
}
},
"source": "soda-core"
},
{
"identity": "d699fa63",
"identities": {
"v1": "d5f914e2",
"v2": "7728b87d",
"v3": "d699fa63"
},
"name": "C40 nulos",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - missing_percent(c40):\n      warn: when between 0.5 and 1\n      fail: when > 1\n      name: C40 nulos\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 163,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "c40",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-c40-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 1.0
},
"warn": {
"greaterThanOrEqual": 0.5,
"lessThanOrEqual": 1.0
}
},
"source": "soda-core"
},
{
"identity": "fce62f21",
"identities": {
"v1": "ade78162",
"v2": "732031de",
"v3": "fce62f21"
},
"name": "C40 deve ser maior ou igual a 0",
"type": "generic",
"definition": "checks for db_maritimo [Five]:\n  - min(c40) >= 0:\n      name: C40 deve ser maior ou igual a 0\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 170,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "c40",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Five-c40-min"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0,
"fail": {
"lessThan": 0.0
}
},
"source": "soda-core"
},
{
"identity": "c74809f1",
"identities": {
"v1": "c8c4290f",
"v2": "6965c197",
"v3": "c74809f1"
},
"name": "CEs duplicados",
"type": "generic",
"definition": "checks for db_maritimo [Duplicated]:\n  - duplicate_count(nrcemercante) = 0:\n      name: CEs duplicados\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 28,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "nrcemercante",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Duplicated-nrcemercante-duplicate_count"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0,
"fail": {
"greaterThan": 0.0,
"lessThan": 0.0
}
},
"source": "soda-core"
},
{
"identity": "21ad50e6",
"identities": {
"v1": "666340bf",
"v2": "e0377996",
"v3": "21ad50e6"
},
"name": "Sem registros criados na data de hoje",
"type": "generic",
"definition": "checks for db_maritimo [Today]:\n  - row_count > 0:\n      name: Sem registros criados na data de hoje\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 15,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": null,
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Today-row_count"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 1,
"fail": {
"lessThanOrEqual": 0.0
}
},
"source": "soda-core"
},
{
"identity": "d0a35a60",
"identities": {
"v1": "d814b245",
"v2": "2d7e6643",
"v3": "d0a35a60"
},
"name": "Data Opera\u00e7\u00e3o null",
"type": "generic",
"definition": "checks for db_maritimo [Today]:\n  - missing_percent(dtoperacao) = 0:\n      name: Data Opera\u00e7\u00e3o null\n\n",
"resourceAttributes": [],
"location": {
"filePath": "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml",
"line": 18,
"col": 5
},
"dataSource": "cabotagem",
"table": "db_maritimo",
"column": "dtoperacao",
"metrics": [
"metric-Soda Core CLI-cabotagem-db_maritimo-Today-dtoperacao-missing_percent"
],
"outcome": "pass",
"diagnostics": {
"blocks": [],
"value": 0.0,
"fail": {
"greaterThan": 0.0,
"lessThan": 0.0
}
},
"source": "soda-core"
}
],
"queries": [
{
"name": "cabotagem.db_maritimo[Five].aggregation[0]",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": null,
"sql": "SELECT \n  COUNT(),\n  COUNT(CASE WHEN id IS NULL THEN 1 END),\n  COUNT(CASE WHEN nrcemercante IS NULL THEN 1 END),\n  COUNT(CASE WHEN nmempresanavegacao IS NULL THEN 1 END),\n  COUNT(CASE WHEN cdempresanavegacao IS NULL THEN 1 END),\n  COUNT(CASE WHEN tipo_carga IS NULL THEN 1 END),\n  COUNT(CASE WHEN NOT (tipo_carga IS NULL) AND NOT (tipo_carga IN ('CONTEINER','SOLTA','GRANEL LIQUIDO','SOLIDOS','VEICULOS','LIQUIDOS','GASOSOS','OUTROS')) THEN 1 END),\n  COUNT(CASE WHEN nmterminalportuariocarregamento IS NULL THEN 1 END),\n  COUNT(CASE WHEN nmportoorigem IS NULL THEN 1 END),\n  COUNT(CASE WHEN nmportodestino IS NULL THEN 1 END),\n  COUNT(CASE WHEN tipoconhecimento IS NULL THEN 1 END),\n  COUNT(CASE WHEN cdembarcacao IS NULL THEN 1 END),\n  COUNT(CASE WHEN nmembarcacao IS NULL THEN 1 END),\n  COUNT(CASE WHEN nrviagem IS NULL THEN 1 END),\n  COUNT(CASE WHEN cdconsignatario IS NULL THEN 1 END),\n  COUNT(CASE WHEN nmconsignatario IS NULL THEN 1 END),\n  COUNT(CASE WHEN cdshipper IS NULL THEN 1 END),\n  COUNT(CASE WHEN teus IS NULL THEN 1 END),\n  MIN(teus),\n  COUNT(CASE WHEN c20 IS NULL THEN 1 END),\n  MIN(c20),\n  COUNT(CASE WHEN c40 IS NULL THEN 1 END),\n  MIN(c40) \nFROM public.db_maritimo\nWHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL",
"exception": null,
"duration": "0:03:52.761337"
},
{
"name": "cabotagem.db_maritimo[Five].tipo_carga.failed_rows[missing_count]",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "tipo_carga",
"sql": "SELECT * FROM public.db_maritimo \n WHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL AND tipo_carga IS NULL \n LIMIT 100",
"exception": null,
"duration": "0:00:27.917024"
},
{
"name": "cabotagem.db_maritimo[Five].tipo_carga.failed_rows[missing_count].failing_sql",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "tipo_carga",
"sql": "SELECT * FROM public.db_maritimo \n WHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL AND tipo_carga IS NULL",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Five].tipo_carga.failed_rows[missing_count].passing_sql",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "tipo_carga",
"sql": "SELECT * FROM public.db_maritimo \n WHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL AND NOT (tipo_carga IS NULL)",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Five].nmterminalportuariocarregamento.failed_rows[missing_count]",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "nmterminalportuariocarregamento",
"sql": "SELECT * FROM public.db_maritimo \n WHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL AND nmterminalportuariocarregamento IS NULL \n LIMIT 100",
"exception": null,
"duration": "0:00:02.222375"
},
{
"name": "cabotagem.db_maritimo[Five].nmterminalportuariocarregamento.failed_rows[missing_count].failing_sql",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "nmterminalportuariocarregamento",
"sql": "SELECT * FROM public.db_maritimo \n WHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL AND nmterminalportuariocarregamento IS NULL",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Five].nmterminalportuariocarregamento.failed_rows[missing_count].passing_sql",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "nmterminalportuariocarregamento",
"sql": "SELECT * FROM public.db_maritimo \n WHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL AND NOT (nmterminalportuariocarregamento IS NULL)",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Five].cdshipper.failed_rows[missing_count]",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "cdshipper",
"sql": "SELECT * FROM public.db_maritimo \n WHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL AND cdshipper IS NULL \n LIMIT 100",
"exception": null,
"duration": "0:00:02.219778"
},
{
"name": "cabotagem.db_maritimo[Five].cdshipper.failed_rows[missing_count].failing_sql",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "cdshipper",
"sql": "SELECT * FROM public.db_maritimo \n WHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL AND cdshipper IS NULL",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Five].cdshipper.failed_rows[missing_count].passing_sql",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "cdshipper",
"sql": "SELECT * FROM public.db_maritimo \n WHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL AND NOT (cdshipper IS NULL)",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Five].id.duplicate_count",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "id",
"sql": "\n          WITH frequencies AS (\n              SELECT COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE id IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY id)\n          SELECT count()\n          FROM frequencies\n          WHERE frequency > 1",
"exception": null,
"duration": "0:00:00.512133"
},
{
"name": "cabotagem.db_maritimo[Five].id.duplicate_count.failing_sql",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "id",
"sql": "\n          WITH frequencies AS (\n              SELECT id\n              FROM public.db_maritimo\n              WHERE id IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY id\n              HAVING count() > 1)\n          SELECT main.\n          FROM public.db_maritimo main\n          JOIN frequencies ON main.id = frequencies.id\n",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Five].id.duplicate_count.passing_sql",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "id",
"sql": "\n          WITH frequencies AS (\n              SELECT id\n              FROM public.db_maritimo\n              WHERE id IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY id\n              HAVING count() <= 1)\n          SELECT main.\n          FROM public.db_maritimo main\n          JOIN frequencies ON main.id = frequencies.id\n",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Five].id.duplicate_count.failing_rows_sql_aggregated",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "id",
"sql": "\n          WITH frequencies AS (\n              SELECT id, COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE id IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY id)\n          SELECT \n          FROM frequencies\n          WHERE frequency > 1\n          ORDER BY frequency DESC\nLIMIT 100",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Five].duplicate_count[id].failed_rows.aggregated",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": null,
"sql": "\n          WITH frequencies AS (\n              SELECT id, COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE id IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY id)\n          SELECT \n          FROM frequencies\n          WHERE frequency > 1\n          ORDER BY frequency DESC\nLIMIT 100",
"exception": null,
"duration": "0:00:00.512143"
},
{
"name": "cabotagem.db_maritimo[Five].nrcemercante.duplicate_count",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "nrcemercante",
"sql": "\n          WITH frequencies AS (\n              SELECT COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY nrcemercante)\n          SELECT count()\n          FROM frequencies\n          WHERE frequency > 1",
"exception": null,
"duration": "0:00:00.409388"
},
{
"name": "cabotagem.db_maritimo[Five].nrcemercante.duplicate_count.failing_sql",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "nrcemercante",
"sql": "\n          WITH frequencies AS (\n              SELECT nrcemercante\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY nrcemercante\n              HAVING count() > 1)\n          SELECT main.\n          FROM public.db_maritimo main\n          JOIN frequencies ON main.nrcemercante = frequencies.nrcemercante\n",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Five].nrcemercante.duplicate_count.passing_sql",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "nrcemercante",
"sql": "\n          WITH frequencies AS (\n              SELECT nrcemercante\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY nrcemercante\n              HAVING count() <= 1)\n          SELECT main.\n          FROM public.db_maritimo main\n          JOIN frequencies ON main.nrcemercante = frequencies.nrcemercante\n",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Five].nrcemercante.duplicate_count.failing_rows_sql_aggregated",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": "nrcemercante",
"sql": "\n          WITH frequencies AS (\n              SELECT nrcemercante, COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY nrcemercante)\n          SELECT \n          FROM frequencies\n          WHERE frequency > 1\n          ORDER BY frequency DESC\nLIMIT 100",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Five].duplicate_count[nrcemercante].failed_rows.aggregated",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Five",
"column": null,
"sql": "\n          WITH frequencies AS (\n              SELECT nrcemercante, COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY nrcemercante)\n          SELECT \n          FROM frequencies\n          WHERE frequency > 1\n          ORDER BY frequency DESC\nLIMIT 100",
"exception": null,
"duration": "0:00:00.392143"
},
{
"name": "cabotagem.db_maritimo[Duplicated].nrcemercante.duplicate_count",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Duplicated",
"column": "nrcemercante",
"sql": "\n          WITH frequencies AS (\n              SELECT COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND nrblconhecimento is not null AND deleted_at is null and dtoperacao >= DATE(current_date - interval '45 days') and dtoperacao < DATE(current_date + interval '15 days') AND lista_fcl is not null AND lista_fcl != '' AND categoriacarga = 'N' AND nrcemercante IS NOT NULL\n              GROUP BY nrcemercante)\n          SELECT count()\n          FROM frequencies\n          WHERE frequency > 1",
"exception": null,
"duration": "0:02:05.249778"
},
{
"name": "cabotagem.db_maritimo[Duplicated].nrcemercante.duplicate_count.failing_sql",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Duplicated",
"column": "nrcemercante",
"sql": "\n          WITH frequencies AS (\n              SELECT nrcemercante\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND nrblconhecimento is not null AND deleted_at is null and dtoperacao >= DATE(current_date - interval '45 days') and dtoperacao < DATE(current_date + interval '15 days') AND lista_fcl is not null AND lista_fcl != '' AND categoriacarga = 'N' AND nrcemercante IS NOT NULL\n              GROUP BY nrcemercante\n              HAVING count() > 1)\n          SELECT main.\n          FROM public.db_maritimo main\n          JOIN frequencies ON main.nrcemercante = frequencies.nrcemercante\n",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Duplicated].nrcemercante.duplicate_count.passing_sql",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Duplicated",
"column": "nrcemercante",
"sql": "\n          WITH frequencies AS (\n              SELECT nrcemercante\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND nrblconhecimento is not null AND deleted_at is null and dtoperacao >= DATE(current_date - interval '45 days') and dtoperacao < DATE(current_date + interval '15 days') AND lista_fcl is not null AND lista_fcl != '' AND categoriacarga = 'N' AND nrcemercante IS NOT NULL\n              GROUP BY nrcemercante\n              HAVING count() <= 1)\n          SELECT main.\n          FROM public.db_maritimo main\n          JOIN frequencies ON main.nrcemercante = frequencies.nrcemercante\n",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Duplicated].nrcemercante.duplicate_count.failing_rows_sql_aggregated",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Duplicated",
"column": "nrcemercante",
"sql": "\n          WITH frequencies AS (\n              SELECT nrcemercante, COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND nrblconhecimento is not null AND deleted_at is null and dtoperacao >= DATE(current_date - interval '45 days') and dtoperacao < DATE(current_date + interval '15 days') AND lista_fcl is not null AND lista_fcl != '' AND categoriacarga = 'N' AND nrcemercante IS NOT NULL\n              GROUP BY nrcemercante)\n          SELECT \n          FROM frequencies\n          WHERE frequency > 1\n          ORDER BY frequency DESC\nLIMIT 100",
"exception": null,
"duration": null
},
{
"name": "cabotagem.db_maritimo[Duplicated].duplicate_count[nrcemercante].failed_rows.aggregated",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Duplicated",
"column": null,
"sql": "\n          WITH frequencies AS (\n              SELECT nrcemercante, COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND nrblconhecimento is not null AND deleted_at is null and dtoperacao >= DATE(current_date - interval '45 days') and dtoperacao < DATE(current_date + interval '15 days') AND lista_fcl is not null AND lista_fcl != '' AND categoriacarga = 'N' AND nrcemercante IS NOT NULL\n              GROUP BY nrcemercante)\n          SELECT \n          FROM frequencies\n          WHERE frequency > 1\n          ORDER BY frequency DESC\nLIMIT 100",
"exception": null,
"duration": "0:00:00.920886"
},
{
"name": "cabotagem.db_maritimo[Today].aggregation[0]",
"dataSource": "cabotagem",
"table": "db_maritimo",
"partition": "Today",
"column": null,
"sql": "SELECT \n  COUNT(),\n  COUNT(CASE WHEN dtoperacao IS NULL THEN 1 END) \nFROM public.db_maritimo\nWHERE categoriacarga = 'N' AND created_at > current_date AND deleted_at is NULL",
"exception": null,
"duration": "0:00:00.021923"
}
],
"automatedMonitoringChecks": [],
"profiling": [],
"metadata": [],
"logs": [
{
"level": "info",
"message": "Soda Core 3.0.41",
"timestamp": "2023-07-03T14:11:57+00:00",
"index": 1
},
{
"level": "debug",
"message": "Reading configuration file "/app/flows/Governanca/soda/files/cabotagem/config_psql.yml"",
"timestamp": "2023-07-03T14:11:57+00:00",
"index": 2
},
{
"level": "debug",
"message": "Reading SodaCL file "/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml"",
"timestamp": "2023-07-03T14:11:57+00:00",
"index": 3
},
{
"level": "debug",
"message": "Scan execution starts",
"timestamp": "2023-07-03T14:11:57+00:00",
"index": 4
},
{
"level": "debug",
"message": "Postgres connection properties: host="db-prd-sistema-database-ro.logcomex.io", port="5432", database="sistema", user="usr_soda", options="-c search_path=public", connection_timeout="None"",
"timestamp": "2023-07-03T14:12:00+00:00",
"index": 5
},
{
"level": "debug",
"message": "Query cabotagem.db_maritimo[Five].aggregation[0]:\nSELECT \n  COUNT(),\n  COUNT(CASE WHEN id IS NULL THEN 1 END),\n  COUNT(CASE WHEN nrcemercante IS NULL THEN 1 END),\n  COUNT(CASE WHEN nmempresanavegacao IS NULL THEN 1 END),\n  COUNT(CASE WHEN cdempresanavegacao IS NULL THEN 1 END),\n  COUNT(CASE WHEN tipo_carga IS NULL THEN 1 END),\n  COUNT(CASE WHEN NOT (tipo_carga IS NULL) AND NOT (tipo_carga IN ('CONTEINER','SOLTA','GRANEL LIQUIDO','SOLIDOS','VEICULOS','LIQUIDOS','GASOSOS','OUTROS')) THEN 1 END),\n  COUNT(CASE WHEN nmterminalportuariocarregamento IS NULL THEN 1 END),\n  COUNT(CASE WHEN nmportoorigem IS NULL THEN 1 END),\n  COUNT(CASE WHEN nmportodestino IS NULL THEN 1 END),\n  COUNT(CASE WHEN tipoconhecimento IS NULL THEN 1 END),\n  COUNT(CASE WHEN cdembarcacao IS NULL THEN 1 END),\n  COUNT(CASE WHEN nmembarcacao IS NULL THEN 1 END),\n  COUNT(CASE WHEN nrviagem IS NULL THEN 1 END),\n  COUNT(CASE WHEN cdconsignatario IS NULL THEN 1 END),\n  COUNT(CASE WHEN nmconsignatario IS NULL THEN 1 END),\n  COUNT(CASE WHEN cdshipper IS NULL THEN 1 END),\n  COUNT(CASE WHEN teus IS NULL THEN 1 END),\n  MIN(teus),\n  COUNT(CASE WHEN c20 IS NULL THEN 1 END),\n  MIN(c20),\n  COUNT(CASE WHEN c40 IS NULL THEN 1 END),\n  MIN(c40) \nFROM public.db_maritimo\nWHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL",
"timestamp": "2023-07-03T14:12:01+00:00",
"index": 6
},
{
"level": "debug",
"message": "Query cabotagem.db_maritimo[Five].tipo_carga.failed_rows[missing_count]:\nSELECT * FROM public.db_maritimo \n WHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL AND tipo_carga IS NULL \n LIMIT 100",
"timestamp": "2023-07-03T14:15:54+00:00",
"index": 7
},
{
"level": "info",
"message": "Sending failed row samples to Soda Cloud",
"timestamp": "2023-07-03T14:16:21+00:00",
"index": 8
},
{
"level": "debug",
"message": "Query cabotagem.db_maritimo[Five].nmterminalportuariocarregamento.failed_rows[missing_count]:\nSELECT * FROM public.db_maritimo \n WHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL AND nmterminalportuariocarregamento IS NULL \n LIMIT 100",
"timestamp": "2023-07-03T14:16:22+00:00",
"index": 9
},
{
"level": "info",
"message": "Sending failed row samples to Soda Cloud",
"timestamp": "2023-07-03T14:16:22+00:00",
"index": 10
},
{
"level": "debug",
"message": "Query cabotagem.db_maritimo[Five].cdshipper.failed_rows[missing_count]:\nSELECT * FROM public.db_maritimo \n WHERE dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL AND cdshipper IS NULL \n LIMIT 100",
"timestamp": "2023-07-03T14:16:24+00:00",
"index": 11
},
{
"level": "info",
"message": "Sending failed row samples to Soda Cloud",
"timestamp": "2023-07-03T14:16:25+00:00",
"index": 12
},
{
"level": "debug",
"message": "Query cabotagem.db_maritimo[Five].id.duplicate_count:\n\n          WITH frequencies AS (\n              SELECT COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE id IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY id)\n          SELECT count()\n          FROM frequencies\n          WHERE frequency > 1",
"timestamp": "2023-07-03T14:16:26+00:00",
"index": 13
},
{
"level": "debug",
"message": "Query cabotagem.db_maritimo[Five].duplicate_count[id].failed_rows.aggregated:\n\n          WITH frequencies AS (\n              SELECT id, COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE id IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY id)\n          SELECT \n          FROM frequencies\n          WHERE frequency > 1\n          ORDER BY frequency DESC\nLIMIT 100",
"timestamp": "2023-07-03T14:16:27+00:00",
"index": 14
},
{
"level": "debug",
"message": "Query cabotagem.db_maritimo[Five].nrcemercante.duplicate_count:\n\n          WITH frequencies AS (\n              SELECT COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY nrcemercante)\n          SELECT count()\n          FROM frequencies\n          WHERE frequency > 1",
"timestamp": "2023-07-03T14:16:27+00:00",
"index": 15
},
{
"level": "debug",
"message": "Query cabotagem.db_maritimo[Five].duplicate_count[nrcemercante].failed_rows.aggregated:\n\n          WITH frequencies AS (\n              SELECT nrcemercante, COUNT() AS fre
11:18:35 AM
soda_scan_execute-1578bf38-0
quency\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND dtoperacao BETWEEN (current_date - interval '6 days') AND (current_date - interval '1 days') AND categoriacarga = 'N' AND deleted_at IS NULL\n              GROUP BY nrcemercante)\n          SELECT \n          FROM frequencies\n          WHERE frequency > 1\n          ORDER BY frequency DESC\nLIMIT 100",
"timestamp": "2023-07-03T14:16:28+00:00",
"index": 16
},
{
"level": "debug",
"message": "Query cabotagem.db_maritimo[Duplicated].nrcemercante.duplicate_count:\n\n          WITH frequencies AS (\n              SELECT COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND nrblconhecimento is not null AND deleted_at is null and dtoperacao >= DATE(current_date - interval '45 days') and dtoperacao < DATE(current_date + interval '15 days') AND lista_fcl is not null AND lista_fcl != '' AND categoriacarga = 'N' AND nrcemercante IS NOT NULL\n              GROUP BY nrcemercante)\n          SELECT count()\n          FROM frequencies\n          WHERE frequency > 1",
"timestamp": "2023-07-03T14:16:28+00:00",
"index": 17
},
{
"level": "debug",
"message": "Query cabotagem.db_maritimo[Duplicated].duplicate_count[nrcemercante].failed_rows.aggregated:\n\n          WITH frequencies AS (\n              SELECT nrcemercante, COUNT() AS frequency\n              FROM public.db_maritimo\n              WHERE nrcemercante IS NOT NULL \nAND nrblconhecimento is not null AND deleted_at is null and dtoperacao >= DATE(current_date - interval '45 days') and dtoperacao < DATE(current_date + interval '15 days') AND lista_fcl is not null AND lista_fcl != '' AND categoriacarga = 'N' AND nrcemercante IS NOT NULL\n              GROUP BY nrcemercante)\n          SELECT \n          FROM frequencies\n          WHERE frequency > 1\n          ORDER BY frequency DESC\nLIMIT 100",
"timestamp": "2023-07-03T14:18:33+00:00",
"index": 18
},
{
"level": "debug",
"message": "Query cabotagem.db_maritimo[Today].aggregation[0]:\nSELECT \n  COUNT(),\n  COUNT(CASE WHEN dtoperacao IS NULL THEN 1 END) \nFROM public.db_maritimo\nWHERE categoriacarga = 'N' AND created_at > current_date AND deleted_at is NULL",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 19
},
{
"level": "info",
"message": "Scan summary:",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 20
},
{
"level": "debug",
"message": "11/11 queries OK",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 21
},
{
"level": "debug",
"message": "  cabotagem.db_maritimo[Five].aggregation[0] [OK] 0:03:52.761337",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 22
},
{
"level": "debug",
"message": "  cabotagem.db_maritimo[Five].tipo_carga.failed_rows[missing_count] [OK] 0:00:27.917024",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 23
},
{
"level": "debug",
"message": "  cabotagem.db_maritimo[Five].nmterminalportuariocarregamento.failed_rows[missing_count] [OK] 0:00:02.222375",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 24
},
{
"level": "debug",
"message": "  cabotagem.db_maritimo[Five].cdshipper.failed_rows[missing_count] [OK] 0:00:02.219778",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 25
},
{
"level": "debug",
"message": "  cabotagem.db_maritimo[Five].id.duplicate_count [OK] 0:00:00.512133",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 26
},
{
"level": "debug",
"message": "  cabotagem.db_maritimo[Five].duplicate_count[id].failed_rows.aggregated [OK] 0:00:00.512143",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 27
},
{
"level": "debug",
"message": "  cabotagem.db_maritimo[Five].nrcemercante.duplicate_count [OK] 0:00:00.409388",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 28
},
{
"level": "debug",
"message": "  cabotagem.db_maritimo[Five].duplicate_count[nrcemercante].failed_rows.aggregated [OK] 0:00:00.392143",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 29
},
{
"level": "debug",
"message": "  cabotagem.db_maritimo[Duplicated].nrcemercante.duplicate_count [OK] 0:02:05.249778",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 30
},
{
"level": "debug",
"message": "  cabotagem.db_maritimo[Duplicated].duplicate_count[nrcemercante].failed_rows.aggregated [OK] 0:00:00.920886",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 31
},
{
"level": "debug",
"message": "  cabotagem.db_maritimo[Today].aggregation[0] [OK] 0:00:00.021923",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 32
},
{
"level": "info",
"message": "27/28 checks PASSED: ",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 33
},
{
"level": "info",
"message": "    db_maritimo [Five] in cabotagem",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 34
},
{
"level": "info",
"message": "      Quantidade de embarques ultimos 5 dias [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 35
},
{
"level": "info",
"message": "        check_value: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 36
},
{
"level": "info",
"message": "      IDs duplicados ultimos 5 dias [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 37
},
{
"level": "info",
"message": "        check_value: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 38
},
{
"level": "info",
"message": "      ID nulo [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 39
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 40
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 41
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 42
},
{
"level": "info",
"message": "      CEs duplicados ultimos 5 dias [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 43
},
{
"level": "info",
"message": "        check_value: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 44
},
{
"level": "info",
"message": "      N\u00famero CE nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 45
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 46
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 47
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 48
},
{
"level": "info",
"message": "      Armador nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 49
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 50
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 51
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 52
},
{
"level": "info",
"message": "      Codigo Armador nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 53
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 54
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 55
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 56
},
{
"level": "info",
"message": "      Tipo Carga nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 57
},
{
"level": "info",
"message": "        check_value: 0.04",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 58
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 59
},
{
"level": "info",
"message": "        missing_count: 3",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 60
},
{
"level": "info",
"message": "      Tipo de Carga com valor diferente de 'CONTEINER','SOLTA','GRANEL LIQUIDO', 'SOLIDOS', 'VEICULOS', 'LIQUIDOS', 'GASOSOS' e'OUTROS' [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 61
},
{
"level": "info",
"message": "        check_value: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 62
},
{
"level": "info",
"message": "      Porto de Origem nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 63
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 64
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 65
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 66
},
{
"level": "info",
"message": "      Porto de Destino nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 67
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 68
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 69
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 70
},
{
"level": "info",
"message": "      CRITICAL - Tipo Conhecimento nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 71
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 72
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 73
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 74
},
{
"level": "info",
"message": "      Cod Navio nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 75
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 76
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 77
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 78
},
{
"level": "info",
"message": "      Navio nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 79
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 80
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 81
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 82
},
{
"level": "info",
"message": "      Nr Viagem nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 83
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 84
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 85
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 86
},
{
"level": "info",
"message": "      CNPJ Destinat\u00e1rio nulo [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 87
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 88
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 89
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 90
},
{
"level": "info",
"message": "      Nome Destinat\u00e1rio nulo [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 91
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 92
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 93
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 94
},
{
"level": "info",
"message": "      CNPJ Remetente nulo [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 95
},
{
"level": "info",
"message": "        check_value: 0.36",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 96
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 97
},
{
"level": "info",
"message": "        missing_count: 25",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 98
},
{
"level": "info",
"message": "      Teus nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 99
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 100
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 101
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 102
},
{
"level": "info",
"message": "      Teus deve ser maior ou igual a 0 [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 103
},
{
"level": "info",
"message": "        check_value: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 104
},
{
"level": "info",
"message": "      c20 nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 105
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 106
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 107
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 108
},
{
"level": "info",
"message": "      C20 deve ser maior ou igual a 0 [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 109
},
{
"level": "info",
"message": "        check_value: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 110
},
{
"level": "info",
"message": "      C40 nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 111
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 112
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 113
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 114
},
{
"level": "info",
"message": "      C40 deve ser maior ou igual a 0 [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 115
},
{
"level": "info",
"message": "        check_value: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 116
},
{
"level": "info",
"message": "    db_maritimo [Duplicated] in cabotagem",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 117
},
{
"level": "info",
"message": "      CEs duplicados [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 118
},
{
"level": "info",
"message": "        check_value: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 119
},
{
"level": "info",
"message": "    db_maritimo [Today] in cabotagem",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 120
},
{
"level": "info",
"message": "      Sem registros criados na data de hoje [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 121
},
{
"level": "info",
"message": "        check_value: 1",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 122
},
{
"level": "info",
"message": "      Data Opera\u00e7\u00e3o null [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [PASSED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 123
},
{
"level": "info",
"message": "        check_value: 0.0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 124
},
{
"level": "info",
"message": "        row_count: 1",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 125
},
{
"level": "info",
"message": "        missing_count: 0",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 126
},
{
"level": "info",
"message": "1/28 checks WARNED: ",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 127
},
{
"level": "info",
"message": "    db_maritimo [Five] in cabotagem",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 128
},
{
"level": "info",
"message": "      Terminal de Embarque nulos [/app/flows/Governanca/soda/files/cabotagem/db_maritimo.yml] [WARNED]",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 129
},
{
"level": "info",
"message": "        check_value: 4.28",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 130
},
{
"level": "info",
"message": "        row_count: 6933",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 131
},
{
"level": "info",
"message": "        missing_count: 297",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 132
},
{
"level": "info",
"message": "Only 1 warning. 0 failure. 0 errors. 27 pass.",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 133
},
{
"level": "info",
"message": "Sending results to Soda Cloud",
"timestamp": "2023-07-03T14:18:34+00:00",
"index": 134
}
],
"type": "sodaCoreInsertScanResults",
"token": "5PtZTJ_Hwwf_ZB8Bq-G38CX0M3NaWe_gZ5C8VTJMESmF9bwRk4gv4A"
}
11:18:35 AM
soda_scan_execute-1578bf38-0
[11:18:37] Soda Cloud Trace: 8039336511216826762
11:18:37 AM
soda_scan_execute-1578bf38-0
Saving scan results to 2023-07-03T14:11:57.361157+00:00--soda_scan_execute-1578bf38-0.json
11:18:37 AM
soda_scan_execute-1578bf38-0
Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 1214, in orchestrate_task_run
result = await task.fn(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect_soda_core/tasks.py", line 142, in soda_scan_execute
raise e
File "/usr/local/lib/python3.8/site-packages/prefect_soda_core/tasks.py", line 134, in soda_scan_execute
soda_logs = await shell_run_command.fn(
File "/usr/local/lib/python3.8/site-packages/prefect_shell/commands.py", line 103, in shell_run_command
raise RuntimeError(msg)
RuntimeError: Command failed with exit code 1:
Saving scan results to 2023-07-03T14:11:57.361157+00:00--soda_scan_execute-1578bf38-0.json
11:18:38 AM
soda_scan_execute-1578bf38-0
Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 596, in orchestrate_flow_run
result = await run_sync(flow_call)
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 117, in run_sync_in_interruptible_worker_thread
tg.start_soon(
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 597, in aexit
raise exceptions[0]
File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 97, in capture_worker_thread_and_result
result = __fn(*args, **kwargs)
File "flows/Governanca/soda/soda_cabotagem_flow.py", line 25, in soda_cabotagem_captura_flow
run_soda_scan('psql',
File "/tmp/tmpvrxsf6t_prefect/flows/lib_classes/libsoda.py", line 30, in run_soda_scan
return soda_scan_execute(
File "/usr/local/lib/python3.8/site-packages/prefect/tasks.py", line 295, in call
return enter_task_run_engine(
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 736, in enter_task_run_engine
return run_async_from_worker_thread(begin_run)
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 137, in run_async_from_worker_thread
return anyio.from_thread.run(call)
File "/usr/local/lib/python3.8/site-packages/anyio/from_thread.py", line 47, in run
return asynclib.run_async_from_thread(func, *args)
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 906, in run_async_from_thread
return f.result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
raise self._exception
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 874, in get_task_call_return_value
return await future._result()
File "/usr/local/lib/python3.8/site-packages/prefect/futures.py", line 237, in _result
return final_state.result(raise_on_failure=raise_on_failure)
File "/usr/local/lib/python3.8/site-packages/prefect/orion/schemas/states.py", line 145, in result
raise data
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 1214, in orchestrate_task_run
result = await task.fn(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect_soda_core/tasks.py", line 142, in soda_scan_execute
raise e
File "/usr/local/lib/python3.8/site-packages/prefect_soda_core/tasks.py", line 134, in soda_scan_execute
soda_logs = await shell_run_command.fn(
File "/usr/local/lib/python3.8/site-packages/prefect_shell/commands.py", line 103, in shell_run_command
raise RuntimeError(msg)
RuntimeError: Command failed with exit code 1:
Saving scan results to 2023-07-03T14:11:57.361157+00:00--soda_scan_execute-1578bf38-0.json`

@AlessandroLollo The code is handling only exception code 2 but warnings return with exit code 1

Another Problem, the soda_logs variable only exists in the return if return_scan_result_file_content is True, if it is false, an error occurs because the variable is created only inside the try

I get exit code 1 upon validation errors, not 2. Can "Command failed with exit code 1:" be also added to the if statement above?

Hey @ccueto36 👋
Would you be open to submit a PR for your use case?

authors: AlessandroLollo carlosnizolli ccueto36

repo: sodadata/prefect-soda-core