text:
filter_op and filter_ref not working correctly.

Having a real tough time with this one, unsure why it continues to index transfer events even though there is not matching account in the send_account_created. Any thoughts on why my filter_ref and filter_op is not working for send_account_created?

config.json
{
  "dashboard": {
    "root_password": "$DASHBOARD_ROOT_PASSWORD"
  },
  "pg_url": "$DATABASE_URL",
  "eth_sources": [
    {
      "name": "base_block_headers",
      "url": "$BASE_RPC_URL",
      "chain_id": "$BASE_CHAIN_ID",
      "batch_size": 100,
      "concurrency": 1
    },
    {
      "name": "base_logs",
      "poll_duration": "12s",
      "url": "$BASE_RPC_URL",
      "chain_id": "$BASE_CHAIN_ID",
      "batch_size": 1,
      "concurrency": 1
    }
  ],
  "integrations": [
    {
      "name": "send_account_created",
      "enabled": true,
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "log_addr",
          "column": "log_addr",
          "filter_op": "contains",
          "filter_arg": [
            "0x008c9561857b6555584d20aC55110335759Aa2c2",
            "0x1E6a7C7aA90Ea75d9aD9e02AEA71e3acbe05F8F5",
            "0xe4C5EF95e8cDA5DB09393a08E30645F883e187B8"
          ]
        }
      ],
      "event": {
        "type": "event",
        "anonymous": false,
        "inputs": [
          {
            "name": "account",
            "type": "address",
            "indexed": true,
            "column": "account"
          }
        ],
        "name": "AccountCreated"
      },
      "table": {
        "name": "send_account_created",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "user_op_hash",
            "type": "bytea"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "account",
            "type": "bytea"
          }
        ]
      },
      "sources": [
        {
          "name": "base_logs",
          "poll_duration": "12s",
          "start": 16363498
        }
      ]
    },
    {
      "name": "send_account_transfers",
      "enabled": true,
      "table": {
        "name": "send_account_transfers",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "f",
            "type": "bytea"
          },
          {
            "name": "t",
            "type": "bytea"
          },
          {
            "name": "v",
            "type": "numeric"
          }
        ]
      },
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "log_addr",
          "column": "log_addr"
        }
      ],
      "event": {
        "type": "event",
        "name": "Transfer",
        "anonymous": false,
        "inputs": [
          {
            "indexed": true,
            "name": "from",
            "type": "address",
            "column": "f",

          "filter_op": "contains",
          "filter_ref": {
            "integration": "send_account_created",
            "column": "account"
          }
          },
          {
            "indexed": true,
            "name": "to",
            "type": "address",
            "column": "t"
          },
          {
            "indexed": false,
            "name": "value",
            "type": "uint256",
            "column": "v"
          }
        ]
      },
      "sources": [
        {
          "name": "base_logs",
          "poll_duration": "12s",
          "start": 16363498
        }
      ]
    },
    {
      "name": "send_token_transfers",
      "enabled": true,
      "table": {
        "name": "send_token_transfers",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "f",
            "type": "bytea"
          },
          {
            "name": "t",
            "type": "bytea"
          },
          {
            "name": "v",
            "type": "numeric"
          }
        ]
      },
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "log_addr",
          "column": "log_addr",
          "filter_op": "contains",
          "filter_arg": [
            "0x3f14920c99BEB920Afa163031c4e47a3e03B3e4A",
            "0x7cEfbe54c37a35dCdaD29b86373ca8353a2F4680"
          ]
        }
      ],
      "event": {
        "type": "event",
        "name": "Transfer",
        "anonymous": false,
        "inputs": [
          {
            "indexed": true,
            "name": "from",
            "type": "address",
            "column": "f"
          },
          {
            "indexed": true,
            "name": "to",
            "type": "address",
            "column": "t"
          },
          {
            "indexed": false,
            "name": "value",
            "type": "uint256",
            "column": "v"
          }
        ]
      },
      "sources": [
        {
          "name": "base_logs",
          "poll_duration": "12s",
          "start": 16363498
        }
      ]
    },
    {
      "name": "send_revenues_safe_receives",
      "enabled": true,
      "table": {
        "name": "send_revenues_safe_receives",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "sender",
            "type": "bytea"
          },
          {
            "name": "v",
            "type": "numeric"
          }
        ]
      },
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "log_addr",
          "column": "log_addr",
          "filter_op": "contains",
          "filter_arg": [
            "0x269cD0a2afd1BAbdA7A74ab1dC853869a37aa4a7",
            "0x71fa02bb11e4b119bEDbeeD2f119F62048245301"
          ]
        }
      ],
      "event": {
        "type": "event",
        "name": "SafeReceived",
        "inputs": [
          {
            "name": "sender",
            "internalType": "address",
            "type": "address",
            "indexed": true,
            "column": "sender"
          },
          {
            "name": "value",
            "type": "uint256",
            "indexed": false,
            "column": "v"
          }
        ]
      },
      "sources": [
        {
          "name": "base_logs",
          "poll_duration": "12s",
          "start": 16363498
        }
      ]
    },
    {
      "name": "send_account_signing_key_added",
      "enabled": true,
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "log_addr",
          "column": "log_addr"
        }
      ],
      "event": {
        "type": "event",
        "anonymous": false,
        "inputs": [
          {
            "name": "account",
            "type": "address",
            "indexed": true,
            "column": "account",
                      "filter_op": "contains",
          "filter_ref": {
            "integration": "send_account_created",
            "column": "account"
          }
          },
          {
            "name": "keySlot",
            "type": "uint8",
            "indexed": false,
            "column": "key_slot"
          },
          {
            "name": "key",
            "type": "bytes32[2]",
            "indexed": false,
            "column": "key"
          }
        ],
        "name": "SigningKeyAdded"
      },
      "table": {
        "name": "send_account_signing_key_added",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "account",
            "type": "bytea"
          },
          {
            "name": "key_slot",
            "type": "smallint"
          },
          {
            "name": "key",
            "type": "bytea"
          }
        ]
      },
      "sources": [
        {
          "name": "base_logs",
          "poll_duration": "12s",
          "start": 16363498
        }
      ]
    },
    {
      "name": "send_account_signing_key_removed",
      "enabled": true,
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "log_addr",
          "column": "log_addr"
        }
      ],
      "event": {
        "type": "event",
        "anonymous": false,
        "inputs": [
          {
            "name": "account",
            "type": "address",
            "indexed": true,
            "column": "account",
                      "filter_op": "contains",
          "filter_ref": {
            "integration": "send_account_created",
            "column": "account"
          }
          },
          {
            "name": "keySlot",
            "type": "uint8",
            "indexed": false,
            "column": "key_slot"
          },
          {
            "name": "key",
            "type": "bytes32[2]",
            "indexed": false,
            "column": "key"
          }
        ],
        "name": "SigningKeyRemoved"
      },
      "table": {
        "name": "send_account_signing_key_removed",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "account",
            "type": "bytea"
          },
          {
            "name": "key_slot",
            "type": "smallint"
          },
          {
            "name": "key",
            "type": "bytea"
          }
        ]
      },
      "sources": [
        {
          "name": "base_logs",
          "poll_duration": "12s",
          "start": 16363498
        }
      ]
    },
    {
      "name": "send_account_receives",
      "enabled": true,
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_num",
          "column": "block_num"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "tx_idx",
          "column": "tx_idx"
        },
        {
          "name": "log_idx",
          "column": "log_idx"
        },
        {
          "name": "log_addr",
          "column": "log_addr"
        }
      ],
      "table": {
        "name": "send_account_receives",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "block_num",
            "type": "numeric"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "tx_idx",
            "type": "numeric"
          },
          {
            "name": "log_idx",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "sender",
            "type": "bytea"
          },
          {
            "name": "value",
            "type": "numeric"
          }
        ]
      },
      "event": {
        "type": "event",
        "anonymous": false,
        "inputs": [
          {
            "column": "sender",
            "name": "sender",
            "internalType": "address",
            "type": "address",
            "indexed": true,
          "filter_op": "contains",
          "filter_ref": {
            "integration": "send_account_created",
            "column": "account"
          }
          },
          {
            "column": "value",
            "name": "value",
            "internalType": "uint256",
            "type": "uint256",
            "indexed": false
          }
        ],
        "name": "Received"
      },
      "sources": [
        {
          "name": "base_logs",
          "poll_duration": "12s",
          "start": 16363498
        }
      ]
    }
  ]
}

what version are you running?
are there rows in the send_account_created table?

I was running shovel 1.6 and yes, send_account_created had some rows.
Downgrading to 1.1 for now since I don't remember this problem before but only now did we ship this to prod on 1.6.
Seems like latest main didn't have the issue either.

main (commit: 2096) is working for you?

I just downloaded this version vmain 2096. Seems to be working for the other integrations now. However, I can't get the biggest integration to work now, send_account_transfers.  Shovel just prints these lines, but never converges.
l=info  v=2096 msg=new-task ig=send_account_transfers src=base_logs
l=info  v=2096 msg=start at config ig=send_account_transfers src=base_logs num=16363498

ok. great. i will cut a 1.7 release soon. a lot of bug fixes have been put in lately.

jsut wanted to come back and say that the integrations with block filter ref work. but the integrations with event level filter refs do not. (send_account_transfers).

Can you provide the tx-hash/log-idx of the depending and dependent transactions? I will try to reproduce.

sure thing, we can use my send account on base. https://basescan.org/address/0x9446a0efe0852a327ae6732f5047509c5a784414
INSERT INTO public.send_account_created (block_num, tx_hash, tx_idx, log_idx) VALUES (16364175, E'\\xAC7BD97406631F35DD01FA3C3C5AFE721D1F0D94F619A61186D163AC11E0D7EB', 74, 266);

Seems like it did alot better today when i re-ran it locally, but still missed some transactions. Only indexed 16 txs vs the 38 currently on mainnet. https://basescan.org/tokentxns?a=0x9446a0efe0852a327ae6732f5047509c5a784414

export DASHBOARD_ROOT_PASSWORD=testing
export DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres
export BASE_NAME=base
export BASE_CHAIN_ID=8453
export BASE_BLOCK_START=16363498

config.json
{
  "dashboard": {
    "root_password": "testing"
  },
  "pg_url": "$DATABASE_URL",
  "eth_sources": [
    {
      "name": "base_block_headers",
      "url": "$BASE_RPC_URL",
      "chain_id": "$BASE_CHAIN_ID",
      "batch_size": 100,
      "concurrency": 1
    },
    {
      "name": "base_logs",
      "url": "$BASE_RPC_URL",
      "chain_id": "$BASE_CHAIN_ID",
      "batch_size": 2000,
      "concurrency": 2
    }
  ],
  "integrations": [
    {
      "name": "send_account_created",
      "enabled": true,
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "log_addr",
          "column": "log_addr",
          "filter_op": "contains",
          "filter_arg": [
            "0x008c9561857b6555584d20aC55110335759Aa2c2",
            "0x1E6a7C7aA90Ea75d9aD9e02AEA71e3acbe05F8F5",
            "0xe4C5EF95e8cDA5DB09393a08E30645F883e187B8"
          ]
        }
      ],
      "event": {
        "type": "event",
        "anonymous": false,
        "inputs": [
          {
            "name": "account",
            "type": "address",
            "indexed": true,
            "column": "account"
          }
        ],
        "name": "AccountCreated"
      },
      "table": {
        "name": "send_account_created",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "user_op_hash",
            "type": "bytea"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "account",
            "type": "bytea"
          }
        ]
      },
      "sources": [
        {
          "name": "base_logs",
          "start": "$BASE_BLOCK_START"
        }
      ]
    },
    {
      "name": "send_account_transfers",
      "enabled": true,
      "table": {
        "name": "send_account_transfers",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "f",
            "type": "bytea"
          },
          {
            "name": "t",
            "type": "bytea"
          },
          {
            "name": "v",
            "type": "numeric"
          }
        ]
      },
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "log_addr",
          "column": "log_addr"
        }
      ],
      "event": {
        "type": "event",
        "name": "Transfer",
        "anonymous": false,
        "inputs": [
          {
            "indexed": true,
            "name": "from",
            "type": "address",
            "column": "f",
            "filter_op": "contains",
            "filter_ref": {
              "integration": "send_account_created",
              "column": "account"
            }
          },
          {
            "indexed": true,
            "name": "to",
            "type": "address",
            "column": "t",
            "filter_op": "contains",
            "filter_ref": {
              "integration": "send_account_created",
              "column": "account"
            }
          },
          {
            "indexed": false,
            "name": "value",
            "type": "uint256",
            "column": "v"
          }
        ]
      },
      "sources": [
        {
          "name": "base_logs",
          "start": "$BASE_BLOCK_START"
        }
      ]
    },
    {
      "name": "send_token_transfers",
      "enabled": true,
      "table": {
        "name": "send_token_transfers",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "f",
            "type": "bytea"
          },
          {
            "name": "t",
            "type": "bytea"
          },
          {
            "name": "v",
            "type": "numeric"
          }
        ]
      },
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "log_addr",
          "column": "log_addr",
          "filter_op": "contains",
          "filter_arg": [
            "0x3f14920c99BEB920Afa163031c4e47a3e03B3e4A",
            "0x7cEfbe54c37a35dCdaD29b86373ca8353a2F4680"
          ]
        }
      ],
      "event": {
        "type": "event",
        "name": "Transfer",
        "anonymous": false,
        "inputs": [
          {
            "indexed": true,
            "name": "from",
            "type": "address",
            "column": "f"
          },
          {
            "indexed": true,
            "name": "to",
            "type": "address",
            "column": "t"
          },
          {
            "indexed": false,
            "name": "value",
            "type": "uint256",
            "column": "v"
          }
        ]
      },
      "sources": [
        {
          "name": "base_logs",
          "start": "$BASE_BLOCK_START"
        }
      ]
    },
    {
      "name": "send_revenues_safe_receives",
      "enabled": true,
      "table": {
        "name": "send_revenues_safe_receives",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "sender",
            "type": "bytea"
          },
          {
            "name": "v",
            "type": "numeric"
          }
        ]
      },
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "log_addr",
          "column": "log_addr",
          "filter_op": "contains",
          "filter_arg": [
            "0x269cD0a2afd1BAbdA7A74ab1dC853869a37aa4a7",
            "0x71fa02bb11e4b119bEDbeeD2f119F62048245301"
          ]
        }
      ],
      "event": {
        "type": "event",
        "name": "SafeReceived",
        "inputs": [
          {
            "name": "sender",
            "internalType": "address",
            "type": "address",
            "indexed": true,
            "column": "sender"
          },
          {
            "name": "value",
            "type": "uint256",
            "indexed": false,
            "column": "v"
          }
        ]
      },
      "sources": [
        {
          "name": "base_logs",
          "start": "$BASE_BLOCK_START"
        }
      ]
    },
    {
      "name": "send_account_signing_key_added",
      "enabled": true,
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "log_addr",
          "column": "log_addr",
          "filter_op": "contains",
          "filter_ref": {
            "integration": "send_account_created",
            "column": "account"
          }
        }
      ],
      "event": {
        "type": "event",
        "anonymous": false,
        "inputs": [
          {
            "name": "account",
            "type": "address",
            "indexed": true,
            "column": "account"
          },
          {
            "name": "keySlot",
            "type": "uint8",
            "indexed": false,
            "column": "key_slot"
          },
          {
            "name": "key",
            "type": "bytes32[2]",
            "indexed": false,
            "column": "key"
          }
        ],
        "name": "SigningKeyAdded"
      },
      "table": {
        "name": "send_account_signing_key_added",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "account",
            "type": "bytea"
          },
          {
            "name": "key_slot",
            "type": "smallint"
          },
          {
            "name": "key",
            "type": "bytea"
          }
        ]
      },
      "sources": [
        {
          "name": "base_logs",
          "start": "$BASE_BLOCK_START"
        }
      ]
    },
    {
      "name": "send_account_signing_key_removed",
      "enabled": true,
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "log_addr",
          "column": "log_addr",
          "filter_op": "contains",
          "filter_ref": {
            "integration": "send_account_created",
            "column": "account"
          }
        }
      ],
      "event": {
        "type": "event",
        "anonymous": false,
        "inputs": [
          {
            "name": "account",
            "type": "address",
            "indexed": true,
            "column": "account"
          },
          {
            "name": "keySlot",
            "type": "uint8",
            "indexed": false,
            "column": "key_slot"
          },
          {
            "name": "key",
            "type": "bytes32[2]",
            "indexed": false,
            "column": "key"
          }
        ],
        "name": "SigningKeyRemoved"
      },
      "table": {
        "name": "send_account_signing_key_removed",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "account",
            "type": "bytea"
          },
          {
            "name": "key_slot",
            "type": "smallint"
          },
          {
            "name": "key",
            "type": "bytea"
          }
        ]
      },
      "sources": [
        {
          "name": "base_logs",
          "start": "$BASE_BLOCK_START"
        }
      ]
    },
    {
      "name": "send_account_receives",
      "enabled": true,
      "block": [
        {
          "name": "chain_id",
          "column": "chain_id"
        },
        {
          "name": "block_num",
          "column": "block_num"
        },
        {
          "name": "block_time",
          "column": "block_time"
        },
        {
          "name": "tx_hash",
          "column": "tx_hash"
        },
        {
          "name": "tx_idx",
          "column": "tx_idx"
        },
        {
          "name": "log_idx",
          "column": "log_idx"
        },
        {
          "name": "log_addr",
          "column": "log_addr",
          "filter_op": "contains",
          "filter_ref": {
            "integration": "send_account_created",
            "column": "account"
          }
        }
      ],
      "table": {
        "name": "send_account_receives",
        "columns": [
          {
            "name": "chain_id",
            "type": "numeric"
          },
          {
            "name": "block_num",
            "type": "numeric"
          },
          {
            "name": "block_time",
            "type": "numeric"
          },
          {
            "name": "tx_hash",
            "type": "bytea"
          },
          {
            "name": "tx_idx",
            "type": "numeric"
          },
          {
            "name": "log_idx",
            "type": "numeric"
          },
          {
            "name": "log_addr",
            "type": "bytea"
          },
          {
            "name": "sender",
            "type": "bytea"
          },
          {
            "name": "value",
            "type": "numeric"
          }
        ]
      },
      "event": {
        "type": "event",
        "anonymous": false,
        "inputs": [
          {
            "column": "sender",
            "name": "sender",
            "internalType": "address",
            "type": "address",
            "indexed": true,
            "filter_op": "contains",
            "filter_ref": {
              "integration": "send_account_created",
              "column": "account"
            }
          },
          {
            "column": "value",
            "name": "value",
            "internalType": "uint256",
            "type": "uint256",
            "indexed": false
          }
        ],
        "name": "Received"
      },
      "sources": [
        {
          "name": "base_logs",
          "start": "$BASE_BLOCK_START"
        }
      ]
    }
  ]
}

Please provide the tx-hash/log-idx of the depending and dependent transactions in plain text format

sure thing here is my send account up to block height 16455562 on base.
send_account_created.csv
send_account_transfers.csv

I briefly looked through the CSV files and screenshots and it's not obvious to me which 2 transactions are required to reproduce the bug.
I am looking for data in this format:
Dependent transaction/log: 0xaaa / 1
Depending transaction/log: 0xbbb / 2
Ideally the transactions are close in age so that I don't have to process too many blocks to reproduce.
Looking through screenshots and downloading CSV files is not ideal. Bug reports for shovel (and other open source projects) should contain as little user-specific context as possible. The simpler you can make the bug report the quicker I can grok, reproduce, and fix.

Sorry for the half-ass bug report. I was putting out some fires last week/weekend due to send.app launching onchain. Spent some time isolating it. I tried to narrow it down for you. You can see my repro here, https://github.com/0xBigBoss/shovel-repro-262.
To make it easier, I was only looking at this one tx_hash for my send account. I've run the repro a few times and it seems to always miss the bottom depending transactions. I'm using Alchemy as a backend.

dependent transaction
send_account_created

block_num
16364175

tx_hash
0xAC7BD97406631F35DD01FA3C3C5AFE721D1F0D94F619A61186D163AC11E0D7EB

tx_idx
74

log_idx
266

depending transactions: send_account_transfers

block_num
tx_hash
tx_idx
log_idx

16364261
0x85DD9491CB735C45E6B4F05829B57B07DBA369F5ACBF70AA5B20D246EE95D24C
50
183

16364912
0xCCB288C7B20ADA19BC2325D4F789240AB53D783B89E98CB2F1E4E7442B4786B7
45
117

16365328
0x0B5CCE37B976E38FE68B3D4B461C9D35623DBC260B1622A011E1557CD6024EA1
4
18

16365328
0x0B5CCE37B976E38FE68B3D4B461C9D35623DBC260B1622A011E1557CD6024EA1
4
21

This is perfect! Thank you. I have what I need. I'll report back soon.

@0xBigBoss I believe #263 fixed the bug. Please confirm.

Looks good. Even checked it with the wider block range. Thx again for the superior support and making shovel awesome!

Looks good. Even checked it with the wider block range. Thx again for the superior support and making shovel awesome!

Great.
Thank YOU for hanging with me on the data request. Your last packet of debug data was A+ and helped me find the bug almost immediately.

repo: indexsupply/code


text:
Bug: Box16 shouldn't shutdown when SetSwapInterval fails.

SDL_GL_SetSwapInterval is a V-sync option, when it fails Box16 should not shut down.

Fixed by https://github.com/indigodarkwolf/box16/commit/f968da1217ee291482ff6722207b8925985a57c6

source: gharchive/issue

license: MIT

license_type: permissive

license_source: github-api


text:
Configure routers based on use case

This PR makes it possible to configure which routers get mounted based on the use case. The main reason for this at the moment is not to mount the template model API to the space weather deployment.
It does this by passing the domain set in the dockerfile through to startup.sh then setting it as an environment variable so it can get read by the WSGI script inside python. There's probably a  better way to do this - @kkaris can you advise?

I think is the way to do it considering $domain has to be specified as a build time argument.
The other option I was thinking of was passing in a runtime environment variable using -e DOMAIN=space when starting the container, but then you're left with having to make sure the build time value and the runtime value of domain are the same.

source: gharchive/pull-request

authors: cthoyt kkaris

repo: indralab/mira


text:
Deactivation endpoint

This PR adds an endpoint for deactivating templates using mira.metamodel.ops.deactivate_templates. The endpoint expects either or both of a list of transitions in the form of pairs of subject, outcome or a list of parameters which if found in a template's rate law will null that rate law.

Codecov Report

Merging #226 (0f0dff2) into main (b2f8f51) will increase coverage by 0.21%.
The diff coverage is 93.54%.

:exclamation: Current head 0f0dff2 differs from pull request most recent head a243ab9. Consider uploading reports for the commit a243ab9 to get more accurate results

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff             @@
##             main     #226      +/-   ##
==========================================
+ Coverage   77.75%   77.97%   +0.21%
==========================================
  Files          68       68
  Lines        5615     5716     +101
==========================================
+ Hits         4366     4457      +91
- Misses       1249     1259      +10

Impacted Files
Coverage Δ

mira/examples/sir.py
100.00% <ø> (ø)

mira/dkg/model.py
95.45% <91.07%> (-1.36%)
:arrow_down:

tests/test_model_api.py
96.86% <97.29%> (+0.04%)
:arrow_up:

... and 1 file with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter kkaris


text:
Build with Visual Studio based on .NET Core/5.0

Hello,
I'm wondering if this project can be built with Microsoft tools, preferably Visual Studio?
And, can I build and run it using .NET Core (or the latest, .NET 5)?
Thanks

Thanks for raising this issue we will take a look at it

Hello @KarloX2 ,

It is also possible to integrate this source file  into a Visual Studio project and build it there. The needed references (M2Mqtt, Newtonsoft) can be installed as NuGet packages:

This example was implemented and tested only with Mono under Docker, but it should also be possible using .NET

Thanks, @EvaKaul.
Can you please also tell which project type (in Visual Studio) you recommend?

Hi @KarloX2 ,
I choosed a C# Console Application in VS to build the source code.
But to test the whole ping pong application you have to use an Industrial Edge Device, where the Databus (mqtt broker) is running and config parameters are available.

Hi @EvaKaul.
Thanks, I am trying that right now. I have selected .NET5 as the target platform. After adding the two nuget packages you mentioned, the code compiles, but I get this warning:
warning NU1701: Package 'M2Mqtt 4.3.0' was restored using '.NETFramework,Version=v4.6.1, .NETFramework,Version=v4.6.2, .NETFramework,Version=v4.7, .NETFramework,Version=v4.7.1, .NETFramework,Version=v4.7.2, .NETFramework,Version=v4.8' instead of the project target framework 'net5.0'. This package may not be fully compatible with your project.
So this might be a problem when publishing the code to the IE Device. I'd like to publish it along with .NET5 (self-contained), but legacy .NET Framework (4.x etc.) won't be there, right? Are you sure the M2Mqtt package is the right one to use in this case?

reopened

Hi @KarloX2 ,
I think I didn't got you right.
Building ONLY the C# solution in VS - without any docker extension - worked for me with my default settings (.NET Framework 4.5.2) and with the latest versions of the nuget packages. I got no warnings.

But you want to implement the whole docker application with Visual Studio and .NET 5, right? I did not try this, but for sure this is possible using the right VS docker extensions.

authors: Boehmi-TB EvaKaul KarloX2

repo: industrial-edge/pingpong-csharp


text:
fix: use utc full year when encoding time values

Fix for #105

Landed and released in 5.0.1, thank you!

authors: codeslikejaggars indutny


text:
Api data

Allowing arbitrary datasets by users as children of now implemented abstract class 'data'.
Catalog dataset adult is now a child of abstract data.
Restructuring of data folder for better clarity with new data api.

Nice use of abstract classes! What would be for example another subclass implementation of Data other than DataCatalog?

Thanks :)
It could be every other arbitrary dataset we're not offering in our catalog

Thanks :)
It could be every other arbitrary dataset we're not offering in our catalog

Ah yeah great, so any implementation that doesn't automatically collects the data from the hosted datasets, since the DataCatalog calls load_data() which is only loading data from the repo. Love it!

Thanks :)
It could be every other arbitrary dataset we're not offering in our catalog

Ah yeah great, so any implementation that doesn't automatically collects the data from the hosted datasets, since the DataCatalog calls load_data() which is only loading data from the repo. Love it!

Exactly, an arbitrary dataset from a user can define its own load_data() for a local dataset, for example.
I try to keep this idea for ML - and CF- models.

authors: Philoso-Fish indyfree

repo: indyfree/CARLA


text:
chore(master): release 1.27.0

:robot: I have created a release beep boop
1.27.0 (2023-08-17)
Features

remove local argument (c0d67ee)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/indykite/indykite-sdk-python/releases/tag/v1.27.0 :sunflower:

repo: indykite/indykite-sdk-python


text:
Add a standards repository

In the future it will be very helpful for us to have a standards-repository

this will include how to handle authentication
how to handle server lists
how to handle media repositories and sign them
(more stuff)

I think it will be very good to have a dedicated standard repository, so that

proposals can be tracked after time
someone using Inexor knows how "we" do it
alternate standards can derive
someone "not doing it our way" can point out comparison to the standard (making it easy to explain both semantical and technical changes)

The wiki seems good enough to me?

I don't know. In the wiki

we cannot PR proposals to a standard or discuss on them
updating the proposal is slightly less obvious to all the people involved
proposals can't be locked

Maybe we should first begin with a user documentation a la https://readthedocs.org/

Will that mean that we convert our wiki into readthedocs or would you maintain a separate documentation?

I would suggest to create a user documentation for these audiences: players, mappers and game server owners. Everything related to development would be in the wiki.
This would have multiple effects:

Players, mappers and game server owners won't be confused about all the technical stuff
The wiki would be much cleaner
We can use a github repo as storage

authors: Croydon Fohlen aschaeffer

repo: inexorgame/inexor-core

license: Zlib


text:
[Bug]: After inserting a record into the database, the database in docker cannot start.

Is there an existing issue for the same bug?

[X] I have checked the existing issues.

Version or Commit ID
v0.3.0.dev5
Other environment information
No response
Actual behavior and How to reproduce it
Hello, I have a table which is created by using the following code.
rag_table_obj = db_obj.create_table(
            self.table_name,
            {
                "chunk_id": {"type": "varchar", "constraints": ["PRIMARY KEY"]},
                "kb_id": {"type": "varchar"},
                "user_no": {"type": "varchar"},
                "file_id": {"type": "varchar"},
                "file_name": {"type": "varchar"},
                "file_path": {"type": "varchar"},
                "timestamp": {"type": "varchar"},
                "content": {"type": "varchar"},
                "embedding": {"type": "vector, 768, float"},
            },
            ConflictType.Error
        )

After inserting a 768-dimensional vector data using QAnything from NetEase, infinity in docker cannot be started. The following error is reported in the background. What might be the reason?
libc++abi: terminating due to uncaught exception of type infinity::UnrecoverableException@infinity_exception: offset 238 is out of range 238@src/storage/column_vector/var_buffer.cpp:59

Expected behavior
No response
Additional information
No response

Can you reproduce the fault? If possible, would you please provide the code piece which lead to the fault?
BTW, the latest version is 0.3.0-dev6.

I write a script, but it can't reproduce the error. I attach it and you can try by yourself.
Uploading insert_many_columns_example.txt…

Thanks! I have deleted all the data files. Now it is a completely new environment and can be used temporarily. I don't know how to reproduce this problem. If I encounter it again next time, I will record it. Later, it will be upgraded to the latest version.

authors: JinHai-CN joynn

repo: infiniflow/infinity


text:
[Bug]: Process Begin At: Tue, 10 Sep 2024 15:32:20 GMT Process Duration: 325.09 s Progress Msg: Task dispatched...

Is there an existing issue for the same bug?

[X] I have checked the existing issues.

Branch name
document parsing
Commit ID
83803a7
Other environment information
docker 4.34.1
ragflow:0.10

Actual behavior
When I upload a document for document parsing, Progress Msg: Task dispatched..., and the document cannot be parsed normally
Expected behavior
Parsing documents normally
Steps to reproduce
Every time a file is uploaded for parsing, it appears

Additional information
none

Please check the status of task executor.

Hello, there are no tasks in the Task    ##@Executo.

Start the task executor, which has been shown in docker/entrypoint.sh

流程开始于:
Fri, 20 Sep 2024 06:17:37 GMT
过程持续时间:
848.41 s
进度消息:
Task has been received.
Start to parse.

Please upgrade to v0.14.1 which has much better monitoring on task executor.

authors: KevinHuSh cuntoushifu yuzhichang zxj197865

repo: infiniflow/ragflow


text:
[Bug]: Unable to find installation candidates for primp (0.8.3)

Is there an existing issue for the same bug?

[X] I have checked the existing issues.

RAGFlow workspace code commit ID
f8c9ec4
RAGFlow image version
v0.14.1-slim
Other environment information
OS: macOS 15.1 Apple silicon

Actual behavior
docker build --build-arg ARCH=arm64 --build-arg LIGHTEN=1 --no-cache -f Dockerfile -t infiniflow/ragflow:v0.14.1-slim .
108.2   RuntimeError
108.2
108.2   Unable to find installation candidates for primp (0.8.3)
108.2
108.2   at ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/installation/chooser.py:74 in choose_for
108.2        70│
108.2        71│             links.append(link)
108.2        72│
108.2        73│         if not links:
108.2     →  74│             raise RuntimeError(f"Unable to find installation candidates for {package}")
108.2        75│
108.2        76│         # Get the best link
108.2        77│         chosen = max(links, key=lambda link: self._sort_key(package, link))
108.2        78│
108.2
108.2 Cannot install primp.

Expected behavior
No response
Steps to reproduce
docker build --build-arg ARCH=arm64 --build-arg LIGHTEN=1 --no-cache -f Dockerfile -t infiniflow/ragflow:v0.14.1-slim .

Additional information
No response

same issue.

macOS 15.2 Apple Silicon

Fixed in v0.15.0

authors: jiyzhang southkorea2013 yuzhichang


text:
Update Spring documentation

There are at least 2 spots which need to be updated:

http://docs.spring.io/spring-boot/docs/1.4.1.RELEASE/reference/htmlsingle/#boot-features-caching-provider-infinispan
https://github.com/spring-projects/spring-boot/tree/master/spring-boot-samples/spring-boot-sample-cache

The Spring Caching documentation doesn't need updating. If we wanted to use Spring Boot Starters there, we would need to introduce 3 dependencies:

Infinispan Spring Boot Starter
JCache (since the demo is based on JCache...)
And Infinispan Spring Embedded artifact.

So we would make things worse actually. However I think it is worth to mention that we have Spring Boot Starters.

Submitted both pull requests. Let's wait for some update...

PR: https://github.com/spring-projects/spring-boot/pull/8592

The PR has been closed on spring side

authors: karesti slaskawi

repo: infinispan/infinispan-spring-boot


text:
ISPN-14983 Ensure ClusterRoleMapper and ClusterPermissionMapper are i…

https://issues.redhat.com/browse/ISPN-14983

Failures are unrelated

Thanks @tristantarrant

Sorry my review came too late :P

authors: fax4ever ryanemerson tristantarrant

repo: infinispan/infinispan


text:
ISPN-15291 Functional map with simple cache

https://issues.redhat.com/browse/ISPN-15291
The first commit handles the notification part. The second commit adds a WriteOnlyMap implementation with a simple cache and tests for all the simple variants.

merged thanks @jabolina !

authors: jabolina rigazilla


text:
[#13405] - nested queries improvements - proper handling of OR predicates

Fixes #13405
Implemented proper handling of OR predicates when using it with nested objects in an ickle query.
Problematic query example:
select t.name from org.infinispan.query.model.Team t
join t.firstTeam p
where (p.color ='red' AND p.number=7) or (p.color='blue' AND p.number=7)

After the fix this returns correct results instead of an error

Thanks @gybandi, great that you're expanding the support for this use case.

Thanks for the contribution. I have just one suggestion after that we can merge for me. I don't know if we want update also ParentJoinNestedRemoteTest (not mandatory for me - just an idea).

great idea, I will do that soon! thx!

I restarted the Jenkins CI

Merged, thanks for the contribution!


text:
[REVIEW ONLY] Ispn 3023 Re-implement BoundedConcurrentHashMap using CHMv8 designs

This includes the new BoundedEquivalentConcurrentHashMapV8 which includes support for LRU eviction policy.  Note that LIRS is not supported yet!
I broke this into 2 commits, the first is the existing EquivalentConcurrentHashMapV8 we have already so it would be easier to see what items were added to it.
Changes with the new map

The maxSize can be any long value greater than 0.  Note this used to be an int and would require a power of 2.
The maxSize is now utilized for the entire cache.  It no is no longer per segment.
Eviction is done outside of any locks, this include listener notification.  Prior the eviction itself was done while in the segment lock.
Refactored to be in commons package as it should have been before.

There are a couple concerns I hope to fix still

The custom linked deque implementation produces temporary slack when removing a value that is the head or tail.  This can can cause an issue with relinking the same instance.  This causes an additional allocation that we shouldn't need if fixed https://github.com/wburns/infinispan/commit/7044b29056552d25a51bbfddf62c94374f7505b2#diff-627723e665358c51b733265a3f0d16a1R430
I believe this also affected the previous BoundedConcurrentHashMap, but it seems possible that if a concurrent get and eviction occur at the same time it is possible to retrieve a value from the map and yet have it evicted at the same time.  I am not sure if this is a big issue since if this object was evicted more than likely it won't be accessed again for a long while (which caused it to be put to the head).
I need to evaluate if the previously contributed BatchWrapper is worth keeping.  I actually found an issue with it that could cause the entries to not be updated in the correct order if you had a concurrent read (that his buffer size) and write (this affected even the old BCHM).  This issue is fixed with adding some additional locking.  I am unsure how beneficial the queueing is with the additional contention, since without it the vast majority of writes will never have much if any contention since the tables will grow dynamically unlike the previous BCHM.

Still need some more thought

Refactor eviction policy to be able to be pluggable.  This way a user can provide an alternative eviction policy.  One simple example is LRU insertion order instead of access order as we have right now.  If we do it may be easier to split onEntryHit to have onEntryReadHit and onEntryWriteHit maybe so users can more easily tell when they are in the table lock or not.

Observations

Having more than maxSize: The previous implementation could get up to maxSize + # of segments.  The new one can technically get up to maxSize + # of threads (where threads are doing new puts)

Great stuff @wburns!! The original design of BoundedCHM is by @vblagoje, so maybe he can provide more background info on the batch wrapper and other related topics.
It's good news too that the maxSize is more precise than in previous version.
I'm not sure however whether it'd be good to make the eviction policy pluggable. Standard JSR-107 brings in something called ExpiryPolicy, which works more with life of entries, rather than eviction. If we are going to provide an eviction policy, it might make sense for it to look similar to that. However, I've not seen many requests to tweak eviction itself, so I'd probably leave any plugging out of the scope of this work.

Thanks @galderz, yes I agree the pluggable policy is probably out of scope for such changes.

Closed for now

authors: galderz wburns


text:
Documentation updates for server

Scripting and Domain mode
Please apply to both 8.0.x and master

I have split all new paragraphs to single lines and removed the stray w

Integrated, thanks Tristan!

authors: danberindei tristantarrant


text:
ISPN-6538 Little typo in Clustering modes documentation section

https://issues.jboss.org/browse/ISPN-6538
Thanks,
Andrea

Integrated in master. Thanks @oscerd!

authors: anistor oscerd


text:
ISPN-7550 Remove TriangleAckInterceptor

https://issues.jboss.org/browse/ISPN-7550

@pruivo Could the NonTxPrimaryOwnerLeavingTest.testPrimaryOwnerLeavingDuringPut failure be related to the changes here?
http://ci.infinispan.org/viewLog.html?buildId=51523&buildTypeId=bt9

@danberindei I'm going to loop the test and see what is going on :)

@danberindei I was unable to get a failure locally. I'll investigate if it pops-up again in CI

authors: danberindei pruivo


text:
ISPN-11989 Fix ThreadLocalLeakTest

https://issues.redhat.com/browse/ISPN-11989

Merged, thanks

authors: ryanemerson tristantarrant


text:
IPROTO-363 Fix README.md updater

https://issues.redhat.com/browse/IPROTO-363

thanks @tristantarrant

authors: rigazilla tristantarrant

repo: infinispan/protostream

license_source: bigquery


text:
🛑 Homepage is down

In 7c59306, Homepage (https://infinite8.co) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Homepage is back up in b6e9e14.


text:
PMNavigationActivity fragment changing signals

It'd be great to support something on_appear & on_return when tapping back inside a navigation activity.
I'll put some something quick & dirty in place up at the project level for now.

try on_start and let me know if that works.

It does.  Thanks, I'll be able to use that.  Interestingly, on_load fires too when back tracking.  Not sure if that's expected or not.

Actually, on_return should work.
https://github.com/infinitered/bluepotion/blob/8d606df154daff881086a7008007caf2212ff44c/lib/project/pro_motion/fragments/pm_screen_module.rb#L190
And on_appear is an alias for on_start.

authors: GantMan jamonholmgren skellock

repo: infinitered/bluepotion

license: mit


text:
How to get metrics by Rancher environment

So I've deployed the Prometheus stack and I've updated the pieces as I see you've made some changes to host node discovery and such. Things are working pretty solid, but I'm failing to see how I could separate nodes based on Rancher environment.
Not sure this is the best place for this comment, but the HostMetrics and ContainerMetrics show up with instance=<ip>:<port> and no other metadata AFAICT.
I'm pretty new to Prometheus itself, so maybe there's just some basic item I'm missing.

I'm literally testing that stuff in my reference deployment now!
Are you wanting to view nodes registered to the rancher server, but running in another 'rancher-environment'?

Yeah. Exactly. We have multiple Rancher environments (independent clusters) that are all controlled by 1 Rancher Server.
We use them to break apart both various teams and "environments" in the traditional sense (dev, qa, production, etc).
So it would be great to be able to see cluster performance by Rancher environment - host and container stats.

Also, if there's something I can help with, let me know. I can dig into and test some things.

Thanks @johnrengelman It's something that's on my list to add, but won't work 'out the box'.
The DNS auto-discovery i suspect will only work within an environment (i'll check that).
I previously got hostnames using confd but that got the hosts from metadata services, the trouble with this is that the API keys used in that scenario limit you to a single environment. The cleanest way would be through DNS discovery, i'll have a dig to see if there is a way it can be done.

You can create global API keys in Rancher. You have to do it via the API though (not available in the UI)(https://forums.rancher.com/t/api-key-for-all-environments/279)
I've done this for my use locally (I have single API key that I can use to run rancher-compose against multiple environments).
Anyway...I'm trying some changes to rancher-prometheus exporter now, but I'm guessing we'd need to wrap the node-exporter with something that would allow it to push a label for the metrics that corresponds to it's Rancher Environment.

Today I learned...
If that's the case, i have a confd template that will run through the hosts listed out by the rancher-prometheus-exporter. The exporter gets it's keys through the labels, i'll have a look to see if the global value can be used with labels.
Regarding node-exporter, i'm sort of wondering if we could get those stats out of cadvisor and do away with node exporter in the long term. In the cadvisor GUI, it exposes things like CPU cores etc.

The other thing I'm having trouble grok'ing is how to handle customer Host labels from Rancher. So you can assign free-form labels to hosts in Rancher and use them for scheduling. Might be nice to somehow use that too.

In terms of how to get those labels into Prometheus for querying?

Yeah.

Tricky one, i could see how it can be done if we pulled all that data from the API. Would need a bit of attention within the api-exporter but in theory that info is all there and could be used to populate the config. That being said, i suspect we'd lose DNS auto-discovery. hmm

Actually, thinking about this more, the rancher-prometheus integration doesn't need to scan all environments, you could run a single copy in each environment that points to the same prometheus server. It's just that the data that it kicks out needs to include the Rancher environment name as part of it.

@johnrengelman Will look at getting that added in soon.

+1 for this. I've ended up writing a small script that pulls host information from the Rancher API (using admin keys that can see all environments) and feeding it into Prometheus as using file_sd_configs. Not as pretty as DNS discovery, but seems to work OK.

@ampedandwired That's not actually a bad way of doing it. There is the exporter (https://github.com/infinityworksltd/prometheus-rancher-exporter) that gives some of that info.
If your sticking to the environment model in Rancher being what it is, Federation in Prometheus is probably the most obvious way. Though it's not easy to auto-discover additional environments.
Or again of course, you could utilise something like Consul...

@ampedandwired Please can you share the script and the path where I should run it?
We want to use prometheus but just one for all our environments in Rancher.

@marlinhares Sorry, not easily able to share the script because it's mixed in with a bunch of internal stuff that I can't release. Maybe someday I'll get a chance to separate it out. The basic idea is (pseudocode only):
prometheus_config = []
for rancher_env in rancher_api.get_environments():
  for host in rancher_api.get_hosts(rancher_env):
    ip_address = host["ipAddresses"][0]["address"]
    port = 9104  # CAdvisor port
    target = "#{ip_address}:#{port}"

    prometheus_config.append({
      "targets": [target],
      "labels": {
        "rancher_environment_name": rancher_env["name"]
      }
    })

if prometheus_config != current_prometheus_config():
  with open("/etc/rancher-prometheus/hosts.json") as f:
    f.write(json.dumps(prometheus_config))

This results in a list of entries like this in /etc/rancher-prometheus/hosts.json:
{
  "targets": [
    "10.0.12.6:9104"
  ],
  "labels": {
    "rancher_environment_name": "production"
  }
}

Obviously /etc/rancher-prometheus/ needs to be mounted as a volume on your prometheus container.
And then you can tell prometheus to watch the directory that the file is being written to:
  - job_name: 'rancher'
    file_sd_configs:
      - files: ['/etc/rancher-prometheus/*.json']

Finally I use "global" rancher keys (mentioned in a previous comment above) so I only need one instance of this container for the whole Rancher installation.
Anyway, hope this helps.

authors: Rucknar ampedandwired johnrengelman marlinhares

repo: infinityworksltd/prom-conf


text:
Fix typo in TWiF 41

This was a change I made direct to stable while trying to fix it. This will address the obvious branch conflict.

This was cherry picked into https://github.com/infinyon/fluvio-website/pull/379

repo: infinyon/fluvio-website


text:
[InfluxDB] Clustering Problem - 0.10

Hello,
I tried to install InfluxDB on 3 nodes and configured a cluster. I followed this documentation : https://docs.influxdata.com/influxdb/v0.10/guides/clustering/
influxdb-itx5.gix <-> influxdb-itx2.gix <-> influxdb-th2.gix
==== influxdb-itx5.gix ====
vim /etc/influxdb/influxdb.conf
[meta]
  http-bind-address = "influxdb-itx5.gix:8091"
  bind-address = "influxdb-itx5.gix:8088"

==== influxdb-itx2.gix ====
vim /etc/influxdb/influxdb.conf
[meta]
  http-bind-address = "influxdb-itx2.gix:8091"
  bind-address = "influxdb-itx2.gix:8088"

vim /etc/default/influxdb
  INFLUXD_OPTS="-join influxdb-itx5.gix:8091"

==== influxdb-th2.gix ====
vim /etc/influxdb/influxdb.conf
[meta]
  http-bind-address = "influxdb-th2.gix:8091"
  bind-address = "influxdb-th2.gix:8088"

vim /etc/default/influxdb
  INFLUXD_OPTS="-join influxdb-itx5.gix:8091"

==== My procedure ===
influxdb-itx5.gix => service influxdb start
influxdb-th2.gix => service influxdb start
influxdb-itx2.gix => service influxdb start
My issue is : I cannot have a cluster on data node:
root at influxdb-itx5 ➜  ~  influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.10.0
InfluxDB shell 0.10.0
> show servers
name: data_nodes
----------------
id	http_addr	tcp_addr
1	localhost:8086	influxdb-itx5.gix:8088

name: meta_nodes
----------------
id	http_addr		tcp_addr
1	influxdb-itx5.gix:8091	influxdb-itx5.gix:8088
2	influxdb-th2.gix:8091	influxdb-th2.gix:8088
3	influxdb-itx2.gix:8091	influxdb-itx2.gix:8088

Is there someone who can help me with the configuration
Alexandre Corso
acorso@franceix.net

@corsoalex-pro Per our conversation on the mailing list, the issue was due to the http.bind-address not being bound to the same hostname as the meta parameters. This was actually fixed with #215 a few days ago.

authors: corsoalex-pro rossmcdonald


text:
Link to authentication is broken

This link https://docs.influxdata.com/influxdb/v1.0/administration/authentication_and_authorization/ is broken
URL for relevant page?
That link is in https://docs.influxdata.com/influxdb/v1.0/guides/querying_data/
What products and version are you using?
Where did you look before opening the issue?

The page moved to the query language section: https://docs.influxdata.com/influxdb/v1.0/query_language/authentication_and_authorization/
The https://docs.influxdata.com/influxdb/v1.0/administration/ page also still lists auth/auth as being a child page.

All the links to CREATE, GRANT, REVOKE, SHOW at https://docs.influxdata.com/influxdb/v1.0/query_language/authentication_and_authorization/ have also been broken.
Fix please?

Fixed by https://github.com/influxdata/docs.influxdata.com/pull/719. Thank you for pointing this out, @jameswnl and @dandv.

authors: beckettsean dandv jameswnl rkuchan


text:
Investigate spiky TSI query planning latencies

I noticed recently that the latencies when planning queries using the TSI index can be quite spiky. My suspicion is that this is down to Series File lock contention, or possibly TSI Log File compaction contention.
We should investigate this and do what we can to reduce it.

Here is an example of latencies from a query on a dataset with ~2M series. I know that the vast majority of the latency is eaten up planning.
As you can see there is quite a long tail, with some queries taking upwards of 350ms to respond.

repo: influxdata/influxdb


text:
Merge duplicate points

Feature Request
Proposal: Enable merging of duplicate points
Current behavior: If a point is submitted with an identical measurement, tag set, and timestamp it will silently overwrite the previous point.
https://docs.influxdata.com/influxdb/v0.9/troubleshooting/frequently_encountered_issues/#writing-duplicate-points
Desired behavior: A new endpoint, e.g. /upsert, works exactly as /write in all respects except behaviour with duplicate points.  This new endpoint sums the numeric measurements when the tag set and timestamp are identical to an existing measurement.
Use case: The current behaviour, while intentional, is such a bugbear that there's even a section in the FAQ for it!  When the measurements being inserted into influx are already aggregations from another system its very common for points to collide, and people following the advice in the FAQ are busy filling up their database with distinct events (adding random nanoseconds or tags and hoping those random numbers don't collide) because they aren't able to update-in-place.

The behavior of newer values overwriting older ones within a series is deeply ingrained into the storage engine.  An /upsert API would probably require querying every series being written for identical series which would be very expensive.  This is very unlikely to be implemented in the database itself.
If you have a window of time where duplicate data could arrive, you may be able to use telegraf aggregators to receive the writes, aggregate them and then send to influxdb.

I don't know Go and I don't know the Influx code-base, being just a user, but my guess is that its a ordered map where the key is the tag set and the value is a hash map of field name values.  When you receive the new measurements you have to get the tag set to find the map of fields.  When you have found the place to insert the new value, you can easily check if the value is already there.  In Java, for example, Map.put returns the previous value.

Second this, we have the same problem of trying to add random nano-seconds and hoping it doesn't collide. Saddly we have been finding it still collides frequently leaving the only solution to have a UUID-4 tag to ensure insert uniqueness at a given timestamp.

authors: josephxsxn jwilder williame


text:
feat: Add non-unique u16 Id to ColumnDefinition

This commit adds the column_id field to ColumnDefinition so that the output for a Catalog will contain the id of that column. This is non unique, whereas TableIds and DbIds will be unique. The column_id corresponds to it's index in the schema.
Closes #25386

I'm having a test fail that's not my fault and can't get it to not flake:
Serve command failed: Failed to bind address
I wonder if some address conflict is happening. I think for some tests we bind the address but then drop it and free it up for others to take. Possibly a race condition.

Yeah one of the troubles with our integration tests is that we are launching the service from the CLI, but checking for an open port before starting. So yeah, there may be a race condition between selecting the available port, dropping the port, then starting the CLI with that port- or it is leaving a gap there where it could conflict with something else on the CI server.
Since we are launching with the CLI, we could pass in the --http-bind 0.0.0.0:0 arg directly, which would have it pick a random port and use it to start the server directly, but I'm not sure how we would then report the selected port back to the test harness.

@pauldix and @hiltontj the changes in https://github.com/influxdata/influxdb/pull/25388/commits/d2029942b7c647284dcea19d034510a91fc9d697 should hopefully address your concerns.

authors: hiltontj mgattozzi


text:
Implement cumulative_sum() function

The cumulative_sum() function can be used to sum each new point and
output the current total. For the following points:
cpu value=2 0
cpu value=4 10
cpu value=6 20

This would output the following points:
> SELECT cumulative_sum(value) FROM cpu
time    value
----    -----
0       2
10      6
20      12

As can be seen, each new point adds to the sum of the previous point and
outputs the value with the same timestamp.
The function can also be used with an aggregate like derivative().
> SELECT cumulative_sum(mean(value) FROM cpu WHERE time >= now() - 10m GROUP BY time(1m)

@jwilder @nathanielc

Can you update the description with more details about the use of this function?

Updated the description.

@nathanielc I updated this with your recommendation. Can you take another look?

@jsternberg: The last two lines of the OP mention a derivative function and lack a closed paren, respectively.

Well then I'm just going to look stupid for all eternity 😢 It's merged now. No going back!

Sorry if my comment came across the wrong way. Will delete.

authors: dandv jsternberg jwilder


text:
Bash auto completion: Filter list by task state

Changes

Added filtering to bash auto completion of kapacitor enable to only list disabled tasks
Added filtering to bash auto #completion of kapacitor disable to only list enabled tasks

Required for all non-trivial PRs

[x] Rebased/mergable
[ ] Tests pass
[ ] CHANGELOG.md updated
[x] Sign CLA (if not already signed)

@nathanielc
Do you update CHANGELOG.md when you merge this PR or shall I update CHANGELOG.md in this PR?

@bbczeuz I took care of the CHANGELOG. Thanks for the PR!

authors: bbczeuz nathanielc

repo: influxdata/kapacitor


text:
mappings cannot target metadata

consider the following ResourceSync, taken from our current system:
apiVersion: sinker.influxdata.io/v1alpha1
kind: ResourceSync
metadata:
  name: cluster-0001-kustomization-status
  namespace: customer-0001
spec:
  mappings:
  - fromFieldPath: status
    toFieldPath: spec
  - fromFieldPath: metadata.labels
    toFieldPath: metadata.labels
  source:
    cluster:
      kubeConfig:
        secretRef:
          key: value
          name: cluster-0001-kubeconfig
      namespace: iox
    resourceRef:
      apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
      kind: Kustomization
      name: iox-manifests
  target:
    resourceRef:
      apiVersion: tubernetes.influxdata.io/v1beta2
      kind: FluxcdKustomizationStatus
      name: iox-manifests

note that it has a mapping to copy the labels from source to target.
this doesn't work because on this line it is root.data that is passed into the set_field_path function: https://github.com/influxdata/sinker/blob/main/src/controller.rs#L168
root.data contains { spec: {...}, status: {...} }, the metadata is kept outside of the data object.
the result is that there results in two .metadata objects! somewhere one gets dropped.
i have a failing test for this, will work on a fix.

okay so i fixed the bug and checked what it did to managedFields:
    managedFields:
    - apiVersion: tubernetes.influxdata.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels: {}
        f:spec:
          f:conditions: {}
          f:inventory:
            f:entries: {}
          f:lastAppliedRevision: {}
          f:lastAttemptedRevision: {}
          f:observedGeneration: {}
      manager: sinker.influxdata.io
      operation: Apply
      time: "2023-04-12T13:00:53Z"

not very helpful, because this ResourceSync is modifying the spec and the labels. i would have to test it on a mapping that modifies only labels to be sure, but it looks like it will own the whole labels object.
here is another where it is copying to spec and to labels, but labels actually existed:
    managedFields:
    - apiVersion: tubernetes.influxdata.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels:
            f:app: {}
            f:idpe.influxdata.io/component: {}
            f:influxdata.io/owner: {}
            f:kustomize.toolkit.fluxcd.io/name: {}
            f:kustomize.toolkit.fluxcd.io/namespace: {}
        f:spec:
          f:loadBalancer:
            f:ingress: {}

looks like it will own only the labels it sets?

Nice, that's promising. Thanks for checking.

authors: cannonpalms lukebond

repo: influxdata/sinker


text:
[feature request] Add fill() method like Graphite's KeepLastValue

Graphite offers a method to deal with missing data called KeepLastValue. It "takes one metric [...] and continues the line with the last received value when gaps (‘None’ values) appear in your data..."
Could Influx's fill() be extended to offer similar functionality?

Did you read the manual? https://influxdb.com/docs/v0.9/query_language/data_exploration.html
Isn't fill(previous) what you want?

authors: lexh otoolep

repo: influxdb/influxdb


text:
Fix support for running tlc benchmarks

At some point we lost support for running TLC. We should restore it by fixing the TODO in the tool_cmd function in https://github.com/informalsystems/apalache-tests/tree/master/scripts/mk-run.py

@shonfeder Is TLC support still of interest?  Also, how reliable are the results obtained by running the benchmarks on shared Github action instances?

Hi!

Is TLC support still of interest

Afaik, we have no pressing need for this, but we want to track it for the long term. IMO, I'd not consider integrating it into the current framework, but would instead subordinate this functionality to the longerterm plan to rewrite the benchmarking framework.

how reliable are the results obtained by running the benchmarks on shared Github action instances?

Probably not very. I don't think we'd use these results for scientific publication, but they are useful as a sanity check lest we introduced a change that caused a huge regression in performance.

FWIW: The now-defunct performance/scalability test infrastructure for TLC used sponsored, bare-metal instances from https://metal.equinix.com

Thanks!

FYI: TLC now has this with https://github.com/tlaplus/tlaplus/actions/workflows/perf.yml, which runs a number of benchmarks upon request on a bare-metal instance hooked into Github actions via a self-hosted runner.

authors: lemmy shonfeder

repo: informalsystems/apalache-tests


text:
[FEATURE] Extending operator declarations with type annotations

This topic pops up from time to time. Sometimes, we have to annotate operators, e.g., with types or the sets of assigned variables, #226 and #261.
It looks to me that annotating expressions is a too fine-grained approach. On the other hand, we can annotate operator definitions. I see two options:

Add annotations directly in TlaOperDecl,
Similar to TlaEx, introduce a uid in TlaOperDecl and store annotations as maps from uid to whatever.

I think the second approach is more flexible and extendable, but we will have to extend TransformationListener.
@Kukovec what is your opinion?

Duplicate of #262

repo: informalsystems/apalache


text:
change ldflags and additionalLdFlags type to list

ldflags is a list now in nixpkgs-unstable: https://github.com/NixOS/nixpkgs/blob/891a6348766ff17e4666b43aa94a28390467340e/pkgs/build-support/go/module.nix#L42

@onsails merged, thanks again for the contribution!

authors: JonathanLorimer onsails


text:
Add support for multiple connection hops in create channel command

Part of: #3940
Summary
create channel

[x] Add a new parameter --connection-hops to create channel, which should receive the identifiers of the intermediate connections between two chains. This should work similarly to the --connection-hops parameter in tx chan-open-init.
[x] Add a new method  run_multihop_reusing_connection() to open a multi-hop channel when arguments were passed via --connection-hops.
[x] Add a new associated function Channel::new_multihop() to prepare an instance of Channel for multi-hop channel handshakes.

Tests

[x] Test create channel manually for single-hop channels.
[x] Test create channel manually for channels with 2 connection hops.
[x] Test create channel manually for channels with 2+ connection hops.
[x] Write unit tests for the new --connection-hops parameter.

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate milestone (priority) applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

Closed on #4067.

repo: informalsystems/hermes


text:
p2p: test SecretConnection handshake

Write a test for Secret connection, which covers the whole handshake (exchanging ephemeral keys and so on) #happypath. We probably also want to cover additional scenarios like when a peer is not responding or responding with invalid data (do we want to try fuzzing here?) at each step of the handshake.
Acceptance Criteria

[x] happy path test
[ ] test where the remote peer is not responding
[ ] test where the remote peer provides invalid data

    #[test]
    fn test_evil_peer_does_not_share_eph_key() {
        let (_pipe1, pipe2) = pipe::bipipe_buffered();

        let thread1 = thread::spawn(|| {
            let mut csprng = OsRng {};
            let privkey1: ed25519::Keypair = ed25519::Keypair::generate(&mut csprng);
            let conn1 = SecretConnection::new(pipe2, &privkey1, Version::V0_34);
            assert_eq!(conn1.is_ok(), true);
        });

        thread1.join().expect("thread1 has panicked");
    }

test where peer is not responding fails at the moment because we're using blocking reads (read_exact). we probably need to use mio or threads (which will require static lifetime) to read with a timeout.

    #[test]
    fn test_evil_peer_does_not_share_eph_key() {
        let (_pipe1, pipe2) = pipe::bipipe_buffered();

        let thread1 = thread::spawn(|| {
            let mut csprng = OsRng {};
            let privkey1: ed25519::Keypair = ed25519::Keypair::generate(&mut csprng);
            let conn1 = SecretConnection::new(pipe2, &privkey1, Version::V0_34);
            assert_eq!(conn1.is_ok(), true);
        });

        thread1.join().expect("thread1 has panicked");
    }

test where peer is not responding fails at the moment because we're using blocking reads (read_exact). we probably need to use mio or threads (which will require static lifetime) to read with a timeout.

test where peer is not responding fails at the moment because we're using blocking reads (read_exact). we probably need to use mio or threads (which will require static lifetime) to read with a timeout.

This could be done in a follow-up after the tests are finalised, as first improvement based on it.

test where peer is not responding fails at the moment because we're using blocking reads (read_exact). we probably need to use mio or threads (which will require static lifetime) to read with a timeout.

This could be done in a follow-up after the tests are finalised, as first improvement based on it.

authors: melekes xla

repo: informalsystems/tendermint-rs


text:
adjusting mainContent width for wide screens

Things are spacing out well for screens ~1250 pixels or less - but for wider screens/higher resolution windows, I want to adjust the column classes for greater widths. Basically, on wider screens, the main content won't be as wide as it ideally should be.
image 'col-lg-8' shows the default size for the main content using bootstrap class "col-lg-8" on a wide screen - the browser window width is 2330px
image 'col-lg-12' shows what it would look like in the same window if using the class "col-lg-12".
There are also a couple of other breakpoints in between.
Should be able to do this with some jquery

re-adding the images referred to above.
'col-lg-8' example

'col-lg-12' example

Note that we've fixed this in FaceBase (in the jekyll /data/search/index.html.in file) with additional bootstrap xl-col class and javascript and it's looking good. Should test a vanilla chaise deployment and see if these changes would be useful there.

@crisaless can we move this fix into the general chaise repo? has it already?

just fixed this - I'll assign to you to review. Just test a non-facebase chaise deployment on a widescreen monitor (ie, 27" with 2560 x 1440 resolution). If you don't have one at the ready I know Carl should have that.

okay, i could confirm on dev.isrd.isi.edu/chaise/search/ that the pages are not expanding similar to facebase. Ironically, the record page on isrd-dev expands more than the facebase deployment.
@crisaless is that what you expect?

authors: crisaless robes

repo: informatics-isi-edu/chaise


text:
Internal error in hatrac

This errors happens randomly in GPCR when trying to access the Helix  and Snake plots under the Target details page.
Here's the stack trace for a recent error:
Mon Jun 20 16:29:34.375845 2016] [:error] [pid 28200] [remote 128.9.184.181:24] Traceback (most recent call last):
[Mon Jun 20 16:29:34.375889 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/web/application.py", line 239, in process
[Mon Jun 20 16:29:34.375893 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     return self.handle()
[Mon Jun 20 16:29:34.375896 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
[Mon Jun 20 16:29:34.375899 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     return self._delegate(fn, self.fvars, args)
[Mon Jun 20 16:29:34.375902 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/web/application.py", line 405, in _delegate
[Mon Jun 20 16:29:34.375904 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     return handle_class(f)
[Mon Jun 20 16:29:34.375906 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/web/application.py", line 396, in handle_class
[Mon Jun 20 16:29:34.375909 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     return tocall(*args)
[Mon Jun 20 16:29:34.375911 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/hatrac/rest/init.py", line 67, in GET
[Mon Jun 20 16:29:34.375913 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     return self.METHOD('GET')
[Mon Jun 20 16:29:34.375915 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/hatrac/rest/init.py", line 61, in METHOD
[Mon Jun 20 16:29:34.375917 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     return method(*matchgroups)
[Mon Jun 20 16:29:34.375919 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/hatrac/rest/core.py", line 214, in wrapper
[Mon Jun 20 16:29:34.375921 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     return original_method(*args)
[Mon Jun 20 16:29:34.375923 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/hatrac/rest/core.py", line 502, in GET
[Mon Jun 20 16:29:34.375926 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     return self._GET(*args)
[Mon Jun 20 16:29:34.375928 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/hatrac/rest/name.py", line 191, in _GET
[Mon Jun 20 16:29:34.375930 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     resource = resource.get_current_version()
[Mon Jun 20 16:29:34.375933 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/hatrac/model/directory/pgsql.py", line 286, in get_current_version
[Mon Jun 20 16:29:34.375935 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     return self.directory.get_current_version(self)
[Mon Jun 20 16:29:34.375943 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/hatrac/model/directory/pgsql.py", line 726, in wrapper
[Mon Jun 20 16:29:34.375956 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     return args[0].pc.perform(db_thunk, transform)
[Mon Jun 20 16:29:34.375959 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/hatrac/model/directory/pgsql.py", line 608, in perform
[Mon Jun 20 16:29:34.375961 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     result = bodyfunc(self.conn, self.cur)
[Mon Jun 20 16:29:34.375963 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/hatrac/model/directory/pgsql.py", line 725, in db_thunk
[Mon Jun 20 16:29:34.375966 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     return original_method(*args1, conn=conn, cur=cur)
[Mon Jun 20 16:29:34.375968 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/hatrac/model/directory/pgsql.py", line 1041, in get_current_version
[Mon Jun 20 16:29:34.375971 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     results = self._version_list(conn, cur, object.id, limit=1)
[Mon Jun 20 16:29:34.375974 2016] [:error] [pid 28200] [remote 128.9.184.181:24]   File "/usr/lib/python2.7/site-packages/hatrac/model/directory/pgsql.py", line 1279, in _version_list
[Mon Jun 20 16:29:34.375976 2016] [:error] [pid 28200] [remote 128.9.184.181:24]     ("%d" % limit) if limit is not None else 'NULL'
[Mon Jun 20 16:29:34.375978 2016] [:error] [pid 28200] [remote 128.9.184.181:24] TypeError: %d format: a number is required, not NoneType
[Mon Jun 20 16:29:34.375989 2016] [:error] [pid 28200] [remote 128.9.184.181:24]
and the corresponding line from ssl_access_log:
128.9.184.181 - - [20/Jun/2016:16:29:34 -0700] "GET /hatrac/target/IMPT-5HT1B/IMPT-5HT1B_snake.html HTTP/1.1" 500 21

@bugacov what version of the code (commit ID or tag) is running on this server?

hatrac is at rbk-20160601.1
On Mon, Jun 20, 2016 at 5:10 PM, Karl Czajkowski notifications@github.com
wrote:

@bugacov https://github.com/bugacov what version of the code (commit ID
or tag) is running on this server?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/informatics-isi-edu/hatrac/issues/29#issuecomment-227306304,
or mute the thread
https://github.com/notifications/unsubscribe/AKDL2cpso53NE-DSabgQ-O0gJ8Oz3FBGks5qNyvZgaJpZM4I6Piw
.[image: Web Bug from
https://github.com/notifications/beacon/AKDL2f74ONkPE5dGh8GOIlk-hnUL09msks5qNyvZgaJpZM4I6Piw.gif]

I have a hard time explaining this bug unless it is related to the other lazy sequencing scenario being investigated in #28 already.  I'm only adding another assert statement to check an internal invariant prior to the code that threw the exception in your stack trace.  We'll need to investigate those revised stack traces if you encounter any more errors...

I just got that error again in the staging machine which is under tag  gpcr-20160622.1
(commit ID: face996c07e3103e1c7864bc3533ad74e9c31ccb)
[Thu Jun 23 12:34:04.423235 2016] [:error] [pid 890] [remote 128.9.136.65:160] "in GET /session, referrer arg is 'None', Referrer header is 'https://staging.gpcrconsortium.org
/chaise/record/'"
[Thu Jun 23 12:34:05.513742 2016] [:error] [pid 892] [remote 128.9.136.65:156] Traceback (most recent call last):
[Thu Jun 23 12:34:05.513779 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/web/application.py", line 239, in process
[Thu Jun 23 12:34:05.513783 2016] [:error] [pid 892] [remote 128.9.136.65:156]     return self.handle()
[Thu Jun 23 12:34:05.513785 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
[Thu Jun 23 12:34:05.513788 2016] [:error] [pid 892] [remote 128.9.136.65:156]     return self._delegate(fn, self.fvars, args)
[Thu Jun 23 12:34:05.513790 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/web/application.py", line 405, in _delegate
[Thu Jun 23 12:34:05.513793 2016] [:error] [pid 892] [remote 128.9.136.65:156]     return handle_class(f)
[Thu Jun 23 12:34:05.513795 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/web/application.py", line 396, in handle_class
[Thu Jun 23 12:34:05.513797 2016] [:error] [pid 892] [remote 128.9.136.65:156]     return tocall(*args)
[Thu Jun 23 12:34:05.513799 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/hatrac/rest/init.py", line 67, in GET
[Thu Jun 23 12:34:05.513801 2016] [:error] [pid 892] [remote 128.9.136.65:156]     return self.METHOD('GET')
[Thu Jun 23 12:34:05.513803 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/hatrac/rest/init.py", line 61, in METHOD
[Thu Jun 23 12:34:05.513805 2016] [:error] [pid 892] [remote 128.9.136.65:156]     return method(*matchgroups)
[Thu Jun 23 12:34:05.513807 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/hatrac/rest/core.py", line 214, in wrapper
[Thu Jun 23 12:34:05.513809 2016] [:error] [pid 892] [remote 128.9.136.65:156]     return original_method(*args)
[Thu Jun 23 12:34:05.513811 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/hatrac/rest/core.py", line 502, in GET
[Thu Jun 23 12:34:05.513813 2016] [:error] [pid 892] [remote 128.9.136.65:156]     return self._GET(*args)
[Thu Jun 23 12:34:05.513815 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/hatrac/rest/name.py", line 188, in _GET
[Thu Jun 23 12:34:05.513817 2016] [:error] [pid 892] [remote 128.9.136.65:156]     path, name
[Thu Jun 23 12:34:05.513819 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/hatrac/rest/core.py", line 261, in resolve
[Thu Jun 23 12:34:05.513821 2016] [:error] [pid 892] [remote 128.9.136.65:156]     return web.ctx.hatrac_directory.name_resolve(fullname, raise_notfound)
[Thu Jun 23 12:34:05.513823 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/hatrac/model/directory/pgsql.py", line 726, in wrapper
[Thu Jun 23 12:34:05.513825 2016] [:error] [pid 892] [remote 128.9.136.65:156]     return args[0].pc.perform(db_thunk, transform)
[Thu Jun 23 12:34:05.513827 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/hatrac/model/directory/pgsql.py", line 608, in perform
[Thu Jun 23 12:34:05.513829 2016] [:error] [pid 892] [remote 128.9.136.65:156]     result = bodyfunc(self.conn, self.cur)
[Thu Jun 23 12:34:05.513831 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/hatrac/model/directory/pgsql.py", line 725, in db_thunk
[Thu Jun 23 12:34:05.513843 2016] [:error] [pid 892] [remote 128.9.136.65:156]     return original_method(*args1, conn=conn, cur=cur)
[Thu Jun 23 12:34:05.513846 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/hatrac/model/directory/pgsql.py", line 1016, in name_resolve
[Thu Jun 23 12:34:05.513848 2016] [:error] [pid 892] [remote 128.9.136.65:156]     return HatracName.construct(self, **self._name_lookup(conn, cur, name))
[Thu Jun 23 12:34:05.513851 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib/python2.7/site-packages/hatrac/model/directory/pgsql.py", line 1245, in _name_lookup
[Thu Jun 23 12:34:05.513853 2016] [:error] [pid 892] [remote 128.9.136.65:156]     for row in list(cur):
[Thu Jun 23 12:34:05.513855 2016] [:error] [pid 892] [remote 128.9.136.65:156]   File "/usr/lib64/python2.7/site-packages/psycopg2/extras.py", line 91, in iter
[Thu Jun 23 12:34:05.513857 2016] [:error] [pid 892] [remote 128.9.136.65:156]     first = res.next()
[Thu Jun 23 12:34:05.513859 2016] [:error] [pid 892] [remote 128.9.136.65:156] ProgrammingError: no results to fetch
[Thu Jun 23 12:34:05.513868 2016] [:error] [pid 892] [remote 128.9.136.65:156]

This cannot seem to be reproduced and there are doubts about the server state at time of error...

authors: bugacov karlcz

repo: informatics-isi-edu/hatrac


text:
Add Cosmo theme to header.xml and fix light navbar icons

Cosmo theme is currently missing from header.xml and therefore not in the sample theme selector on the website. Adding that one here.
The light navbar sample (hdr.navbar.example.xml) currently shows all icons simultaneously. Changing to d-light, etc... fixes this.

@jason-fox Looks like we forgot to update the alternate header in 97a2b1b.
Merging this to re-align.

Thanks for the tip, Roger! Much appreciated.

authors: infotexture lakokkonen

repo: infotexture/dita-bootstrap


text:
Pass manifest to cron job

If the manifest file is specified in puppet facts, use it for the cron
job.

also, installing sysstat for mpstat

repo: infrahouse/puppet-code


text:
Add devstack plugin

This PR adds a devstack plugin. This makes it possible to deploy sg-core in a container while deploying devstack.
Example of working local.conf used while testing this (remove the "jwysogla-devstack") once this is merged. This local.conf will result in a running container with sg-core and ceilometer configured to send metrics to that container:
[[local|localrc]]
ADMIN_PASSWORD=secret
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD

CEILOMETER_BACKEND=sg-core

USE_PYTHON3="True"

VERBOSE="True"

VERBOSE_NO_TIMESTAMP="True"

LIBS_FROM_GIT=aodh,ceilometer,cinder,devstack,devstack-gate,dib-utils,diskimage-builder,glance,heat,keystone,neutron,nova,placement,requirements,swift,sg-core

enable_plugin aodh https://opendev.org/openstack/aodh
enable_plugin ceilometer https://opendev.org/openstack/ceilometer refs/changes/87/898087/1
enable_plugin heat https://opendev.org/openstack/heat
enable_plugin sg-core https://github.com/infrawatch/sg-core jwysogla-devstack

There is another change, that's worth to mention: https://review.opendev.org/c/openstack/ceilometer/+/898087 That will add support for a sg-core backend to ceilometer as can be seen in the example local.conf

Looks like the work is already starting to pay off. I think I can see the same error, which is caught by our CI during my testing in devstack. With the recent rewrite of the TCP publisher in ceilometer, it looks like it fails to properly connect if ceilometer is started before sg-core. Restart of the notification-agent seems to fix the issue. I'll investigate the issue further and propose a fix on the ceilometer side.

Also another thing. I tested a full devstack deployment using the local.conf mentioned above to test with centos stream 9 and with debian bookworm.

@vyzigold this looks great! In future it would be nice to curl sg-core for some metrics after we've created some resources on the cloud (images, volumes).

@yadneshk This is my next step. I have an addition to the telemetry-tempest-plugin almost ready, which makes use of all of this and it queries sg-core.

Since I mentioned it here. Here is a fix for the ceilometer issue. We should be able to merge it once the ceilometer CI is fixed sometime next week probably. After that the sg-core CI here should start succeeding.
https://review.opendev.org/c/openstack/ceilometer/+/898215

Gr8 work, thanks a lot Jaromir.

It's been awhile since I've used devstack. I'm hoping to try this out today; I'll let you know how it goes.

I followed the instructions and got myself a working devstack with a running sg-core container and a set of ceilometer processes:
[stack@stream9 devstack]$ podman ps
CONTAINER ID  IMAGE                              COMMAND     CREATED         STATUS         PORTS       NAMES
3a231c4a1b7a  quay.io/infrawatch/sg-core:latest              30 minutes ago  Up 30 minutes              sg-core

[stack@stream9 devstack]$ ps ax | grep ceil
 248393 ?        Ssl    0:01 ceilometer-polling: master process [/usr/local/bin/ceilometer-polling --polling-namespaces central --config-file /etc/ceilometer/ceilometer.conf]
 248872 ?        Sl     0:01 ceilometer-polling: AgentManager worker(0)
 249237 ?        Ssl    0:01 ceilometer-agent-notification: master process [/usr/local/bin/ceilometer-agent-notification --config-file /etc/ceilometer/ceilometer.conf]
 249581 ?        Sl     0:12 ceilometer-agent-notification: NotificationService worker(0)
 249583 ?        Sl     0:09 ceilometer-agent-notification: NotificationService worker(1)
 249979 ?        Ssl    0:01 ceilometer-polling: master process [/usr/local/bin/ceilometer-polling --polling-namespaces compute --config-file /etc/ceilometer/ceilometer.conf]
 250269 ?        Sl     0:00 ceilometer-polling: AgentManager worker(0)

I tried running tempest to get some ceilometer activity, but my VM ran out of storage and OSP is borked. Whoops.
Anyways, I can hit the sg_core instance.
[stack@stream9 devstack]$ curl localhost:3000/metrics
# HELP sg_total_ceilometer_metric_decode_count
# TYPE sg_total_ceilometer_metric_decode_count counter
sg_total_ceilometer_metric_decode_count{source="SG"} 0
# HELP sg_total_ceilometer_metric_decode_error_count
# TYPE sg_total_ceilometer_metric_decode_error_count counter
sg_total_ceilometer_metric_decode_error_count{source="SG"} 0
# HELP sg_total_ceilometer_msg_received_count
# TYPE sg_total_ceilometer_msg_received_count counter
sg_total_ceilometer_msg_received_count{source="SG"} 0

I'll try to revive my devstack to see if I can get actual metrics flowing, but for now I'm gonna say this is good since it's headed in the right direction and doesn't break anything.

If you deploy devstack and have some image there (I noticed, there is usually a cirros and fedora image right after I deploy), then ceilometer will start sending metrics about the image size without the need to run anything else. (So just stack.sh and wait enough time). The issue is, that ceilometer needs a restart after stack.sh is finished to work, which is wrong and there is a fix already proposed upstream, which is waiting for CI to work. This is also the reason why the CI doesn't work here.

I'll wait until the end of the day. If I don't see a change in pyparsing, which is the first step for our CI to start succeeding and if nobody minds me doing this, I'll just merge this. I think we all agreed, that this is an isolated change and it won't break anything.

authors: csibbitt jokke-ilujo vyzigold yadneshk

repo: infrawatch/sg-core


text:
Documentation problem with "set" function

The README for the set function says that if the value is not specified, it defaults to zero. In fact, the value defaults to 1.
I think the implementation is correct, given the name of the function, and the documentation should be changed.

You're absolutely, right! Nice finding, I fixed it!

authors: guystreeter infusion


text:
Time drift among computer time vs Dynap-se recordings

There is a time difference between the time of a C++ program (via time.h portable time) that sends stimulations and the time returned of the spikes (AEDAT 3.0 recordings).
Computer time (time.h) says 30s, while recording time says 28.5s.

While some drift over time is expected, the two timesources run on different crystals, we're talking about 1s over 10 hours (in that range).
Is there any code for me to look at? It's difficult to say where the problem is... libcaer does a very precise time check of the timestamps VS computer time, and warns if the drift goes about 100us, when you compile it with debug mode active (cmake -DCMAKE_BUILD_TYPE=Debug .); first check if that actually complains. If it doesn't, then the problem is in the stimulation code and how it measures time; there's a lot of things one can get wrong there.

No update in 60 days, closing for lack of information.

authors: federicohyo llongi

repo: inilabs/libcaer