text:
no access to variables "ProtocolType", "UseGMParameterIDs", "VFrameFormat" and "BusType" variables

How the variables are defines in a DBC file
ProtocolType
BA_DEF_  "ProtocolType" STRING ;
BA_DEF_DEF_  "ProtocolType" "GMLAN";
BA_ "ProtocolType" "GMLAN";

UseGMParameterIDs
BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_  "UseGMParameterIDs" 1;

BusType
BA_DEF_  "BusType" STRING ;
BA_DEF_DEF_  "BusType" "";
BA_ "BusType" "CAN";

the 3 above variables  should be accessible through a cantools.database.can.database.Database instance
the variable below should be accessible through a cantools.database.can.message.Message instance
VFrameFormat
BA_DEF_ BO_  "VFrameFormat" ENUM  "StandardCAN","ExtendedCAN","StandardCAN_FD","ExtendedCAN_FD","J1939PG";
BA_DEF_DEF_  "VFrameFormat" "";

BA_ "VFrameFormat" BO_ {MESSAGE ID} 0;

where {MESSAGE ID} is replaced with the id of the message
It is important to have access to these variables when dealing with a GMLAN CAN data system in GM vehicles. This is because of how the arbitration id's are broken down on these systems and hot the message id's are stored in a DBC file for an extended GMLAN id.
received arbitration id on a GMLAN CAN system
0x104F0099
binary of that arbitration id
1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1
with GMLAN the binary data gets broken down like this.
priority                  arbitration_id                                ecu_id
1, 0, 0,        0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0,     0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1

arbitration_id = 0x278
ecu_id = 0x99
priority = 0x4
Now in the world of DBC files the priority and the ecu_id are not important and are not used in the id of a message
so we want to replace those bits with 0,s
priority                  arbitration_id                                ecu_id
0, 0, 0,         0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0,     0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0

arbitration_id = 0x4F0000
message id stored in the dbc file would be 0x4F0000
That is what the purpose of "ProtocolType" and "UseGMParameterIDs" is for. It is the mechanism used to identify the DBC
file as a GMLAN DBC file and that GMLAN Extended id's are being used. This would be the trigger to tell an application
that it needs to modify the arbitration id on received messages to account for this.
I personally feel that this should be handled internally in cantools and if that is what you would like
done let me know and I can make the needed changes for bitshifting the frame id of the DBC is GMLAN
and the GMExtended ids are being used.
I do not even know how to go about accessing the variables needed to do this, I would need a little help
in isolating those.
Or I can handle it in my application code which is fine also bit I would still need access to the above
mentioned variables.
using the following code in a DBC file triggers a response when using Vector CANdb++
BA_DEF_  "ProtocolType" STRING ;
BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_  "ProtocolType" "GMLAN";
BA_DEF_DEF_  "UseGMParameterIDs" 1;
BA_ "ProtocolType" "GMLAN";

when I view a message properties sheet inside of CANdb++ I see the following

and when I go to change the id I get the follow

you can see in that last image that I am not allowed to alter the priority or the ecu_id.
I have not been able to isolate exactly what "Reserved" is for. I am going to tinker with it some this evening and see if I am able to isolate the bits that make up this number.

OK I isolated the reserved bits
priority                  arbitration_id                      reserved                   ecu_id
1, 0, 0,        0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0,     0, 0, 0, 0, 0,       1, 0, 0, 1, 1, 0, 0, 1

It appears there is an error in CANdb++ with regards to this.
I cannot set an arbitration id of 0x1FFFFFFF
1, 1, 1,     1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,      1, 1, 1, 1, 1,       1, 1, 1, 1, 1, 1, 1, 1
It throws an error that is not handled by the application
the highest value that can be set in CANdb++ is 0x1FFFFEFF
1, 1, 1,     1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,     1, 1, 1, 1, 0,       1, 1, 1, 1, 1, 1, 1, 1
which breaks down to
priority = 0x7
arbitration id = 0x1FFF
reserved = 0x1E
ecu_id = 0xFF

"ProtocolType", "UseGMParameterIDs", "VFrameFormat" and "BusType" should be readable in the database.dbc and message.dbc... attributes.
I don't have any experience from GMLAN CAN systems. I've used J1939, and there is a file called j1939.py in this repository with some helper functions for that format. I guess the best approach would be to integrate j1939 and GMLAN into the database and message classes, but I'm not sure how to do it to make it useful. Possibly better to create new classes. I'm not sure, and I don't have time nor experience to think too much about it. However, I would gladly merge pull requests that adds useful functionality, I just don't have good input on how to design it.

This is what I get when I print out cantools.database.can.database.Database.dbc.attribute_definitions
OrderedDict([('UseGMParameterIDs', attribute_definition('UseGMParameterIDs', 1)), ('ProtocolType', attribute_definition('ProtocolType', GMLAN)), ('BusType', attribute_definition('BusType', ))])

the above should read
OrderedDict([('UseGMParameterIDs', attribute_definition('UseGMParameterIDs', 1)), ('ProtocolType', attribute_definition('ProtocolType', GMLAN)), ('BusType', attribute_definition('BusType', CAN))])

OrderedDict([('GenSigStartValue', attribute_definition('GenSigStartValue', 0)), ('ProtocolType', attribute_definition('ProtocolType', )), ('VFrameFormat', attribute_definition('VFrameFormat', ))])

OrderedDict([('GenSigStartValue', attribute_definition('GenSigStartValue', 0)), ('ProtocolType', attribute_definition('ProtocolType', OBD2)), ('VFrameFormat', attribute_definition('VFrameFormat', ["StandardCAN","ExtendedCAN","StandardCAN_FD","ExtendedCAN_FD","J1939PG"]))])

and the above should read
the examples above were done at the database level and they do not change at the message level, They should change at the message level
OrderedDict([('GenSigStartValue', attribute_definition('GenSigStartValue', 0)), ('ProtocolType', attribute_definition('ProtocolType', OBD2)), ('VFrameFormat', attribute_definition('VFrameFormat', StandardCAN))])

I think @geo4711 implemented most of this. What do you think geo4711?

I was sifting through the code in the file parser for the dbc files and I do not see anything in that that would marry an attribute that has an id or name specified to that object, nor do I see anything that would grab the value from an ENUM definition.
I am unsure as to why BusType remains blank and so does VFrameFormat. without the express ability to know if an attribute is set for a specific object in my case a message it is going to make it impossible to know what has been set where.

I can confirm 100% that this does not assign the attribute to the message
BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_  "UseGMParameterIDs" 1;
BA_ "UseGMParameterIDs" BO_ 2155175936 0;

if I understand how this thing works is the first line declares the attribute the second line sets the global default for the attribute and the 3rd line would set the attribute specific to a message that has the id of 2155175936
The global attribute does get set properly but the object specific one a message in this case does not.

I think @geo4711 implemented most of this. What do you think geo4711?

Hmm, the J1939 features have been already available when I joined the project, as well as attribute access for different classes.
Concerning the issue itself (the special handling of CAN ids at GM):

It seems that you've got a special version of Vector's CANdb++ (or at least the diaglog for CAN message ID is a hidden feature, depending on the existence of "useGMParameterIDs").
At least, the GM concept of splitting an extended CAN ID seems to be proprietary (OEM specific), not public (like SAE-J1939 or other stuff).
I'll add separate comments to the other posts ...

I can confirm 100% that this does not assign the attribute to the message
BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_  "UseGMParameterIDs" 1;
BA_ "UseGMParameterIDs" BO_ 2155175936 0;

if I understand how this thing works is the first line declares the attribute the second line sets the global default for the attribute and the 3rd line would set the attribute specific to a message that has the id of 2155175936
The global attribute does get set properly but the object specific one a message in this case does not.

If I'm right, the first two lines define a database attribute (as there's no keyword like "BO_", "BU_", "SG_".
Did you try to insert this:
BA_DEF_  BO_ "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_ BO_ "UseGMParameterIDs" 1;

The third line should fine as is.
Maybe I mixed up something. But I think the only way to have an attribute individual per message is defining it as a message attribute, not a database attribute.

It does still store the arbitration id with bit 31 set if using the GM Extended id

The bit 31 is used in DBCs independent from how you use the real CAN-ID (29 bit).
So no matter if you use standard "CANextended", J1939 (which is on top of CANextended), GMLAN or any other protocol that physically uses 29-bit-CAN-IDs: I guess the bit 31 will always be set.

OK I did locate part of the problem. I am not sure what is actually happening with the use of the ordered dicts.
if the dbc.py file in the _load_messages function I changed the get_attributes function to this
    def get_attributes(frame_id_dbc):
        """Get attributes for given message.

        """
        from collections import OrderedDict

        attrib = OrderedDict()

        for k, v in definitions.items():
            attrib[k] = Attribute(v.default_value, v)

        if frame_id_dbc in attributes and 'message' in attributes[frame_id_dbc]:
            for k, v in attributes[frame_id_dbc]['message'].items():
                attrib[k] = v

        return attrib

This way if a global parameter is set it gets provided to the message and if the message has any specific attributes set it will get those as well. This is beneficial so that if a global parameter gets changes through a message it is actually only going to change it for that message and not globally. If wanting to change a global attribute it should be done at the database level.
I have not used the saving portion of the library and it may goof that up. Might have to add some form of a check to see if a message attribute differs from a globally set on and if it does then save it as a message specific attribute.
as far as the 0x800000 being marked on the arbitration id is concerned.
I wanted to make sure that the behavior of CABdb++ was the same when using the UseGMParameterIDs attribute. It changes the behavior of how the arbitration id's are entered I didn't know if there were any other changes in behavior.
CANdb++ only allows the middle 13 bits to be provided as the ID, If a dbc file is loaded that has a full id including the reserved, priority and ecu id CANdb++ strips this information out of the id, I do not believe that this is something that cantools should do. It should leave the id's in the dbc file alone. Maybe give a warning? What I elected to do was to check for a message match using the id that has been given. If no match is found then I iterate over the stored messages in the dbc file and I check each one for the UseGMParameterIDs attribute and see if it has been set to a 1. If it has the attribute and it has been set to a 1 i take the supplied frame id and perform this operation on it frame_id >> 13 & 0x1FFF and compare that output to the messages frame id. if there is a match then I return the message
    def get_message_by_frame_id(self, frame_id):
        """Find the message object for given frame id `frame_id`.

        """

        if frame_id & self._frame_id_mask in self._frame_id_to_message:
            return self._frame_id_to_message[frame_id & self._frame_id_mask]

        for message in self.messages:
            if (
                'UseGMParameterIDs' in message.dbc.attributes and
                message.dbc.attributes['UseGMParameterIDs'].value == 1
            ):
                # convert to GM  Parameter Id
                id = frame_id >> 13 & 0x1FFF

                if message.frame_id == id:
                    return message

        raise KeyError(frame_id)

I also had to update the encode_message and decode_message methods
        try:
            message = self.get_message_by_frame_id(frame_id_or_name)
        except KeyError:
            message = self.get_message_by_name(frame_id_or_name)

There is something off with the ordered dicts or with the attributes/default attributes. If I print them out they appear to have nothing stored but if I access them they do in fact have data that has been stored.
The other thing is the get_protocol function in the _load_messages function returns a None if J1939PG is not the selected item in the J1939PG. This is an incorrect usage of the vocabulary. The protocol is not derived from the VFrameFormat attribute, the protocol is derived from the ProtocolType attribute. the VFrameFormat attribute is exactly that a frame format or it could also be considered a frame protocol in the loosest meaning. It is there to describe the layout of the bits in a frame. where as the ProtocolType is there to describe the communications protocol that is encapsulated inside of the frame data.
So with GMLAN as an example. I would set the ProtocolType attribute to GMLAN. This is the indicator to look for the UseGMParameterIDs attribute.
If I set the ProtocolType to OBD2 then I would know that the first byte in the data portion of the frame is always going to be the number of bytes that are populated in the frame because in OBD2 the DLC of the frame is always going to be 8 the unused data portions would simply be trailing 0's.
I know that it would break API but I do feel that keeping the vocabulary in line is extremely important. The protocol should be kept in place anywhere that uses it but it should be changed to be populated by the ProtocolType attribute instead. There needs to be methods and variables added for frame_format that would hold the information from the VFrameFormat attribute if that attribute has been set in the users DBC file.
does that make sense?? I know it is kind of a tangle with how I have explained it and I am trying to think of a better way to do it. It might be easier to make the code changes and create a PR if only for the sake of being able to see it that way which would make it easier to see what I am talking about.

ok so here is some updated files

dbc.py (click to expand)
# Load and dump a CAN database in DBC format.

import re
from collections import OrderedDict as odict
from collections import defaultdict
from decimal import Decimal
from copy import deepcopy

import textparser
from textparser import Sequence
from textparser import choice
from textparser import ZeroOrMore
from textparser import OneOrMore
from textparser import OneOrMoreDict
from textparser import DelimitedList
from textparser import Any
from textparser import AnyUntil
from textparser import tokenize_init
from textparser import Token
from textparser import TokenizeError
from textparser import Optional

from ..attribute_definition import AttributeDefinition
from ..attribute import Attribute
from ..signal import Signal
from ..signal import Decimal as SignalDecimal
from ..signal_group import SignalGroup
from ..message import Message
from ..node import Node
from ..bus import Bus
from ..internal_database import InternalDatabase
from ..environment_variable import EnvironmentVariable

from .utils import num

DBC_FMT = (
    'VERSION "{version}"\r\n'
    '\r\n'
    '\r\n'
    'NS_ : \r\n'
    '\tNS_DESC_\r\n'
    '\tCM_\r\n'
    '\tBA_DEF_\r\n'
    '\tBA_\r\n'
    '\tVAL_\r\n'
    '\tCAT_DEF_\r\n'
    '\tCAT_\r\n'
    '\tFILTER\r\n'
    '\tBA_DEF_DEF_\r\n'
    '\tEV_DATA_\r\n'
    '\tENVVAR_DATA_\r\n'
    '\tSGTYPE_\r\n'
    '\tSGTYPE_VAL_\r\n'
    '\tBA_DEF_SGTYPE_\r\n'
    '\tBA_SGTYPE_\r\n'
    '\tSIG_TYPE_REF_\r\n'
    '\tVAL_TABLE_\r\n'
    '\tSIG_GROUP_\r\n'
    '\tSIG_VALTYPE_\r\n'
    '\tSIGTYPE_VALTYPE_\r\n'
    '\tBO_TX_BU_\r\n'
    '\tBA_DEF_REL_\r\n'
    '\tBA_REL_\r\n'
    '\tBA_DEF_DEF_REL_\r\n'
    '\tBU_SG_REL_\r\n'
    '\tBU_EV_REL_\r\n'
    '\tBU_BO_REL_\r\n'
    '\tSG_MUL_VAL_\r\n'
    '\r\n'
    'BS_:\r\n'
    '\r\n'
    'BU_: {bu}\r\n'
    '{val_table}'
    '\r\n'
    '\r\n'
    '{bo}\r\n'
    '\r\n'
    '{bo_tx_bu}\r\n'
    '\r\n'
    '\r\n'
    '{cm}\r\n'
    '{ba_def}\r\n'
    '{ba_def_def}\r\n'
    '{ba}\r\n'
    '{val}\r\n'
    '{signal_types}\r\n'
    '{sig_group}\r\n'
    '{sig_mux_values}\r\n'
)

# Signal types.
SIGNAL_TYPE_FLOAT = 1
SIGNAL_TYPE_DOUBLE = 2

FLOAT_SIGNAL_TYPES = [
    SIGNAL_TYPE_FLOAT,
    SIGNAL_TYPE_DOUBLE
]

FLOAT_LENGTH_TO_SIGNAL_TYPE = {
    32: SIGNAL_TYPE_FLOAT,
    64: SIGNAL_TYPE_DOUBLE
}

ATTRIBUTE_DEFINITION_LONG_NODE_NAME = AttributeDefinition(
    'SystemNodeLongSymbol',
    default_value='',
    kind='BU_',
    type_name='STRING')

ATTRIBUTE_DEFINITION_LONG_MESSAGE_NAME = AttributeDefinition(
    'SystemMessageLongSymbol',
    default_value='',
    kind='BO_',
    type_name='STRING')

ATTRIBUTE_DEFINITION_LONG_SIGNAL_NAME = AttributeDefinition(
    'SystemSignalLongSymbol',
    default_value='',
    kind='SG_',
    type_name='STRING')

def to_int(value):
    return int(Decimal(value))

class Parser(textparser.Parser):

    def tokenize(self, string):
        keywords = set([
            'BA_',
            'BA_DEF_',
            'BA_DEF_DEF_',
            'BA_DEF_DEF_REL_',
            'BA_DEF_REL_',
            'BA_DEF_SGTYPE_',
            'BA_REL_',
            'BA_SGTYPE_',
            'BO_',
            'BO_TX_BU_',
            'BS_',
            'BU_',
            'BU_BO_REL_',
            'BU_EV_REL_',
            'BU_SG_REL_',
            'CAT_',
            'CAT_DEF_',
            'CM_',
            'ENVVAR_DATA_',
            'EV_',
            'EV_DATA_',
            'FILTER',
            'NS_',
            'NS_DESC_',
            'SG_',
            'SG_MUL_VAL_',
            'SGTYPE_',
            'SGTYPE_VAL_',
            'SIG_GROUP_',
            'SIG_TYPE_REF_',
            'SIG_VALTYPE_',
            'SIGTYPE_VALTYPE_',
            'VAL_',
            'VAL_TABLE_',
            'VERSION'
        ])

        names = {
            'LPAREN': '(',
            'RPAREN': ')',
            'LBRACE': '[',
            'RBRACE': ']',
            'COMMA':  ',',
            'AT':     '@',
            'SCOLON': ';',
            'COLON':  ':',
            'PIPE':   '|',
            'SIGN':   '+/-'
        }

        token_specs = [
            ('SKIP',     r'[ \r\n\t]+|//.*?\n'),
            ('NUMBER',   r'[-+]?\d+\.?\d*([eE][+-]?\d+)?'),
            ('WORD',     r'[A-Za-z0-9_]+'),
            ('STRING',   r'"(\\"|[^"])*?"'),
            ('LPAREN',   r'\('),
            ('RPAREN',   r'\)'),
            ('LBRACE',   r'\['),
            ('RBRACE',   r'\]'),
            ('COMMA',    r','),
            ('PIPE',     r'\|'),
            ('AT',       r'@'),
            ('SIGN',     r'[+-]'),
            ('SCOLON',   r';'),
            ('COLON',    r':'),
            ('MISMATCH', r'.')
        ]

        tokens, token_regex = tokenize_init(token_specs)

        for mo in re.finditer(token_regex, string, re.DOTALL):
            kind = mo.lastgroup

            if kind == 'SKIP':
                pass
            elif kind == 'STRING':
                value = mo.group(kind)[1:-1].replace('\\"', '"')
                tokens.append(Token(kind, value, mo.start()))
            elif kind != 'MISMATCH':
                value = mo.group(kind)

                if value in keywords:
                    kind = value

                if kind in names:
                    kind = names[kind]

                tokens.append(Token(kind, value, mo.start()))
            else:
                raise TokenizeError(string, mo.start())

        return tokens

    def grammar(self):
        version = Sequence('VERSION', 'STRING')

        ns = Sequence('NS_', ':', AnyUntil(Sequence(Any(), ':')))

        bs = Sequence('BS_', ':')

        nodes = Sequence('BU_', ':', ZeroOrMore('WORD'))

        signal = Sequence(
            'SG_', choice(Sequence('WORD', 'WORD'), Sequence('WORD')), ':',
            'NUMBER', '|', 'NUMBER', '@', 'NUMBER', '+/-',
            '(', 'NUMBER', ',', 'NUMBER', ')',
            '[', 'NUMBER', '|', 'NUMBER', ']',
            'STRING',
            DelimitedList('WORD'))

        message = Sequence(
            'BO_', 'NUMBER', 'WORD', ':', 'NUMBER', 'WORD', ZeroOrMore(signal))

        environment_variable = Sequence(
            'EV_', 'WORD', ':', 'NUMBER',
            '[', 'NUMBER', '|', 'NUMBER', ']',
            'STRING', 'NUMBER', 'NUMBER', 'WORD', 'WORD', ';')

        comment = Sequence(
            'CM_',
            choice(
                Sequence('SG_', 'NUMBER', 'WORD', 'STRING'),
                Sequence('BO_', 'NUMBER', 'STRING'),
                Sequence('EV_', 'WORD', 'STRING'),
                Sequence('BU_', 'WORD', 'STRING'),
                'STRING'),
            ';')

        attribute_definition = Sequence(
            'BA_DEF_',
            Optional(choice('SG_', 'BO_', 'EV_', 'BU_')),
            'STRING',
            'WORD',
            Optional(choice(DelimitedList('STRING'), ZeroOrMore('NUMBER'))),
            ';')

        attribute_definition_default = Sequence(
            'BA_DEF_DEF_', 'STRING', choice('NUMBER', 'STRING'), ';')

        attribute = Sequence(
            'BA_', 'STRING',
            ZeroOrMore(choice(Sequence('BO_', 'NUMBER'),
                              Sequence('SG_', 'NUMBER', 'WORD'),
                              Sequence('BU_', 'WORD'),
                              Sequence('EV_', 'WORD'))),
            choice('NUMBER', 'STRING'),
            ';')

        attribute_definition_rel = Sequence(
            'BA_DEF_REL_',
            Optional('BU_SG_REL_'),
            'STRING',
            'WORD',
            choice(DelimitedList('STRING'), OneOrMore('NUMBER')),
            ';')

        attribute_definition_default_rel = Sequence(
            'BA_DEF_DEF_REL_', 'STRING', choice('NUMBER', 'STRING'), ';')

        attribute_rel = Sequence(
            'BA_REL_', 'STRING', 'BU_SG_REL_', 'WORD', 'SG_', 'NUMBER',
            'WORD', choice('NUMBER', 'STRING'), ';')

        choice_ = Sequence(
            'VAL_',
            Optional('NUMBER'),
            'WORD',
            ZeroOrMore(Sequence('NUMBER', 'STRING')),
            ';')

        value_table = Sequence(
            'VAL_TABLE_', 'WORD', ZeroOrMore(Sequence('NUMBER', 'STRING')), ';')

        signal_type = Sequence(
            'SIG_VALTYPE_', 'NUMBER', 'WORD', ':', 'NUMBER', ';')

        signal_multiplexer_values = Sequence(
            'SG_MUL_VAL_',
            'NUMBER',
            'WORD',
            'WORD',
            DelimitedList(Sequence('NUMBER', 'NUMBER')),
            ';')

        message_add_sender = Sequence(
            'BO_TX_BU_', 'NUMBER', ':', DelimitedList('WORD'), ';')

        signal_group = Sequence(
            'SIG_GROUP_', 'NUMBER', 'WORD', 'NUMBER', ':', ZeroOrMore('WORD'), ';')

        return OneOrMoreDict(
            choice(
                message,
                comment,
                attribute_definition,
                value_table,
                choice_,
                attribute,
                attribute_rel,
                attribute_definition_rel,
                attribute_definition_default,
                attribute_definition_default_rel,
                signal_group,
                signal_type,
                signal_multiplexer_values,
                message_add_sender,
                environment_variable,
                nodes,
                ns,
                bs,
                version))

class DbcSpecifics(object):

    def __init__(self,
                 attributes=None,
                 attribute_definitions=None,
                 environment_variables=None,
                 value_tables=None):
        if attributes is None:
            attributes = odict()

        if attribute_definitions is None:
            attribute_definitions = odict()

        if environment_variables is None:
            environment_variables = odict()

        if value_tables is None:
            value_tables = odict()

        self._attributes = attributes
        self._attribute_definitions = attribute_definitions
        self._environment_variables = environment_variables
        self._value_tables = value_tables

    @property
    def attributes(self):
        """The DBC specific attributes of the parent object (database, node,
        message or signal) as a dictionary.

        """

        return self._attributes

    @attributes.setter
    def attributes(self, value):
        self._attributes = value

    @property
    def attribute_definitions(self):
        """The DBC specific attribute definitions as dictionary.

        """

        return self._attribute_definitions

    @property
    def value_tables(self):
        """An ordered dictionary of all value tables. Only valid for DBC
        specifiers on database level.

        """

        return self._value_tables

    @property
    def environment_variables(self):
        """An ordered dictionary of all environment variables. Only valid for
        DBC specifiers on database level.

        """

        return self._environment_variables

class LongNamesConverter(object):

    def __init__(self, database):
        self._database = database
        self._next_index_per_cut_name = defaultdict(int)
        self._short_names = set()

    def convert(self, name):
        short_name = None

        if len(name) == 32:
            self._short_names.add(name)
        elif len(name) > 32:
            cut_name = name[:27]
            short_name = name[:32]

            if short_name in self._short_names:
                index = self._next_index_per_cut_name[cut_name]
                self._next_index_per_cut_name[cut_name] += 1
                short_name = '{}_{:04d}'.format(name[:27], index)
            else:
                self._next_index_per_cut_name[cut_name] = 0
                self._short_names.add(short_name)

        return short_name

def get_dbc_frame_id(message):
    frame_id = message.frame_id

    if message.is_extended_frame:
        frame_id |= 0x80000000

    return frame_id

def _get_node_name(attributes, name):
    try:
        return attributes['node'][name]['SystemNodeLongSymbol'].value
    except (KeyError, TypeError):
        return name

def _get_environment_variable_name(attributes, name):
    try:
        return attributes['envvar'][name]['SystemEnvVarLongSymbol'].value
    except (KeyError, TypeError):
        return name

def _dump_version(database):
    return '' if database.version is None else database.version

def _dump_nodes(database):
    bu = []

    for node in database.nodes:
        bu.append(node.name)

    return bu

def _dump_value_tables(database):
    if database.dbc is None:
        return []

    val_table = []

    for name, choices in database.dbc.value_tables.items():
        choices = [
            '{} "{}"'.format(number, text)
            for number, text in reversed(sorted(choices.items()))
        ]
        val_table.append('VAL_TABLE_ {} {} ;'.format(name, ' '.join(choices)))

    return val_table + ['']

def _dump_messages(database):
    bo = []

    def format_mux(signal):
        if signal.is_multiplexer:
            return ' M'
        elif signal.multiplexer_ids is not None:
            return ' m{}'.format(signal.multiplexer_ids[0])
        else:
            return ''

    def format_receivers(signal):
        if signal.receivers:
            return ' ' + ','.join(signal.receivers)
        else:
            return 'Vector__XXX'

    def format_senders(message):
        if message.senders:
            return message.senders[0]
        else:
            return 'Vector__XXX'

    for message in database.messages:
        msg = []
        msg.append(
            'BO_ {frame_id} {name}: {length} {senders}'.format(
                frame_id=get_dbc_frame_id(message),
                name=message.name,
                length=message.length,
                senders=format_senders(message)))

        for signal in message.signals[::-1]:
            fmt = (' SG_ {name}{mux} : {start}|{length}@{byte_order}{sign}'
                   ' ({scale},{offset})'
                   ' [{minimum}|{maximum}] "{unit}" {receivers}')
            msg.append(fmt.format(
                name=signal.name,
                mux=format_mux(signal),
                start=signal.start,
                length=signal.length,
                receivers=format_receivers(signal),
                byte_order=(0 if signal.byte_order == 'big_endian' else 1),
                sign=('-' if signal.is_signed else '+'),
                scale=signal.scale,
                offset=signal.offset,
                minimum=(0 if signal.minimum is None else signal.minimum),
                maximum=(0 if signal.maximum is None else signal.maximum),
                unit='' if signal.unit is None else signal.unit))

        bo.append('\r\n'.join(msg))

    return bo

def _dump_senders(database):
    bo_tx_bu = []

    for message in database.messages:
        if len(message.senders) > 1:
            bo_tx_bu.append(
                'BO_TX_BU_ {frame_id} : {senders};'.format(
                    frame_id=get_dbc_frame_id(message),
                    senders=','.join(message.senders)))

    return bo_tx_bu

def _dump_comments(database):
    cm = []

    for bus in database.buses:
        if bus.comment is not None:
            cm.append(
                'CM_ "{comment}";'.format(
                    comment=bus.comment) )

    for node in database.nodes:
        if node.comment is not None:
            cm.append(
                'CM_ BU_ {name} "{comment}";'.format(
                    name=node.name,
                    comment=node.comment.replace('"', '\\"')))

    for message in database.messages:
        if message.comment is not None:
            cm.append(
                'CM_ BO_ {frame_id} "{comment}";'.format(
                    frame_id=get_dbc_frame_id(message),
                    comment=message.comment.replace('"', '\\"')))

        for signal in message.signals[::-1]:
            if signal.comment is not None:
                cm.append(
                    'CM_ SG_ {frame_id} {name} "{comment}";'.format(
                        frame_id=get_dbc_frame_id(message),
                        name=signal.name,
                        comment=signal.comment.replace('"', '\\"')))

    return cm

def _dump_signal_types(database):
    valtype = []

    for message in database.messages:
        for signal in message.signals:
            if not signal.is_float:
                continue

            valtype.append(
                'SIG_VALTYPE_ {} {} : {};'.format(
                    get_dbc_frame_id(message),
                    signal.name,
                    FLOAT_LENGTH_TO_SIGNAL_TYPE[signal.length]))

    return valtype

def _dump_attribute_definitions(database):
    ba_def = []

    if database.dbc is None:
        return ba_def

    definitions = database.dbc.attribute_definitions

    def get_value(definition, value):
        if definition.minimum is None:
            value = ''
        else:
            value = ' {}'.format(value)

        return value

    def get_minimum(definition):
        return get_value(definition, definition.minimum)

    def get_maximum(definition):
        return get_value(definition, definition.maximum)

    def get_kind(definition):
        return '' if definition.kind is None else definition.kind + ' '

    for definition in definitions.values():
        if definition.type_name == 'ENUM':
            choices = ','.join(['"{}"'.format(choice)
                                for choice in definition.choices])
            ba_def.append(
                'BA_DEF_ {kind} "{name}" {type_name}  {choices};'.format(
                    kind=get_kind(definition),
                    name=definition.name,
                    type_name=definition.type_name,
                    choices=choices))
        elif definition.type_name in ['INT', 'FLOAT', 'HEX']:
            ba_def.append(
                'BA_DEF_ {kind} "{name}" {type_name}{minimum}{maximum};'.format(
                    kind=get_kind(definition),
                    name=definition.name,
                    type_name=definition.type_name,
                    minimum=get_minimum(definition),
                    maximum=get_maximum(definition)))
        elif definition.type_name == 'STRING':
            ba_def.append(
                'BA_DEF_ {kind} "{name}" {type_name} ;'.format(
                    kind=get_kind(definition),
                    name=definition.name,
                    type_name=definition.type_name))

    return ba_def

def _dump_attribute_definition_defaults(database):
    ba_def_def = []

    if database.dbc is None:
        return ba_def_def

    definitions = database.dbc.attribute_definitions

    for definition in definitions.values():
        if definition.default_value is not None:
            if definition.type_name in ["STRING", "ENUM"]:
                fmt = 'BA_DEF_DEF_  "{name}" "{value}";'
            else:
                fmt = 'BA_DEF_DEF_  "{name}" {value};'

            ba_def_def.append(fmt.format(name=definition.name,
                                         value=definition.default_value))

    return ba_def_def

def _dump_attributes(database):
    ba = []

    def get_value(attribute):
        result = attribute.value

        if attribute.definition.type_name == "STRING":
            result = '"' + attribute.value + '"'

        return result

    if database.dbc is not None:
        if database.dbc.attributes is not None:
            for attribute in database.dbc.attributes.values():
                ba.append(
                    'BA_ "{name}" {value};'.format(name=attribute.definition.name,
                                                    value=get_value(attribute)))

    for node in database.nodes:
        if node.dbc is not None:
            if node.dbc.attributes is not None:
                for attribute in node.dbc.attributes.values():
                    if attribute.value == attribute.definition.default_value:
                        continue

                    ba.append(
                        'BA_ "{name}" {kind} {node_name} {value};'.format(
                            name=attribute.definition.name,
                            kind=attribute.definition.kind,
                            node_name=node.name,
                            value=get_value(attribute)))

    for message in database.messages:
        if message.dbc is not None:
            if message.dbc.attributes is not None:
                for attribute in message.dbc.attributes.values():
                    if attribute.value == attribute.definition.default_value:
                        continue

                    ba.append(
                        'BA_ "{name}" {kind} {frame_id} {value};'.format(
                            name=attribute.definition.name,
                            kind=attribute.definition.kind,
                            frame_id=get_dbc_frame_id(message),
                            value=get_value(attribute)))

        for signal in message.signals[::-1]:
            if signal.dbc is not None:
                if signal.dbc.attributes is not None:
                    for attribute in signal.dbc.attributes.values():
                        if attribute.value == attribute.definition.default_value:
                            continue

                        ba.append(
                            'BA_ "{name}" {kind} {frame_id} {signal_name} {value};'.format(
                                name=attribute.definition.name,
                                kind=attribute.definition.kind,
                                frame_id=get_dbc_frame_id(message),
                                signal_name=signal.name,
                                value=get_value(attribute)))

    return ba

def _dump_choices(database):
    val = []

    for message in database.messages:
        for signal in message.signals[::-1]:
            if signal.choices is None:
                continue

            val.append(
                'VAL_ {frame_id} {name} {choices} ;'.format(
                    frame_id=get_dbc_frame_id(message),
                    name=signal.name,
                    choices=' '.join(['{value} "{text}"'.format(value=value,
                                                                text=text)
                                      for value, text in signal.choices.items()])))

    return val

def _dump_signal_groups(database):
    sig_group = []

    for message in database.messages:
        if message.signal_groups is None:
            continue

        for signal_group in message.signal_groups:
            all_sig_names = list(map(lambda sig: sig.name, message.signals))
            signal_group.signal_names = list(filter(lambda sig_name: sig_name in all_sig_names, signal_group.signal_names))
            sig_group.append(
                'SIG_GROUP_ {frame_id} {signal_group_name} {repetitions} : {signal_names};'.format(
                    frame_id=get_dbc_frame_id(message),
                    signal_group_name=signal_group.name,
                    repetitions=signal_group.repetitions,
                    signal_names=' '.join(signal_group.signal_names)
                ))

    return sig_group

def _is_extended_mux_needed(messages):
    """Check for messages with more than one mux signal or signals with
    more than one multiplexer value.

    """

    for message in messages:
        multiplexers = [
            signal.name
            for signal in message.signals
            if signal.is_multiplexer
        ]

        if len(multiplexers) > 1:
            return True

        for signal in message.signals:
            if signal.multiplexer_ids:
                if len(signal.multiplexer_ids) > 1:
                    return True

    return False

def _create_mux_ranges(multiplexer_ids):
    """Create a list of ranges based on a list of single values.

    Example:
        Input:  [1, 2, 3, 5,      7, 8, 9]
        Output: [[1, 3], [5, 5], [7, 9]]

    """

    ordered = sorted(multiplexer_ids)
    # Anything but ordered[0] - 1
    prev_value = ordered[0]
    ranges = []

    for value in ordered:
        if value == prev_value + 1:
            ranges[-1][1] = value
        else:
            ranges.append([value, value])

        prev_value = value

    return ranges

def _dump_signal_mux_values(database):
    """Create multiplex entries ("SG_MUL_VAL_") if extended multiplexing
    is used.

    """

    if not _is_extended_mux_needed(database.messages):
        return []

    sig_mux_values = []

    for message in database.messages:
        for signal in message.signals:
            if not signal.multiplexer_ids:
                continue

            ranges = ', '.join([
                '{}-{}'.format(minimum, maximum)
                for minimum, maximum in _create_mux_ranges(signal.multiplexer_ids)
            ])

            sig_mux_values.append(
                'SG_MUL_VAL_ {frame_id} {name} {multiplexer} {ranges};'.format(
                    frame_id=get_dbc_frame_id(message),
                    name=signal.name,
                    multiplexer=signal.multiplexer_signal,
                    ranges=ranges))

    return sig_mux_values

def _load_comments(tokens):
    comments = defaultdict(dict)

    for comment in tokens.get('CM_', []):
        if not isinstance(comment[1], list):
            comments['database']['bus'] = comment[1]
            continue

        item = comment[1]
        kind = item[0]

        if kind == 'SG_':
            frame_id = int(item[1])

            if 'signal' not in comments[frame_id]:
                comments[frame_id]['signal'] = {}

            comments[frame_id]['signal'][item[2]] = item[3]
        elif kind == 'BO_':
            frame_id = int(item[1])
            comments[frame_id]['message'] = item[2]
        elif kind == 'BU_':
            node_name = item[1]
            comments[node_name] = item[2]
        elif kind == 'EV_':
            environment_variable_name = item[1]
            comments[environment_variable_name] = item[2]

    return comments

def _load_attribute_definitions(tokens):
    return tokens.get('BA_DEF_', [])

def _load_attribute_definition_defaults(tokens):
    defaults = odict()

    for default_attr in tokens.get('BA_DEF_DEF_', []):
        defaults[default_attr[1]] = default_attr[2]

    return defaults

def _load_attributes(tokens, definitions):
    attributes = odict()
    attributes['node'] = odict()

    def to_object(attribute):
        value = attribute[3]

        definition = definitions[attribute[1]]

        if definition.type_name in ['INT', 'HEX', 'ENUM']:
            value = to_int(value)
        elif definition.type_name == 'FLOAT':
            value = Decimal(value)

        return Attribute(value=value,
                         definition=definition)

    for attribute in tokens.get('BA_', []):
        name = attribute[1]

        if len(attribute[2]) > 0:
            item = attribute[2][0]
            kind = item[0]

            if kind == 'SG_':
                frame_id_dbc = int(item[1])
                signal = item[2]

                if frame_id_dbc not in attributes:
                    attributes[frame_id_dbc] = {}
                    attributes[frame_id_dbc]['message'] = odict()

                if 'signal' not in attributes[frame_id_dbc]:
                    attributes[frame_id_dbc]['signal'] = odict()

                if signal not in attributes[frame_id_dbc]['signal']:
                    attributes[frame_id_dbc]['signal'][signal] = odict()

                attributes[frame_id_dbc]['signal'][signal][name] = to_object(attribute)
            elif kind == 'BO_':
                frame_id_dbc = int(item[1])

                if frame_id_dbc not in attributes:
                    attributes[frame_id_dbc] = {}
                    attributes[frame_id_dbc]['message'] = odict()

                attributes[frame_id_dbc]['message'][name] = to_object(attribute)
            elif kind == 'BU_':
                node = item[1]

                if node not in attributes['node']:
                    attributes['node'][node] = odict()

                attributes['node'][node][name] = to_object(attribute)
            elif kind == 'EV_':
                envvar = item[1]

                if 'envvar' not in attributes:
                    attributes['envvar'] = odict()

                if envvar not in attributes['envvar']:
                    attributes['envvar'][envvar] = odict()

                attributes['envvar'][envvar][name] = to_object(attribute)
        else:
            if 'database' not in attributes:
                attributes['database'] = odict()

            attributes['database'][name] = to_object(attribute)

    return attributes

def _load_value_tables(tokens):
    """Load value tables, that is, choice definitions.

    """

    value_tables = odict()

    for value_table in tokens.get('VAL_TABLE_', []):
        name = value_table[1]
        choices = {int(number): text for number, text in value_table[2]}
        value_tables[name] = choices

    return value_tables

def _load_environment_variables(tokens, comments, attributes):
    environment_variables = odict()

    for env_var in tokens.get('EV_', []):
        name = _get_environment_variable_name(attributes, env_var[1])
        environment_variables[name] = EnvironmentVariable(
            name=name,
            env_type=int(env_var[3]),
            minimum=num(env_var[5]),
            maximum=num(env_var[7]),
            unit=env_var[9],
            initial_value=num(env_var[10]),
            env_id=int(env_var[11]),
            access_type=env_var[12],
            access_node=env_var[13],
            comment=comments.get(env_var[1], None))

    return environment_variables

def _load_choices(tokens):
    choices = defaultdict(dict)

    for choice in tokens.get('VAL_', []):
        if len(choice[1]) == 0:
            continue

        od = odict((int(''.join(v[0])), v[1]) for v in choice[3])

        if len(od) == 0:
            continue

        frame_id = int(choice[1][0])
        choices[frame_id][choice[2]] = od

    return choices

def _load_message_senders(tokens, attributes):
    """Load additional message senders.

    """

    message_senders = defaultdict(list)

    for senders in tokens.get('BO_TX_BU_', []):
        frame_id = int(senders[1])
        message_senders[frame_id] += [
            _get_node_name(attributes, sender) for sender in senders[3]
        ]

    return message_senders

def _load_signal_types(tokens):
    """Load signal types.

    """

    signal_types = defaultdict(dict)

    for signal_type in tokens.get('SIG_VALTYPE_', []):
        frame_id = int(signal_type[1])
        signal_name = signal_type[2]
        signal_types[frame_id][signal_name] = int(signal_type[4])

    return signal_types

def _load_signal_multiplexer_values(tokens):
    """Load additional signal multiplexer values.

    """

    signal_multiplexer_values = defaultdict(dict)

    for signal_multiplexer_value in tokens.get('SG_MUL_VAL_', []):
        frame_id = int(signal_multiplexer_value[1])
        signal_name = signal_multiplexer_value[2]
        multiplexer_signal = signal_multiplexer_value[3]
        multiplexer_ids = []

        for lower, upper in signal_multiplexer_value[4]:
            lower = int(lower)
            upper = int(upper[1:])
            # ToDo: Probably store ranges as tuples to not run out of
            #       memory on huge ranges.
            multiplexer_ids.extend(range(lower, upper + 1))

        if multiplexer_signal not in signal_multiplexer_values[frame_id]:
            signal_multiplexer_values[frame_id][multiplexer_signal] = {}

        multiplexer_signal = signal_multiplexer_values[frame_id][multiplexer_signal]
        multiplexer_signal[signal_name] = multiplexer_ids

    return signal_multiplexer_values

def _load_signal_groups(tokens):
    """Load signal groups.

    """

    signal_groups = defaultdict(list)

    for signal_group in tokens.get('SIG_GROUP_',[]):
        frame_id = int(signal_group[1])
        signal_groups[frame_id].append(SignalGroup(name=signal_group[2],
                                                   repetitions=int(signal_group[3]),
                                                   signal_names=signal_group[5]))

    return signal_groups

def _load_signals(tokens,
                  comments,
                  attributes,
                  definitions,
                  choices,
                  signal_types,
                  signal_multiplexer_values,
                  frame_id_dbc,
                  multiplexer_signal):
    signal_to_multiplexer = {}

    try:
        signal_multiplexer_values = signal_multiplexer_values[frame_id_dbc]

        for multiplexer_name, items in signal_multiplexer_values.items():
            for name in items:
                signal_to_multiplexer[name] = multiplexer_name
    except KeyError:
        pass

    def get_attributes(frame_id_dbc, signal):
        """Get attributes for given signal.

        """

        try:
            return attributes[frame_id_dbc]['signal'][signal]
        except KeyError:
            return None

    def get_comment(frame_id_dbc, signal):
        """Get comment for given signal.

        """

        try:
            return comments[frame_id_dbc]['signal'][signal]
        except:
            return None

    def get_choices(frame_id_dbc, signal):
        """Get choices for given signal.

        """

        try:
            return choices[frame_id_dbc][signal]
        except KeyError:
            return None

    def get_is_multiplexer(signal):
        if len(signal[1]) == 2:
            return signal[1][1].endswith('M')
        else:
            return False

    def get_multiplexer_ids(signal, multiplexer_signal):
        ids = []

        if multiplexer_signal is not None:
            if len(signal) == 2 and not signal[1].endswith('M'):
                value = signal[1][1:].rstrip('M')
                ids.append(int(value))
        else:
            multiplexer_signal = get_multiplexer_signal(signal,
                                                        multiplexer_signal)

        try:
            ids.extend(
                signal_multiplexer_values[multiplexer_signal][signal[0]])
        except KeyError:
            pass

        if ids:
            return list(set(ids))

    def get_multiplexer_signal(signal, multiplexer_signal):
        if len(signal) != 2:
            return

        if multiplexer_signal is None:
            try:
                return signal_to_multiplexer[signal[0]]
            except KeyError:
                pass
        elif signal[0] != multiplexer_signal:
            return multiplexer_signal

    def get_receivers(receivers):
        if receivers == ['Vector__XXX']:
            receivers = []

        return [_get_node_name(attributes, receiver) for receiver in receivers]

    def get_minimum(minimum, maximum):
        if minimum == maximum == '0':
            return None
        else:
            return num(minimum)

    def get_maximum(minimum, maximum):
        if minimum == maximum == '0':
            return None
        else:
            return num(maximum)

    def get_minimum_decimal(minimum, maximum):
        if minimum == maximum == '0':
            return None
        else:
            return Decimal(minimum)

    def get_maximum_decimal(minimum, maximum):
        if minimum == maximum == '0':
            return None
        else:
            return Decimal(maximum)

    def get_is_float(frame_id_dbc, signal):
        """Get is_float for given signal.

        """

        try:
            return signal_types[frame_id_dbc][signal] in FLOAT_SIGNAL_TYPES
        except KeyError:
            return False

    def get_signal_name(frame_id_dbc, name):
        signal_attributes = get_attributes(frame_id_dbc, name)

        try:
            return signal_attributes['SystemSignalLongSymbol'].value
        except (KeyError, TypeError):
            return name

    def get_signal_initial_value(frame_id_dbc, name):
        signal_attributes = get_attributes(frame_id_dbc, name)

        try:
            return signal_attributes['GenSigStartValue'].value
        except (KeyError, TypeError):
            return None

    def get_signal_spn(frame_id_dbc, name):
        signal_attributes = get_attributes(frame_id_dbc, name)

        try:
            return signal_attributes['SPN'].value
        except (KeyError, TypeError):
            return None

    signals = []

    for signal in tokens:
        signals.append(
            Signal(name=get_signal_name(frame_id_dbc, signal[1][0]),
                   start=int(signal[3]),
                   length=int(signal[5]),
                   receivers=get_receivers(signal[20]),
                   byte_order=('big_endian'
                               if signal[7] == '0'
                               else 'little_endian'),
                   is_signed=(signal[8] == '-'),
                   initial=get_signal_initial_value(frame_id_dbc, signal[1][0]),
                   scale=num(signal[10]),
                   offset=num(signal[12]),
                   minimum=get_minimum(signal[15], signal[17]),
                   maximum=get_maximum(signal[15], signal[17]),
                   decimal=SignalDecimal(Decimal(signal[10]),
                                         Decimal(signal[12]),
                                         get_minimum_decimal(signal[15],
                                                             signal[17]),
                                         get_maximum_decimal(signal[15],
                                                             signal[17])),
                   unit=(None if signal[19] == '' else signal[19]),
                   spn=get_signal_spn(frame_id_dbc, signal[1][0]),
                   choices=get_choices(frame_id_dbc,
                                       signal[1][0]),
                   dbc_specifics=DbcSpecifics(get_attributes(frame_id_dbc, signal[1][0]),
                                              definitions),
                   comment=get_comment(frame_id_dbc,
                                       signal[1][0]),
                   is_multiplexer=get_is_multiplexer(signal),
                   multiplexer_ids=get_multiplexer_ids(signal[1],
                                                       multiplexer_signal),
                   multiplexer_signal=get_multiplexer_signal(signal[1],
                                                             multiplexer_signal),
                   is_float=get_is_float(frame_id_dbc, signal[1][0])))

    return signals

def _load_messages(tokens,
                   comments,
                   attributes,
                   definitions,
                   choices,
                   message_senders,
                   signal_types,
                   signal_multiplexer_values,
                   strict,
                   bus_name,
                   signal_groups):
    """Load messages.

    """

    def get_attributes(frame_id_dbc):
        """Get attributes for given message.

        """
        from collections import OrderedDict

        attrib = OrderedDict()

        for k, v in definitions.items():
            attrib[k] = Attribute(v.default_value, v)

        if frame_id_dbc in attributes and 'message' in attributes[frame_id_dbc]:
            for k, v in attributes[frame_id_dbc]['message'].items():
                attrib[k] = v

        return attrib

    def get_comment(frame_id_dbc):
        """Get comment for given message.

        """

        try:
            return comments[frame_id_dbc]['message']
        except:
            return None

    def get_send_type(frame_id_dbc):
        """Get send type for a given message.

        """

        message_attributes = get_attributes(frame_id_dbc)

        if 'GenMsgSendType' in message_attributes:
            result = message_attributes['GenMsgSendType'].value
            # Resolve ENUM index to ENUM text
            result = message_attributes['GenMsgSendType'].definition.choices[int(result)]
        else:
            result = None

        return result

    def get_cycle_time(frame_id_dbc):
        """Get cycle time for a given message.

        """

        message_attributes = get_attributes(frame_id_dbc)

        if 'GenMsgCycleTime' in message_attributes:
            result = int(message_attributes['GenMsgCycleTime'].value)
        else:
            result = None

        return result

    def get_frame_format(frame_id_dbc):
        """Gets the frame format for a given message

        """
        message_attributes = get_attributes(frame_id_dbc)

        if 'VFrameFormat' in message_attributes:
            frame_format = message_attributes['VFrameFormat'].value
            frame_format = message_attributes['VFrameFormat'].definition.choices[frame_format]
        else:
            frame_format = ''

        return 'j1939' if frame_format == 'J1939PG' else None

    def get_protocol(frame_id_dbc):
        """Get protocol for a given message.

        """

        message_attributes = get_attributes(frame_id_dbc)

        if 'ProtocolType' in message_attributes:
            protocol_type = message_attributes['ProtocolType'].value
        else:
            protocol_type = None

        return protocol_type

    def get_message_name(frame_id_dbc, name):
        message_attributes = get_attributes(frame_id_dbc)

        if 'SystemMessageLongSymbol' in message_attributes:
            result = message_attributes['SystemMessageLongSymbol'].value
        else:
            result = name

        return result

    def get_signal_groups(frame_id_dbc):
        try:
            return signal_groups[frame_id_dbc]
        except:
            return None

    messages = []

    for message in tokens.get('BO_', []):
        # Any message named VECTOR__INDEPENDENT_SIG_MSG contains
        # signals not assigned to any message. Cantools does not yet
        # support unassigned signals. Discard them for now.
        if message[2] == 'VECTOR__INDEPENDENT_SIG_MSG':
            continue

        # Frame id.
        frame_id_dbc = int(message[1])
        frame_id = frame_id_dbc & 0x7fffffff
        is_extended_frame = bool(frame_id_dbc & 0x80000000)

        # Senders.
        senders = [_get_node_name(attributes, message[5])]

        for node in message_senders.get(frame_id_dbc, []):
            if node not in senders:
                senders.append(node)

        if senders == ['Vector__XXX']:
            senders = []

        # Signal multiplexing.
        multiplexer_signal = None

        for signal in message[6]:
            if len(signal[1]) == 2:
                if signal[1][1].endswith('M'):
                    if multiplexer_signal is None:
                        multiplexer_signal = signal[1][0]
                    else:
                        multiplexer_signal = None
                        break

        signals = _load_signals(message[6],
                                comments,
                                attributes,
                                definitions,
                                choices,
                                signal_types,
                                signal_multiplexer_values,
                                frame_id_dbc,
                                multiplexer_signal)

        messages.append(
            Message(frame_id=frame_id,
                    is_extended_frame=is_extended_frame,
                    name=get_message_name(frame_id_dbc, message[2]),
                    length=int(message[4], 0),
                    senders=senders,
                    send_type=get_send_type(frame_id_dbc),
                    cycle_time=get_cycle_time(frame_id_dbc),
                    dbc_specifics=DbcSpecifics(get_attributes(frame_id_dbc),
                                               definitions),
                    signals=signals,
                    comment=get_comment(frame_id_dbc),
                    strict=strict,
                    protocol=get_protocol(frame_id_dbc),
                    frame_format=get_frame_format(frame_id_dbc),
                    bus_name=bus_name,
                    signal_groups=get_signal_groups(frame_id_dbc)))

    return messages

def _load_version(tokens):
    return tokens.get('VERSION', [[None, None]])[0][1]

def _load_bus(attributes, comments):
    try:
        bus_name = attributes['database']['DBName'].value
    except KeyError:
        return None

    try:
        bus_baudrate = attributes['database']['Baudrate'].value
    except KeyError:
        bus_baudrate = None

    try:
        bus_comment = comments['database']['bus']
    except KeyError:
        bus_comment = None

    return Bus(bus_name, baudrate=bus_baudrate, comment=bus_comment)

def _load_nodes(tokens, comments, attributes, definitions):
    nodes = None

    for token in tokens.get('BU_', []):
        nodes = [Node(name=_get_node_name(attributes, node),
                      comment=comments.get(node, None),
                      dbc_specifics=DbcSpecifics(attributes['node'].get(node, None),
                                                 definitions))
                 for node in token[2]]

    return nodes

def get_attribute_definition(database, name, default):
    if database.dbc is None:
        database.dbc = DbcSpecifics()

    if name not in database.dbc.attribute_definitions:
        database.dbc.attribute_definitions[name] = default

    return database.dbc.attribute_definitions[name]

def get_long_node_name_attribute_definition(database):
    return get_attribute_definition(database,
                                    'SystemNodeLongSymbol',
                                    ATTRIBUTE_DEFINITION_LONG_NODE_NAME)

def get_long_message_name_attribute_definition(database):
    return get_attribute_definition(database,
                                    'SystemMessageLongSymbol',
                                    ATTRIBUTE_DEFINITION_LONG_MESSAGE_NAME)

def get_long_signal_name_attribute_definition(database):
    return get_attribute_definition(database,
                                    'SystemSignalLongSymbol',
                                    ATTRIBUTE_DEFINITION_LONG_SIGNAL_NAME)

def try_remove_attribute(dbc, name):
    try:
        dbc.attributes.pop(name)
    except (KeyError, AttributeError):
        pass

def make_node_names_unique(database):
    converter = LongNamesConverter(database)

    for node in database.nodes:
        name = converter.convert(node.name)
        try_remove_attribute(node.dbc, 'SystemNodeLongSymbol')

        if name is None:
            continue

        for message in database.messages:
            for index, sender in enumerate(message.senders):
                if sender == node.name:
                    message.senders[index] = name

            for signal in message.signals:
                for index, receiver in enumerate(signal.receivers):
                    if receiver == node.name:
                        signal.receivers[index] = name

        if node.dbc is None:
            node.dbc = DbcSpecifics()

        node.dbc.attributes['SystemNodeLongSymbol'] = Attribute(
            node.name,
            get_long_node_name_attribute_definition(database))
        node.name = name

def make_message_names_unique(database):
    converter = LongNamesConverter(database)

    for message in database.messages:
        name = converter.convert(message.name)
        try_remove_attribute(message.dbc, 'SystemMessageLongSymbol')

        if name is None:
            continue

        if message.dbc is None:
            message.dbc = DbcSpecifics()

        message.dbc.attributes['SystemMessageLongSymbol'] = Attribute(
            message.name,
            get_long_message_name_attribute_definition(database))
        message.name = name

def make_signal_names_unique(database):
    converter = LongNamesConverter(database)

    for message in database.messages:
        for signal in message.signals:
            name = converter.convert(signal.name)
            try_remove_attribute(signal.dbc, 'SystemSignalLongSymbol')

            if name is None:
                continue

            if signal.dbc is None:
                signal.dbc = DbcSpecifics()

            signal.dbc.attributes['SystemSignalLongSymbol'] = Attribute(
                signal.name,
                get_long_signal_name_attribute_definition(database))
            signal.name = name

def make_names_unique(database):
    """Make message, signal and node names unique and add attributes for
    their long names.

    """

    make_node_names_unique(database)
    make_message_names_unique(database)
    make_signal_names_unique(database)

    return database

def dump_string(database):
    """Format database in DBC file format.

    """

    # Make a deep copy of the database as names and attributes will be
    # modified for items with long names.
    database = deepcopy(database)

    database = make_names_unique(database)
    bu = _dump_nodes(database)
    val_table = _dump_value_tables(database)
    bo = _dump_messages(database)
    bo_tx_bu = _dump_senders(database)
    cm = _dump_comments(database)
    signal_types = _dump_signal_types(database)
    ba_def = _dump_attribute_definitions(database)
    ba_def_def = _dump_attribute_definition_defaults(database)
    ba = _dump_attributes(database)
    val = _dump_choices(database)
    sig_group = _dump_signal_groups(database)
    sig_mux_values = _dump_signal_mux_values(database)

    return DBC_FMT.format(version=_dump_version(database),
                          bu=' '.join(bu),
                          val_table='\r\n'.join(val_table),
                          bo='\r\n\r\n'.join(bo),
                          bo_tx_bu='\r\n'.join(bo_tx_bu),
                          cm='\r\n'.join(cm),
                          signal_types='\r\n'.join(signal_types),
                          ba_def='\r\n'.join(ba_def),
                          ba_def_def='\r\n'.join(ba_def_def),
                          ba='\r\n'.join(ba),
                          val='\r\n'.join(val),
                          sig_group='\r\n'.join(sig_group),
                          sig_mux_values='\r\n'.join(sig_mux_values))

def get_definitions_dict(definitions, defaults):
    result = odict()

    def convert_value(definition, value):
        if definition.type_name in ['INT', 'HEX']:
            value = to_int(value)
        elif definition.type_name == 'FLOAT':
            value = Decimal(value)

        return value

    for item in definitions:
        if len(item[1]) > 0:
            kind = item[1][0]
        else:
            kind = None

        definition = AttributeDefinition(name=item[2],
                                         kind=kind,
                                         type_name=item[3])
        values = item[4][0]

        if len(values) > 0:
            if definition.type_name == "ENUM":
                definition.choices = values
            elif definition.type_name in ['INT', 'FLOAT', 'HEX']:
                definition.minimum = convert_value(definition, values[0])
                definition.maximum = convert_value(definition, values[1])

        try:
            value = defaults[definition.name]
            definition.default_value = convert_value(definition, value)
        except KeyError:
            definition.default_value = None

        result[definition.name] = definition

    return result

def load_string(string, strict=True):
    """Parse given string.

    """

    tokens = Parser().parse(string)

    comments = _load_comments(tokens)
    definitions = _load_attribute_definitions(tokens)
    defaults = _load_attribute_definition_defaults(tokens)
    attribute_definitions = get_definitions_dict(definitions, defaults)
    attributes = _load_attributes(tokens, attribute_definitions)
    bus = _load_bus(attributes, comments)
    value_tables = _load_value_tables(tokens)
    choices = _load_choices(tokens)
    message_senders = _load_message_senders(tokens, attributes)
    signal_types = _load_signal_types(tokens)
    signal_multiplexer_values = _load_signal_multiplexer_values(tokens)
    signal_groups = _load_signal_groups(tokens)
    messages = _load_messages(tokens,
                              comments,
                              attributes,
                              attribute_definitions,
                              choices,
                              message_senders,
                              signal_types,
                              signal_multiplexer_values,
                              strict,
                              bus.name if bus else None,
                              signal_groups)
    nodes = _load_nodes(tokens, comments, attributes, attribute_definitions)
    version = _load_version(tokens)
    environment_variables = _load_environment_variables(tokens, comments, attributes)
    dbc_specifics = DbcSpecifics(attributes.get('database', None),
                                 attribute_definitions,
                                 environment_variables,
                                 value_tables)

    return InternalDatabase(messages,
                            nodes,
                            [bus] if bus else [],
                            version,
                            dbc_specifics)

database.py (click to expand)
import logging

from .formats import arxml
from .formats import dbc
from .formats import kcd
from .formats import sym
from .internal_database import InternalDatabase
from ...compat import fopen

LOGGER = logging.getLogger(__name__)

class Database(object):
    """This class contains all messages, signals and definitions of a CAN
    network.

    The factory functions :func:`load()<cantools.database.load()>`,
    :func:`load_file()<cantools.database.load_file()>` and
    :func:`load_string()<cantools.database.load_string()>` returns
    instances of this class.

    If `strict` is ``True`` an exception is raised if any signals are
    overlapping or if they don't fit in their message.

    """

    def __init__(self,
                 messages=None,
                 nodes=None,
                 buses=None,
                 version=None,
                 dbc_specifics=None,
                 frame_id_mask=None,
                 strict=True):
        self._messages = messages if messages else []
        self._nodes = nodes if nodes else []
        self._buses = buses if buses else []
        self._name_to_message = {}
        self._frame_id_to_message = {}
        self._version = version
        self._dbc = dbc_specifics

        if frame_id_mask is None:
            frame_id_mask = 0xffffffff

        self._frame_id_mask = frame_id_mask
        self._strict = strict
        self.refresh()

    @property
    def messages(self):
        """A list of messages in the database.

        Use :meth:`.get_message_by_frame_id()` or
        :meth:`.get_message_by_name()` to find a message by its frame
        id or name.

        """

        return self._messages

    @property
    def nodes(self):
        """A list of nodes in the database.

        """

        return self._nodes

    @property
    def buses(self):
        """A list of CAN buses in the database.

        """

        return self._buses

    @property
    def version(self):
        """The database version, or ``None`` if unavailable.

        """

        return self._version

    @version.setter
    def version(self, value):
        self._version = value

    @property
    def dbc(self):
        """An object containing dbc specific properties like e.g. attributes.

        """

        return self._dbc

    @dbc.setter
    def dbc(self, value):
        self._dbc = value

    def add_arxml(self, fp):
        """Read and parse ARXML data from given file-like object and add the
        parsed data to the database.

        """

        self.add_arxml_string(fp.read())

    def add_arxml_file(self, filename, encoding='utf-8'):
        """Open, read and parse ARXML data from given file and add the parsed
        data to the database.

        `encoding` specifies the file encoding.

        """

        with fopen(filename, 'r', encoding=encoding) as fin:
            self.add_arxml(fin)

    def add_arxml_string(self, string):
        """Parse given ARXML data string and add the parsed data to the
        database.

        """

        database = arxml.load_string(string, self._strict)

        self._messages += database.messages
        self._nodes = database.nodes
        self._buses = database.buses
        self._version = database.version
        self._dbc = database.dbc
        self.refresh()

    def add_dbc(self, fp):
        """Read and parse DBC data from given file-like object and add the
        parsed data to the database.

        >>> db = cantools.database.Database()
        >>> with open ('foo.dbc', 'r') as fin:
        ...     db.add_dbc(fin)

        """

        self.add_dbc_string(fp.read())

    def add_dbc_file(self, filename, encoding='cp1252'):
        """Open, read and parse DBC data from given file and add the parsed
        data to the database.

        `encoding` specifies the file encoding.

        >>> db = cantools.database.Database()
        >>> db.add_dbc_file('foo.dbc')

        """

        with fopen(filename, 'r', encoding=encoding) as fin:
            self.add_dbc(fin)

    def add_dbc_string(self, string):
        """Parse given DBC data string and add the parsed data to the
        database.

        >>> db = cantools.database.Database()
        >>> with open ('foo.dbc', 'r') as fin:
        ...     db.add_dbc_string(fin.read())

        """

        database = dbc.load_string(string, self._strict)

        self._messages += database.messages
        self._nodes = database.nodes
        self._buses = database.buses
        self._version = database.version
        self._dbc = database.dbc
        self.refresh()

    def add_kcd(self, fp):
        """Read and parse KCD data from given file-like object and add the
        parsed data to the database.

        """

        self.add_kcd_string(fp.read())

    def add_kcd_file(self, filename, encoding='utf-8'):
        """Open, read and parse KCD data from given file and add the parsed
        data to the database.

        `encoding` specifies the file encoding.

        """

        with fopen(filename, 'r', encoding=encoding) as fin:
            self.add_kcd(fin)

    def add_kcd_string(self, string):
        """Parse given KCD data string and add the parsed data to the
        database.

        """

        database = kcd.load_string(string, self._strict)

        self._messages += database.messages
        self._nodes = database.nodes
        self._buses = database.buses
        self._version = database.version
        self._dbc = database.dbc
        self.refresh()

    def add_sym(self, fp):
        """Read and parse SYM data from given file-like object and add the
        parsed data to the database.

        """

        self.add_sym_string(fp.read())

    def add_sym_file(self, filename, encoding='utf-8'):
        """Open, read and parse SYM data from given file and add the parsed
        data to the database.

        `encoding` specifies the file encoding.

        """

        with fopen(filename, 'r', encoding=encoding) as fin:
            self.add_sym(fin)

    def add_sym_string(self, string):
        """Parse given SYM data string and add the parsed data to the
        database.

        """

        database = sym.load_string(string, self._strict)

        self._messages += database.messages
        self._nodes = database.nodes
        self._buses = database.buses
        self._version = database.version
        self._dbc = database.dbc
        self.refresh()

    def _add_message(self, message):
        """Add given message to the database.

        """

        if message.name in self._name_to_message:
            LOGGER.warning("Overwriting message '%s' with '%s' in the "
                           "name to message dictionary.",
                           self._name_to_message[message.name].name,
                           message.name)

        masked_frame_id = (message.frame_id & self._frame_id_mask)

        if masked_frame_id in self._frame_id_to_message:
            LOGGER.warning(
                "Overwriting message '%s' with '%s' in the frame id to message "
                "dictionary because they have identical masked frame ids 0x%x.",
                self._frame_id_to_message[masked_frame_id].name,
                message.name,
                masked_frame_id)

        self._name_to_message[message.name] = message
        self._frame_id_to_message[masked_frame_id] = message

    def as_dbc_string(self):
        """Return the database as a string formatted as a DBC file.

        """

        return dbc.dump_string(InternalDatabase(self._messages,
                                                self._nodes,
                                                self._buses,
                                                self._version,
                                                self._dbc))

    def as_kcd_string(self):
        """Return the database as a string formatted as a KCD file.

        """

        return kcd.dump_string(InternalDatabase(self._messages,
                                                self._nodes,
                                                self._buses,
                                                self._version,
                                                self._dbc))

    def get_message_by_name(self, name):
        """Find the message object for given name `name`.

        """

        return self._name_to_message[name]

    def get_message_by_frame_id(self, frame_id):
        """Find the message object for given frame id `frame_id`.

        """

        if frame_id & self._frame_id_mask in self._frame_id_to_message:
            return self._frame_id_to_message[frame_id & self._frame_id_mask]

        for message in self.messages:
            if (
                'UseGMParameterIDs' in message.dbc.attributes and
                message.dbc.attributes['UseGMParameterIDs'].value == 1
            ):
                # convert to GM  Parameter Id
                id = frame_id >> 13 & 0x1FFF

                if message.frame_id == id:
                    return message

        raise KeyError(frame_id)

    def get_node_by_name(self, name):
        """Find the node object for given name `name`.

        """

        for node in self._nodes:
            if node.name == name:
                return node

        raise KeyError(name)

    def get_bus_by_name(self, name):
        """Find the bus object for given name `name`.

        """

        for bus in self._buses:
            if bus.name == name:
                return bus

        raise KeyError(name)

    def encode_message(self,
                       frame_id_or_name,
                       data,
                       scaling=True,
                       padding=False,
                       strict=True):
        """Encode given signal data `data` as a message of given frame id or
        name `frame_id_or_name`. `data` is a dictionary of signal
        name-value entries.

        If `scaling` is ``False`` no scaling of signals is performed.

        If `padding` is ``True`` unused bits are encoded as 1.

        If `strict` is ``True`` all signal values must be within their
        allowed ranges, or an exception is raised.

        >>> db.encode_message(158, {'Bar': 1, 'Fum': 5.0})
        b'\\x01\\x45\\x23\\x00\\x11'
        >>> db.encode_message('Foo', {'Bar': 1, 'Fum': 5.0})
        b'\\x01\\x45\\x23\\x00\\x11'

        """

        try:
            message = self.get_message_by_frame_id(frame_id_or_name)
        except KeyError:
            message = self.get_message_by_name(frame_id_or_name)

        return message.encode(data, scaling, padding, strict)

    def decode_message(self,
                       frame_id_or_name,
                       data,
                       decode_choices=True,
                       scaling=True):
        """Decode given signal data `data` as a message of given frame id or
        name `frame_id_or_name`. Returns a dictionary of signal
        name-value entries.

        If `decode_choices` is ``False`` scaled values are not
        converted to choice strings (if available).

        If `scaling` is ``False`` no scaling of signals is performed.

        >>> db.decode_message(158, b'\\x01\\x45\\x23\\x00\\x11')
        {'Bar': 1, 'Fum': 5.0}
        >>> db.decode_message('Foo', b'\\x01\\x45\\x23\\x00\\x11')
        {'Bar': 1, 'Fum': 5.0}

        """

        try:
            message = self.get_message_by_frame_id(frame_id_or_name)
        except KeyError:
            message = self.get_message_by_name(frame_id_or_name)

        return message.decode(data, decode_choices, scaling)

    def refresh(self):
        """Refresh the internal database state.

        This method must be called after modifying any message in the
        database to refresh the internal lookup tables used when
        encoding and decoding messages.

        """

        self._name_to_message = {}
        self._frame_id_to_message = {}

        for message in self._messages:
            message.refresh(self._strict)
            self._add_message(message)

    def __repr__(self):
        lines = []

        lines.append("version('{}')".format(self._version))
        lines.append('')

        if self._nodes:
            for node in self._nodes:
                lines.append(repr(node))

            lines.append('')

        for message in self._messages:
            lines.append(repr(message))

            for signal in message.signals:
                lines.append('  ' + repr(signal))

            lines.append('')

        return '\n'.join(lines)

I did make the changes needed to handle the writing of the file as well with respect to the attributes so if an objects attribute value matches the default value for that attribute a BA_ entry will not be written to the file for that object

If I'm right, the first two lines define a database attribute (as there's no keyword like "BO_", "BU_", "SG_".

Unfortunately Vector has the DBC file specification under tight lock and key and the only way to know is to reverse engineer it. I am doing this by making changes in the CANdb++ app saving the file and then doing a before and after file comparison to see what has changed. This is what I have put together as far as how the attributes work, I may be incorrect and if I am please correct me. I am all for an education and I do like to know how every aspect of my applications work. I don't like to blindly use external libraries in my applications.
if the attribute deceleration is an INT it would lay out like this

declare attribute
attribute name
attribute type
min
max

BA_DEF_
"UseGMParameterIDs"
INT
0
0

attribute default
attribute name
default value

BA_DEF_DEF_
"UseGMParameterIDs"
1

set attribute
attribute name
object type
object id/name
value
note

BA_
"UseGMParameterIDs"

1
no object specified so this would be a database default covering all objects

BA_
"UseGMParameterIDs"
BO_
2155175936
0
BO_ = object type message, and next to it is the id of the message we want to set the attribute on

If the value declaration is an ENUM it would lay out like this

declare attribute
attribute name
attribute type
comma separated values

BA_DEF_
"VFrameFormat"
ENUM
"StandardCAN","ExtendedCAN","StandardCAN_FD","ExtendedCAN_FD","J1939PG"

sets the default to an empty string

attribute default
attribute name
default value

BA_DEF_DEF_
"VFrameFormat"
""

setting the attribute to specific messages and the value is 0

set attribute
attribute name
object type
object id/name
value

BA_
"VFrameFormat"
BO_
2015
0

BA_
"VFrameFormat"
BO_
2015
0

the 0 value should translate to "StandardCAN" which is gotten from the attribute deceleration, the index of 0 in the enumeration is "StandardCAN"

OK here are the changes that should properly handle the GM Extended frame ids, J1939 frame ids and the proper handling of the VFrameFormat, UseGMParameterIDs and ProtocolType variables.
This is going to check if there is a message for the given frame id and if there isn't then it will check the frame format type and if the frame format type is either GM or J1939 it will modify the passed in frame id according to specification and then it will check to see if there is a message that exists with that id, and if there is it will return that messge.

message.py (click to expand)
# A CAN message.

import binascii
from copy import deepcopy

from ..utils import format_or
from ..utils import start_bit
from ..utils import encode_data
from ..utils import decode_data
from ..utils import create_encode_decode_formats
from ..errors import Error
from ..errors import EncodeError
from ..errors import DecodeError

class Message(object):
    """A CAN message with frame id, comment, signals and other
    information.

    If `strict` is ``True`` an exception is raised if any signals are
    overlapping or if they don't fit in the message.

    """

    def __init__(self,
                 frame_id,
                 name,
                 length,
                 signals,
                 comment=None,
                 senders=None,
                 send_type=None,
                 cycle_time=None,
                 dbc_specifics=None,
                 is_extended_frame=False,
                 bus_name=None,
                 signal_groups=None,
                 strict=True,
                 protocol=None,
                 frame_format=None):
        frame_id_bit_length = frame_id.bit_length()

        if is_extended_frame:
            if frame_id_bit_length > 29:
                raise Error(
                    'Extended frame id 0x{:x} is more than 29 bits in '
                    'message {}.'.format(frame_id, name))
        elif frame_id_bit_length > 11:
            raise Error(
                'Standard frame id 0x{:x} is more than 11 bits in '
                'message {}.'.format(frame_id, name))

        self._frame_id = frame_id
        self._is_extended_frame = is_extended_frame
        self._name = name
        self._length = length
        self._signals = signals
        self._signals.sort(key=start_bit)
        self._comment = comment
        self._senders = senders if senders else []
        self._send_type = send_type
        self._cycle_time = cycle_time
        self._dbc = dbc_specifics
        self._bus_name = bus_name
        self._signal_groups = signal_groups
        self._codecs = None
        self._signal_tree = None
        self._strict = strict
        self._protocol = protocol
        self._frame_format = frame_format
        self.refresh()

    def _create_codec(self, parent_signal=None, multiplexer_id=None):
        """Create a codec of all signals with given parent signal. This is a
        recursive function.

        """

        signals = []
        multiplexers = {}

        # Find all signals matching given parent signal name and given
        # multiplexer id. Root signals' parent and multiplexer id are
        # both None.
        for signal in self._signals:
            if signal.multiplexer_signal != parent_signal:
                continue

            if ((multiplexer_id is not None)
                and (multiplexer_id not in signal.multiplexer_ids)):
                continue

            if signal.is_multiplexer:
                children_ids = set()

                for s in self._signals:
                    if s.multiplexer_signal != signal.name:
                        continue

                    children_ids.update(s.multiplexer_ids)

                # Some CAN messages will have muxes containing only
                # the multiplexer and no additional signals. At Tesla
                # these are indicated in advance by assigning them an
                # enumeration. Here we ensure that any named
                # multiplexer is included, even if it has no child
                # signals.
                if signal.choices:
                    children_ids.update(signal.choices.keys())

                for child_id in children_ids:
                    codec = self._create_codec(signal.name, child_id)

                    if signal.name not in multiplexers:
                        multiplexers[signal.name] = {}

                    multiplexers[signal.name][child_id] = codec

            signals.append(signal)

        return {
            'signals': signals,
            'formats': create_encode_decode_formats(signals,
                                                    self._length),
            'multiplexers': multiplexers
        }

    def _create_signal_tree(self, codec):
        """Create a multiplexing tree node of given codec. This is a recursive
        function.

        """

        nodes = []

        for signal in codec['signals']:
            multiplexers = codec['multiplexers']

            if signal.name in multiplexers:
                node = {
                    signal.name: {
                        mux: self._create_signal_tree(mux_codec)
                        for mux, mux_codec in multiplexers[signal.name].items()
                    }
                }
            else:
                node = signal.name

            nodes.append(node)

        return nodes

    @property
    def frame_id(self):
        """The message frame id.

        """

        return self._frame_id

    @frame_id.setter
    def frame_id(self, value):
        self._frame_id = value

    @property
    def is_extended_frame(self):
        """``True`` if the message is an extended frame, ``False`` otherwise.

        """

        return self._is_extended_frame

    @is_extended_frame.setter
    def is_extended_frame(self, value):
        self._is_extended_frame = value

    @property
    def name(self):
        """The message name as a string.

        """

        return self._name

    @name.setter
    def name(self, value):
        self._name = value

    @property
    def length(self):
        """The message data length in bytes.

        """

        return self._length

    @length.setter
    def length(self, value):
        self._length = value

    @property
    def signals(self):
        """A list of all signals in the message.

        """

        return self._signals

    @property
    def signal_groups(self):
        """A list of all signal groups in the message.

        """

        return self._signal_groups

    @signal_groups.setter
    def signal_groups(self, value):
        self._signal_groups = value

    @property
    def comment(self):
        """The message comment, or ``None`` if unavailable.

        """

        return self._comment

    @comment.setter
    def comment(self, value):
        self._comment = value

    @property
    def senders(self):
        """A list of all sender nodes of this message.

        """

        return self._senders

    @property
    def send_type(self):
        """The message send type, or ``None`` if unavailable.

        """

        return self._send_type

    @property
    def cycle_time(self):
        """The message cycle time, or ``None`` if unavailable.

        """

        return self._cycle_time

    @property
    def dbc(self):
        """An object containing dbc specific properties like e.g. attributes.

        """

        return self._dbc

    @dbc.setter
    def dbc(self, value):
        self._dbc = value

    @property
    def bus_name(self):
        """The message bus name, or ``None`` if unavailable.

        """

        return self._bus_name

    @bus_name.setter
    def bus_name(self, value):
        self._bus_name = value

    @property
    def frame_format(self):
        """
        The message format, or ``None`` if unavailable.

        Only one format is currently supported; ``'j1939'``.
        """
        return self._frame_format

    @frame_format.setter
    def frame_format(self, value):
        self._frame_format = value

    @property
    def protocol(self):
        """
        The message protocol, or ``None`` if unavailable.

        This tells us the protocol being used for the
        data contained within the messge.

        some examples would be

        ``"CAN"``
        ``"OBD2"``
        ``"GMLAN"``
        """

        return self._protocol

    @protocol.setter
    def protocol(self, value):
        self._protocol = value

    @property
    def signal_tree(self):
        """All signal names and multiplexer ids as a tree. Multiplexer signals
        are dictionaries, while other signals are strings.

        >>> foo = db.get_message_by_name('Foo')
        >>> foo.signal_tree
        ['Bar', 'Fum']
        >>> bar = db.get_message_by_name('Bar')
        >>> bar.signal_tree
        [{'A': {0: ['C', 'D'], 1: ['E']}}, 'B']

        """

        return self._signal_tree

    def signal_tree_string(self):
        """Returns the message signal tree as a string.

        """

        def get_prefix(index, length):
            if index < length - 1:
                return '|   '
            else:
                return '    '

        def add_prefix(prefix, lines):
            return [prefix + line for line in lines]

        def format_mux(mux):
            signal_name, multiplexed_signals = list(mux.items())[0]
            multiplexed_signals = sorted(list(multiplexed_signals.items()))
            lines = []

            for index, multiplexed_signal in enumerate(multiplexed_signals):
                multiplexer_id, signal_names = multiplexed_signal
                lines.append('+-- {}'.format(multiplexer_id))
                lines += add_prefix(get_prefix(index, len(multiplexed_signals)),
                                    format_level_lines(signal_names))

            return '+-- {}'.format(signal_name), lines

        def format_level_lines(signal_names):
            lines = []

            for index, signal_name in enumerate(signal_names):
                if isinstance(signal_name, dict):
                    signal_name_line, signal_lines = format_mux(signal_name)
                    signal_lines = add_prefix(get_prefix(index, len(signal_names)),
                                              signal_lines)
                else:
                    signal_name_line = '+-- {}'.format(signal_name)
                    signal_lines = []

                lines.append(signal_name_line)
                lines += signal_lines

            return lines

        lines = format_level_lines(self.signal_tree)
        lines = ['-- {root}'] + add_prefix('   ', lines)

        return '\n'.join(lines)

    def signal_choices_string(self):
        """Returns the signal choices as a string.

        """

        lines = []

        for signal in self._signals:
            if signal.choices:
                lines.append('')
                lines.append(signal.name)

                for value, text in sorted(signal.choices.items()):
                    lines.append('    {} {}'.format(value, text))

        return '\n'.join(lines)

    def layout_string(self, signal_names=True):
        """Returns the message layout as an ASCII art string. Each signal is
        an arrow from LSB ``x`` to MSB ``<``. Overlapping signal bits
        are set to ``X``.

        Set `signal_names` to ``False`` to hide signal names.

        .. code:: text

                                Bit

                   7   6   5   4   3   2   1   0
                 +---+---+---+---+---+---+---+---+
               0 |   |   |   |   |   |<----------|
                 +---+---+---+---+---+---+---+---+
               1 |------x|   |   |   |   |<-x|   |
                 +---+---+---+---+---+---+---+---+
                       |                   +-- Bar
                       +-- Foo
                 +---+---+---+---+---+---+---+---+
               2 |   |   |   |   |   |   |   |   |
           B     +---+---+---+---+---+---+---+---+
           y   3 |----XXXXXXX---x|   |   |   |   |
           t     +---+---+---+---+---+---+---+---+
           e           +-- Fie
                 +---+---+---+---+---+---+---+---+
               4 |-------------------------------|
                 +---+---+---+---+---+---+---+---+
               5 |   |   |<----------------------|
                 +---+---+---+---+---+---+---+---+
                           +-- Fum
                 +---+---+---+---+---+---+---+---+
               6 |   |   |   |   |   |   |   |   |
                 +---+---+---+---+---+---+---+---+
               7 |   |   |   |   |   |   |   |   |
                 +---+---+---+---+---+---+---+---+

        """

        def format_big():
            signals = []

            for signal in self._signals:
                if signal.byte_order != 'big_endian':
                    continue

                formatted = start_bit(signal) * '   '
                formatted += '<{}x'.format((3 * signal.length - 2) * '-')
                signals.append(formatted)

            return signals

        def format_little():
            signals = []

            for signal in self._signals:
                if signal.byte_order != 'little_endian':
                    continue

                formatted = signal.start * '   '
                formatted += 'x{}<'.format((3 * signal.length - 2) * '-')
                end = signal.start + signal.length

                if end % 8 != 0:
                    formatted += (8 - (end % 8)) * '   '

                formatted = ''.join([
                    formatted[i:i + 24][::-1]
                    for i in range(0, len(formatted), 24)
                ])
                signals.append(formatted)

            return signals

        def format_byte_lines():
            # Signal lines.
            signals = format_big() + format_little()

            if len(signals) > 0:
                length = max([len(signal) for signal in signals])

                if length % 24 != 0:
                    length += (24 - (length % 24))

                signals = [signal + (length - len(signal)) * ' ' for signal in signals]

            # Signals union line.
            signals_union = ''

            for chars in zip(*signals):
                head = chars.count('<')
                dash = chars.count('-')
                tail = chars.count('x')

                if head + dash + tail > 1:
                    signals_union += 'X'
                elif head == 1:
                    signals_union += '<'
                elif dash == 1:
                    signals_union += '-'
                elif tail == 1:
                    signals_union += 'x'
                else:
                    signals_union += ' '

            # Split the signals union line into byte lines, 8 bits per
            # line.
            byte_lines = [
                signals_union[i:i + 24]
                for i in range(0, len(signals_union), 24)
            ]

            unused_byte_lines = (self._length - len(byte_lines))

            if unused_byte_lines > 0:
                byte_lines += unused_byte_lines * [24 * ' ']

            # Insert bits separators into each byte line.
            lines = []

            for byte_line in byte_lines:
                line = ''
                prev_byte = None

                for i in range(0, 24, 3):
                    byte_triple = byte_line[i:i + 3]

                    if i == 0:
                        line += '|'
                    elif byte_triple[0] in ' <>x':
                        line += '|'
                    elif byte_triple[0] == 'X':
                        if prev_byte == 'X':
                            line += 'X'
                        elif prev_byte == '-':
                            line += '-'
                        else:
                            line += '|'
                    else:
                        line += '-'

                    line += byte_triple
                    prev_byte = byte_triple[2]

                line += '|'
                lines.append(line)

            # Add byte numbering.
            number_width = len(str(len(lines))) + 4
            number_fmt = '{{:{}d}} {{}}'.format(number_width - 1)
            a = []

            for number, line in enumerate(lines):
                a.append(number_fmt.format(number, line))

            return a, len(lines), number_width

        def add_header_lines(lines, number_width):
            padding = number_width * ' '

            return [
                padding + '               Bit',
                padding + '',
                padding + '  7   6   5   4   3   2   1   0',
                padding + '+---+---+---+---+---+---+---+---+'
            ] + lines

        def add_horizontal_lines(byte_lines, number_width):
            padding = number_width * ' '
            lines = []

            for byte_line in byte_lines:
                lines.append(byte_line)
                lines.append(padding + '+---+---+---+---+---+---+---+---+')

            return lines

        def name_bit(signal):
            offset = start_bit(signal) + signal.length - 1

            if signal.byte_order == 'big_endian':
                return (8 * (offset // 8) + (7 - (offset % 8)))
            else:
                return offset

        def add_signal_names(input_lines,
                             number_of_bytes,
                             number_width):
            # Find MSB and name of all signals.
            padding = number_width * ' '
            signals_per_byte = [[] for _ in range(number_of_bytes)]

            for signal in self._signals:
                byte, bit = divmod(name_bit(signal), 8)
                signals_per_byte[byte].append((bit, '+-- ' + signal.name))

            # Format signal lines.
            signal_lines_per_byte = []

            for signals in signals_per_byte:
                signals = sorted(signals)
                signals_lines = []

                for signal in signals:
                    line = number_width * ' ' + '  ' + signal[1]
                    line = (7 - signal[0]) * '    ' + line
                    chars = [char for char in line]

                    for other_signal in signals:
                        if other_signal[0] > signal[0]:
                            other_signal_msb = (number_width
                                                + 2
                                                + 4 * (7 - other_signal[0]))
                            chars[other_signal_msb] = '|'

                    signals_lines.append(''.join(chars))

                signal_lines_per_byte.append(signals_lines)

            # Insert the signals names lines among other lines.
            lines = []

            for number in range(number_of_bytes):
                lines += input_lines[2 * number: 2 * number + 2]

                if signal_lines_per_byte[number]:
                    lines += signal_lines_per_byte[number]

                    if number + 1 < number_of_bytes:
                        lines.append(
                            padding + '+---+---+---+---+---+---+---+---+')

            return lines

        def add_y_axis_name(lines):
            number_of_matrix_lines = (len(lines) - 3)

            if number_of_matrix_lines < 5:
                lines += (5 - number_of_matrix_lines) * ['     ']

            start_index = 4 + ((number_of_matrix_lines - 4) // 2 - 1)

            if start_index < 4:
                start_index = 4

            axis_lines = start_index * ['  ']
            axis_lines += [' B', ' y', ' t', ' e']
            axis_lines += (len(lines) - start_index - 4) * ['  ']

            return [
                axis_line + line
                for axis_line, line in zip(axis_lines, lines)
            ]

        lines, number_of_bytes, number_width = format_byte_lines()
        lines = add_horizontal_lines(lines, number_width)

        if signal_names:
            lines = add_signal_names(lines,
                                     number_of_bytes,
                                     number_width)

        lines = add_header_lines(lines, number_width)
        lines = add_y_axis_name(lines)
        lines = [line.rstrip() for line in lines]

        return '\n'.join(lines)

    def _get_mux_number(self, decoded, signal_name):
        mux = decoded[signal_name]

        if isinstance(mux, str):
            signal = self.get_signal_by_name(signal_name)
            mux = signal.choice_string_to_number(mux)

        return mux

    def _check_signals_ranges_scaling(self, signals, data):
        for signal in signals:
            value = data[signal.name]

            # Choices are checked later.
            if isinstance(value, str):
                continue

            if signal.minimum is not None:
                if value < signal.minimum:
                    raise EncodeError(
                        "Expected signal '{}' value greater than or equal to "
                        "{} in message '{}', but got {}.".format(signal.name,
                                                                 signal.minimum,
                                                                 self._name,
                                                                 value))

            if signal.maximum is not None:
                if value > signal.maximum:
                    raise EncodeError(
                        "Expected signal '{}' value less than or equal to "
                        "{} in message '{}', but got {}.".format(signal.name,
                                                                 signal.maximum,
                                                                 self.name,
                                                                 value))

    def _check_signals(self, signals, data, scaling):
        for signal in signals:
            if signal.name not in data:
                if signal.initial is None:
                    raise EncodeError(
                        "Expected signal value for '{}' in data, but got {}.".format(
                            signal.name,
                            data))
                data[signal.name] = signal.initial

        if scaling:
            self._check_signals_ranges_scaling(signals, data)

    def _encode(self, node, data, scaling, strict):
        if strict:
            self._check_signals(node['signals'], data, scaling)

        encoded = encode_data(data,
                              node['signals'],
                              node['formats'],
                              scaling)
        padding_mask = node['formats'].padding_mask
        multiplexers = node['multiplexers']

        for signal in multiplexers:
            mux = self._get_mux_number(data, signal)

            try:
                node = multiplexers[signal][mux]
            except KeyError:
                raise EncodeError('expected multiplexer id {}, but got {}'.format(
                    format_or(multiplexers[signal]),
                    mux))

            mux_encoded, mux_padding_mask = self._encode(node,
                                                         data,
                                                         scaling,
                                                         strict)
            encoded |= mux_encoded
            padding_mask &= mux_padding_mask

        return encoded, padding_mask

    def encode(self, data, scaling=True, padding=False, strict=True):
        """Encode given data as a message of this type.

        If `scaling` is ``False`` no scaling of signals is performed.

        If `padding` is ``True`` unused bits are encoded as 1.

        If `strict` is ``True`` all signal values must be within their
        allowed ranges, or an exception is raised.

        >>> foo = db.get_message_by_name('Foo')
        >>> foo.encode({'Bar': 1, 'Fum': 5.0})
        b'\\x01\\x45\\x23\\x00\\x11'

        """

        encoded, padding_mask = self._encode(self._codecs,
                                             data,
                                             scaling,
                                             strict)

        if padding:
            encoded |= padding_mask

        encoded |= (0x80 << (8 * self._length))
        encoded = hex(encoded)[4:].rstrip('L')

        return binascii.unhexlify(encoded)[:self._length]

    def _decode(self, node, data, decode_choices, scaling):
        decoded = decode_data(data,
                              node['signals'],
                              node['formats'],
                              decode_choices,
                              scaling)

        multiplexers = node['multiplexers']

        for signal in multiplexers:
            mux = self._get_mux_number(decoded, signal)

            try:
                node = multiplexers[signal][mux]
            except KeyError:
                raise DecodeError('expected multiplexer id {}, but got {}'.format(
                    format_or(multiplexers[signal]),
                    mux))

            decoded.update(self._decode(node,
                                        data,
                                        decode_choices,
                                        scaling))

        return decoded

    def decode(self, data, decode_choices=True, scaling=True):
        """Decode given data as a message of this type.

        If `decode_choices` is ``False`` scaled values are not
        converted to choice strings (if available).

        If `scaling` is ``False`` no scaling of signals is performed.

        >>> foo = db.get_message_by_name('Foo')
        >>> foo.decode(b'\\x01\\x45\\x23\\x00\\x11')
        {'Bar': 1, 'Fum': 5.0}

        """

        data = data[:self._length]

        return self._decode(self._codecs, data, decode_choices, scaling)

    def get_signal_by_name(self, name):
        for signal in self._signals:
            if signal.name == name:
                return signal

        raise KeyError(name)

    def is_multiplexed(self):
        """Returns ``True`` if the message is multiplexed, otherwise
        ``False``.

        >>> foo = db.get_message_by_name('Foo')
        >>> foo.is_multiplexed()
        False
        >>> bar = db.get_message_by_name('Bar')
        >>> bar.is_multiplexed()
        True

        """

        return bool(self._codecs['multiplexers'])

    def _check_signal(self, message_bits, signal):
        signal_bits = signal.length * [signal.name]

        if signal.byte_order == 'big_endian':
            padding = start_bit(signal) * [None]
            signal_bits = padding + signal_bits
        else:
            signal_bits += signal.start * [None]

            if len(signal_bits) < len(message_bits):
                padding = (len(message_bits) - len(signal_bits)) * [None]
                reversed_signal_bits = padding + signal_bits
            else:
                reversed_signal_bits = signal_bits

            signal_bits = []

            for i in range(0, len(reversed_signal_bits), 8):
                signal_bits = reversed_signal_bits[i:i + 8] + signal_bits

        # Check that the signal fits in the message.
        if len(signal_bits) > len(message_bits):
            raise Error(
                'The signal {} does not fit in message {}.'.format(
                    signal.name,
                    self.name))

        # Check that the signal does not overlap with other
        # signals.
        for offset, signal_bit in enumerate(signal_bits):
            if signal_bit is not None:
                if message_bits[offset] is not None:
                    if isinstance(message_bits[offset], tuple):
                        if signal.name in message_bits[offset]:
                            raise Error(
                                'The signals {} and {} are overlapping in message {}.'.format(
                                    signal.name,
                                    message_bits[offset],
                                    self.name))
                        else:
                            message_bits[offset] += (signal.name,)

                    elif message_bits[offset] == signal.name:
                        raise Error(
                            'The signals {} and {} are overlapping in message {}.'.format(
                                signal.name,
                                message_bits[offset],
                                self.name))
                    else:
                        message_bits[offset] = (message_bits[offset], signal.name)

                else:
                    message_bits[offset] = (signal.name,)

    def _check_mux(self, message_bits, mux):
        signal_name, children = list(mux.items())[0]
        self._check_signal(message_bits,
                           self.get_signal_by_name(signal_name))
        children_message_bits = deepcopy(message_bits)

        for multiplexer_id in sorted(children):
            child_tree = children[multiplexer_id]
            child_message_bits = deepcopy(children_message_bits)
            self._check_signal_tree(child_message_bits, child_tree)

            for i, child_bit in enumerate(child_message_bits):
                if child_bit is not None:
                    message_bits[i] = child_bit

    def _check_signal_tree(self, message_bits, signal_tree):
        for signal_name in signal_tree:
            if isinstance(signal_name, dict):
                self._check_mux(message_bits, signal_name)
            else:
                self._check_signal(message_bits,
                                   self.get_signal_by_name(signal_name))

    def _check_signal_lengths(self):
        for signal in self._signals:
            if signal.length <= 0:
                raise Error(
                    'The signal {} length {} is not greater than 0 in '
                    'message {}.'.format(
                        signal.name,
                        signal.length,
                        self.name))

    def refresh(self, strict=None):
        """Refresh the internal message state.

        If `strict` is ``True`` an exception is raised if any signals
        are overlapping or if they don't fit in the message. This
        argument overrides the value of the same argument passed to
        the constructor.

        """

        self._check_signal_lengths()
        self._codecs = self._create_codec()
        self._signal_tree = self._create_signal_tree(self._codecs)

        if strict is None:
            strict = self._strict

        if strict:
            message_bits = 8 * self.length * [None]
            self._check_signal_tree(message_bits, self.signal_tree)

    def __repr__(self):
        return "message('{}', 0x{:x}, {}, {}, {})".format(
            self._name,
            self._frame_id,
            self._is_extended_frame,
            self._length,
            "'" + self._comment + "'" if self._comment is not None else None)

dbc.py (click to expand)
# Load and dump a CAN database in DBC format.

import re
from collections import OrderedDict as odict
from collections import defaultdict
from decimal import Decimal
from copy import deepcopy

import textparser
from textparser import Sequence
from textparser import choice
from textparser import ZeroOrMore
from textparser import OneOrMore
from textparser import OneOrMoreDict
from textparser import DelimitedList
from textparser import Any
from textparser import AnyUntil
from textparser import tokenize_init
from textparser import Token
from textparser import TokenizeError
from textparser import Optional

from ..attribute_definition import AttributeDefinition
from ..attribute import Attribute
from ..signal import Signal
from ..signal import Decimal as SignalDecimal
from ..signal_group import SignalGroup
from ..message import Message
from ..node import Node
from ..bus import Bus
from ..internal_database import InternalDatabase
from ..environment_variable import EnvironmentVariable

from .utils import num

DBC_FMT = (
    'VERSION "{version}"\r\n'
    '\r\n'
    '\r\n'
    'NS_ : \r\n'
    '\tNS_DESC_\r\n'
    '\tCM_\r\n'
    '\tBA_DEF_\r\n'
    '\tBA_\r\n'
    '\tVAL_\r\n'
    '\tCAT_DEF_\r\n'
    '\tCAT_\r\n'
    '\tFILTER\r\n'
    '\tBA_DEF_DEF_\r\n'
    '\tEV_DATA_\r\n'
    '\tENVVAR_DATA_\r\n'
    '\tSGTYPE_\r\n'
    '\tSGTYPE_VAL_\r\n'
    '\tBA_DEF_SGTYPE_\r\n'
    '\tBA_SGTYPE_\r\n'
    '\tSIG_TYPE_REF_\r\n'
    '\tVAL_TABLE_\r\n'
    '\tSIG_GROUP_\r\n'
    '\tSIG_VALTYPE_\r\n'
    '\tSIGTYPE_VALTYPE_\r\n'
    '\tBO_TX_BU_\r\n'
    '\tBA_DEF_REL_\r\n'
    '\tBA_REL_\r\n'
    '\tBA_DEF_DEF_REL_\r\n'
    '\tBU_SG_REL_\r\n'
    '\tBU_EV_REL_\r\n'
    '\tBU_BO_REL_\r\n'
    '\tSG_MUL_VAL_\r\n'
    '\r\n'
    'BS_:\r\n'
    '\r\n'
    'BU_: {bu}\r\n'
    '{val_table}'
    '\r\n'
    '\r\n'
    '{bo}\r\n'
    '\r\n'
    '{bo_tx_bu}\r\n'
    '\r\n'
    '\r\n'
    '{cm}\r\n'
    '{ba_def}\r\n'
    '{ba_def_def}\r\n'
    '{ba}\r\n'
    '{val}\r\n'
    '{signal_types}\r\n'
    '{sig_group}\r\n'
    '{sig_mux_values}\r\n'
)

# Signal types.
SIGNAL_TYPE_FLOAT = 1
SIGNAL_TYPE_DOUBLE = 2

FLOAT_SIGNAL_TYPES = [
    SIGNAL_TYPE_FLOAT,
    SIGNAL_TYPE_DOUBLE
]

FLOAT_LENGTH_TO_SIGNAL_TYPE = {
    32: SIGNAL_TYPE_FLOAT,
    64: SIGNAL_TYPE_DOUBLE
}

ATTRIBUTE_DEFINITION_LONG_NODE_NAME = AttributeDefinition(
    'SystemNodeLongSymbol',
    default_value='',
    kind='BU_',
    type_name='STRING')

ATTRIBUTE_DEFINITION_LONG_MESSAGE_NAME = AttributeDefinition(
    'SystemMessageLongSymbol',
    default_value='',
    kind='BO_',
    type_name='STRING')

ATTRIBUTE_DEFINITION_LONG_SIGNAL_NAME = AttributeDefinition(
    'SystemSignalLongSymbol',
    default_value='',
    kind='SG_',
    type_name='STRING')

def to_int(value):
    return int(Decimal(value))

class Parser(textparser.Parser):

    def tokenize(self, string):
        keywords = set([
            'BA_',
            'BA_DEF_',
            'BA_DEF_DEF_',
            'BA_DEF_DEF_REL_',
            'BA_DEF_REL_',
            'BA_DEF_SGTYPE_',
            'BA_REL_',
            'BA_SGTYPE_',
            'BO_',
            'BO_TX_BU_',
            'BS_',
            'BU_',
            'BU_BO_REL_',
            'BU_EV_REL_',
            'BU_SG_REL_',
            'CAT_',
            'CAT_DEF_',
            'CM_',
            'ENVVAR_DATA_',
            'EV_',
            'EV_DATA_',
            'FILTER',
            'NS_',
            'NS_DESC_',
            'SG_',
            'SG_MUL_VAL_',
            'SGTYPE_',
            'SGTYPE_VAL_',
            'SIG_GROUP_',
            'SIG_TYPE_REF_',
            'SIG_VALTYPE_',
            'SIGTYPE_VALTYPE_',
            'VAL_',
            'VAL_TABLE_',
            'VERSION'
        ])

        names = {
            'LPAREN': '(',
            'RPAREN': ')',
            'LBRACE': '[',
            'RBRACE': ']',
            'COMMA':  ',',
            'AT':     '@',
            'SCOLON': ';',
            'COLON':  ':',
            'PIPE':   '|',
            'SIGN':   '+/-'
        }

        token_specs = [
            ('SKIP',     r'[ \r\n\t]+|//.*?\n'),
            ('NUMBER',   r'[-+]?\d+\.?\d*([eE][+-]?\d+)?'),
            ('WORD',     r'[A-Za-z0-9_]+'),
            ('STRING',   r'"(\\"|[^"])*?"'),
            ('LPAREN',   r'\('),
            ('RPAREN',   r'\)'),
            ('LBRACE',   r'\['),
            ('RBRACE',   r'\]'),
            ('COMMA',    r','),
            ('PIPE',     r'\|'),
            ('AT',       r'@'),
            ('SIGN',     r'[+-]'),
            ('SCOLON',   r';'),
            ('COLON',    r':'),
            ('MISMATCH', r'.')
        ]

        tokens, token_regex = tokenize_init(token_specs)

        for mo in re.finditer(token_regex, string, re.DOTALL):
            kind = mo.lastgroup

            if kind == 'SKIP':
                pass
            elif kind == 'STRING':
                value = mo.group(kind)[1:-1].replace('\\"', '"')
                tokens.append(Token(kind, value, mo.start()))
            elif kind != 'MISMATCH':
                value = mo.group(kind)

                if value in keywords:
                    kind = value

                if kind in names:
                    kind = names[kind]

                tokens.append(Token(kind, value, mo.start()))
            else:
                raise TokenizeError(string, mo.start())

        return tokens

    def grammar(self):
        version = Sequence('VERSION', 'STRING')

        ns = Sequence('NS_', ':', AnyUntil(Sequence(Any(), ':')))

        bs = Sequence('BS_', ':')

        nodes = Sequence('BU_', ':', ZeroOrMore('WORD'))

        signal = Sequence(
            'SG_', choice(Sequence('WORD', 'WORD'), Sequence('WORD')), ':',
            'NUMBER', '|', 'NUMBER', '@', 'NUMBER', '+/-',
            '(', 'NUMBER', ',', 'NUMBER', ')',
            '[', 'NUMBER', '|', 'NUMBER', ']',
            'STRING',
            DelimitedList('WORD'))

        message = Sequence(
            'BO_', 'NUMBER', 'WORD', ':', 'NUMBER', 'WORD', ZeroOrMore(signal))

        environment_variable = Sequence(
            'EV_', 'WORD', ':', 'NUMBER',
            '[', 'NUMBER', '|', 'NUMBER', ']',
            'STRING', 'NUMBER', 'NUMBER', 'WORD', 'WORD', ';')

        comment = Sequence(
            'CM_',
            choice(
                Sequence('SG_', 'NUMBER', 'WORD', 'STRING'),
                Sequence('BO_', 'NUMBER', 'STRING'),
                Sequence('EV_', 'WORD', 'STRING'),
                Sequence('BU_', 'WORD', 'STRING'),
                'STRING'),
            ';')

        attribute_definition = Sequence(
            'BA_DEF_',
            Optional(choice('SG_', 'BO_', 'EV_', 'BU_')),
            'STRING',
            'WORD',
            Optional(choice(DelimitedList('STRING'), ZeroOrMore('NUMBER'))),
            ';')

        attribute_definition_default = Sequence(
            'BA_DEF_DEF_', 'STRING', choice('NUMBER', 'STRING'), ';')

        attribute = Sequence(
            'BA_', 'STRING',
            ZeroOrMore(choice(Sequence('BO_', 'NUMBER'),
                              Sequence('SG_', 'NUMBER', 'WORD'),
                              Sequence('BU_', 'WORD'),
                              Sequence('EV_', 'WORD'))),
            choice('NUMBER', 'STRING'),
            ';')

        attribute_definition_rel = Sequence(
            'BA_DEF_REL_',
            Optional('BU_SG_REL_'),
            'STRING',
            'WORD',
            choice(DelimitedList('STRING'), OneOrMore('NUMBER')),
            ';')

        attribute_definition_default_rel = Sequence(
            'BA_DEF_DEF_REL_', 'STRING', choice('NUMBER', 'STRING'), ';')

        attribute_rel = Sequence(
            'BA_REL_', 'STRING', 'BU_SG_REL_', 'WORD', 'SG_', 'NUMBER',
            'WORD', choice('NUMBER', 'STRING'), ';')

        choice_ = Sequence(
            'VAL_',
            Optional('NUMBER'),
            'WORD',
            ZeroOrMore(Sequence('NUMBER', 'STRING')),
            ';')

        value_table = Sequence(
            'VAL_TABLE_', 'WORD', ZeroOrMore(Sequence('NUMBER', 'STRING')), ';')

        signal_type = Sequence(
            'SIG_VALTYPE_', 'NUMBER', 'WORD', ':', 'NUMBER', ';')

        signal_multiplexer_values = Sequence(
            'SG_MUL_VAL_',
            'NUMBER',
            'WORD',
            'WORD',
            DelimitedList(Sequence('NUMBER', 'NUMBER')),
            ';')

        message_add_sender = Sequence(
            'BO_TX_BU_', 'NUMBER', ':', DelimitedList('WORD'), ';')

        signal_group = Sequence(
            'SIG_GROUP_', 'NUMBER', 'WORD', 'NUMBER', ':', ZeroOrMore('WORD'), ';')

        return OneOrMoreDict(
            choice(
                message,
                comment,
                attribute_definition,
                value_table,
                choice_,
                attribute,
                attribute_rel,
                attribute_definition_rel,
                attribute_definition_default,
                attribute_definition_default_rel,
                signal_group,
                signal_type,
                signal_multiplexer_values,
                message_add_sender,
                environment_variable,
                nodes,
                ns,
                bs,
                version))

class DbcSpecifics(object):

    def __init__(self,
                 attributes=None,
                 attribute_definitions=None,
                 environment_variables=None,
                 value_tables=None):
        if attributes is None:
            attributes = odict()

        if attribute_definitions is None:
            attribute_definitions = odict()

        if environment_variables is None:
            environment_variables = odict()

        if value_tables is None:
            value_tables = odict()

        self._attributes = attributes
        self._attribute_definitions = attribute_definitions
        self._environment_variables = environment_variables
        self._value_tables = value_tables

    @property
    def attributes(self):
        """The DBC specific attributes of the parent object (database, node,
        message or signal) as a dictionary.

        """

        return self._attributes

    @attributes.setter
    def attributes(self, value):
        self._attributes = value

    @property
    def attribute_definitions(self):
        """The DBC specific attribute definitions as dictionary.

        """

        return self._attribute_definitions

    @property
    def value_tables(self):
        """An ordered dictionary of all value tables. Only valid for DBC
        specifiers on database level.

        """

        return self._value_tables

    @property
    def environment_variables(self):
        """An ordered dictionary of all environment variables. Only valid for
        DBC specifiers on database level.

        """

        return self._environment_variables

class LongNamesConverter(object):

    def __init__(self, database):
        self._database = database
        self._next_index_per_cut_name = defaultdict(int)
        self._short_names = set()

    def convert(self, name):
        short_name = None

        if len(name) == 32:
            self._short_names.add(name)
        elif len(name) > 32:
            cut_name = name[:27]
            short_name = name[:32]

            if short_name in self._short_names:
                index = self._next_index_per_cut_name[cut_name]
                self._next_index_per_cut_name[cut_name] += 1
                short_name = '{}_{:04d}'.format(name[:27], index)
            else:
                self._next_index_per_cut_name[cut_name] = 0
                self._short_names.add(short_name)

        return short_name

def get_dbc_frame_id(message):
    frame_id = message.frame_id

    if message.is_extended_frame:
        frame_id |= 0x80000000

    return frame_id

def _get_node_name(attributes, name):
    try:
        return attributes['node'][name]['SystemNodeLongSymbol'].value
    except (KeyError, TypeError):
        return name

def _get_environment_variable_name(attributes, name):
    try:
        return attributes['envvar'][name]['SystemEnvVarLongSymbol'].value
    except (KeyError, TypeError):
        return name

def _dump_version(database):
    return '' if database.version is None else database.version

def _dump_nodes(database):
    bu = []

    for node in database.nodes:
        bu.append(node.name)

    return bu

def _dump_value_tables(database):
    if database.dbc is None:
        return []

    val_table = []

    for name, choices in database.dbc.value_tables.items():
        choices = [
            '{} "{}"'.format(number, text)
            for number, text in reversed(sorted(choices.items()))
        ]
        val_table.append('VAL_TABLE_ {} {} ;'.format(name, ' '.join(choices)))

    return val_table + ['']

def _dump_messages(database):
    bo = []

    def format_mux(signal):
        if signal.is_multiplexer:
            return ' M'
        elif signal.multiplexer_ids is not None:
            return ' m{}'.format(signal.multiplexer_ids[0])
        else:
            return ''

    def format_receivers(signal):
        if signal.receivers:
            return ' ' + ','.join(signal.receivers)
        else:
            return 'Vector__XXX'

    def format_senders(message):
        if message.senders:
            return message.senders[0]
        else:
            return 'Vector__XXX'

    for message in database.messages:
        msg = []
        msg.append(
            'BO_ {frame_id} {name}: {length} {senders}'.format(
                frame_id=get_dbc_frame_id(message),
                name=message.name,
                length=message.length,
                senders=format_senders(message)))

        for signal in message.signals[::-1]:
            fmt = (' SG_ {name}{mux} : {start}|{length}@{byte_order}{sign}'
                   ' ({scale},{offset})'
                   ' [{minimum}|{maximum}] "{unit}" {receivers}')
            msg.append(fmt.format(
                name=signal.name,
                mux=format_mux(signal),
                start=signal.start,
                length=signal.length,
                receivers=format_receivers(signal),
                byte_order=(0 if signal.byte_order == 'big_endian' else 1),
                sign=('-' if signal.is_signed else '+'),
                scale=signal.scale,
                offset=signal.offset,
                minimum=(0 if signal.minimum is None else signal.minimum),
                maximum=(0 if signal.maximum is None else signal.maximum),
                unit='' if signal.unit is None else signal.unit))

        bo.append('\r\n'.join(msg))

    return bo

def _dump_senders(database):
    bo_tx_bu = []

    for message in database.messages:
        if len(message.senders) > 1:
            bo_tx_bu.append(
                'BO_TX_BU_ {frame_id} : {senders};'.format(
                    frame_id=get_dbc_frame_id(message),
                    senders=','.join(message.senders)))

    return bo_tx_bu

def _dump_comments(database):
    cm = []

    for bus in database.buses:
        if bus.comment is not None:
            cm.append(
                'CM_ "{comment}";'.format(
                    comment=bus.comment) )

    for node in database.nodes:
        if node.comment is not None:
            cm.append(
                'CM_ BU_ {name} "{comment}";'.format(
                    name=node.name,
                    comment=node.comment.replace('"', '\\"')))

    for message in database.messages:
        if message.comment is not None:
            cm.append(
                'CM_ BO_ {frame_id} "{comment}";'.format(
                    frame_id=get_dbc_frame_id(message),
                    comment=message.comment.replace('"', '\\"')))

        for signal in message.signals[::-1]:
            if signal.comment is not None:
                cm.append(
                    'CM_ SG_ {frame_id} {name} "{comment}";'.format(
                        frame_id=get_dbc_frame_id(message),
                        name=signal.name,
                        comment=signal.comment.replace('"', '\\"')))

    return cm

def _dump_signal_types(database):
    valtype = []

    for message in database.messages:
        for signal in message.signals:
            if not signal.is_float:
                continue

            valtype.append(
                'SIG_VALTYPE_ {} {} : {};'.format(
                    get_dbc_frame_id(message),
                    signal.name,
                    FLOAT_LENGTH_TO_SIGNAL_TYPE[signal.length]))

    return valtype

def _dump_attribute_definitions(database):
    ba_def = []

    if database.dbc is None:
        return ba_def

    definitions = database.dbc.attribute_definitions

    def get_value(definition, value):
        if definition.minimum is None:
            value = ''
        else:
            value = ' {}'.format(value)

        return value

    def get_minimum(definition):
        return get_value(definition, definition.minimum)

    def get_maximum(definition):
        return get_value(definition, definition.maximum)

    def get_kind(definition):
        return '' if definition.kind is None else definition.kind + ' '

    for definition in definitions.values():
        if definition.type_name == 'ENUM':
            choices = ','.join(['"{}"'.format(choice)
                                for choice in definition.choices])
            ba_def.append(
                'BA_DEF_ {kind} "{name}" {type_name}  {choices};'.format(
                    kind=get_kind(definition),
                    name=definition.name,
                    type_name=definition.type_name,
                    choices=choices))
        elif definition.type_name in ['INT', 'FLOAT', 'HEX']:
            ba_def.append(
                'BA_DEF_ {kind} "{name}" {type_name}{minimum}{maximum};'.format(
                    kind=get_kind(definition),
                    name=definition.name,
                    type_name=definition.type_name,
                    minimum=get_minimum(definition),
                    maximum=get_maximum(definition)))
        elif definition.type_name == 'STRING':
            ba_def.append(
                'BA_DEF_ {kind} "{name}" {type_name} ;'.format(
                    kind=get_kind(definition),
                    name=definition.name,
                    type_name=definition.type_name))

    return ba_def

def _dump_attribute_definition_defaults(database):
    ba_def_def = []

    if database.dbc is None:
        return ba_def_def

    definitions = database.dbc.attribute_definitions

    for definition in definitions.values():
        if definition.default_value is not None:
            if definition.type_name in ["STRING", "ENUM"]:
                fmt = 'BA_DEF_DEF_  "{name}" "{value}";'
            else:
                fmt = 'BA_DEF_DEF_  "{name}" {value};'

            ba_def_def.append(fmt.format(name=definition.name,
                                         value=definition.default_value))

    return ba_def_def

def _dump_attributes(database):
    ba = []

    def get_value(attribute):
        result = attribute.value

        if attribute.definition.type_name == "STRING":
            result = '"' + attribute.value + '"'

        return result

    if database.dbc is not None:
        if database.dbc.attributes is not None:
            for attribute in database.dbc.attributes.values():
                ba.append(
                    'BA_ "{name}" {value};'.format(name=attribute.definition.name,
                                                    value=get_value(attribute)))

    for node in database.nodes:
        if node.dbc is not None:
            if node.dbc.attributes is not None:
                for attribute in node.dbc.attributes.values():
                    if attribute.value == attribute.definition.default_value:
                        continue

                    ba.append(
                        'BA_ "{name}" {kind} {node_name} {value};'.format(
                            name=attribute.definition.name,
                            kind=attribute.definition.kind,
                            node_name=node.name,
                            value=get_value(attribute)))

    for message in database.messages:
        if message.dbc is not None:
            if message.dbc.attributes is not None:
                for attribute in message.dbc.attributes.values():
                    if attribute.value == attribute.definition.default_value:
                        continue

                    ba.append(
                        'BA_ "{name}" {kind} {frame_id} {value};'.format(
                            name=attribute.definition.name,
                            kind=attribute.definition.kind,
                            frame_id=get_dbc_frame_id(message),
                            value=get_value(attribute)))

        for signal in message.signals[::-1]:
            if signal.dbc is not None:
                if signal.dbc.attributes is not None:
                    for attribute in signal.dbc.attributes.values():
                        if attribute.value == attribute.definition.default_value:
                            continue

                        ba.append(
                            'BA_ "{name}" {kind} {frame_id} {signal_name} {value};'.format(
                                name=attribute.definition.name,
                                kind=attribute.definition.kind,
                                frame_id=get_dbc_frame_id(message),
                                signal_name=signal.name,
                                value=get_value(attribute)))

    return ba

def _dump_choices(database):
    val = []

    for message in database.messages:
        for signal in message.signals[::-1]:
            if signal.choices is None:
                continue

            val.append(
                'VAL_ {frame_id} {name} {choices} ;'.format(
                    frame_id=get_dbc_frame_id(message),
                    name=signal.name,
                    choices=' '.join(['{value} "{text}"'.format(value=value,
                                                                text=text)
                                      for value, text in signal.choices.items()])))

    return val

def _dump_signal_groups(database):
    sig_group = []

    for message in database.messages:
        if message.signal_groups is None:
            continue

        for signal_group in message.signal_groups:
            all_sig_names = list(map(lambda sig: sig.name, message.signals))
            signal_group.signal_names = list(filter(lambda sig_name: sig_name in all_sig_names, signal_group.signal_names))
            sig_group.append(
                'SIG_GROUP_ {frame_id} {signal_group_name} {repetitions} : {signal_names};'.format(
                    frame_id=get_dbc_frame_id(message),
                    signal_group_name=signal_group.name,
                    repetitions=signal_group.repetitions,
                    signal_names=' '.join(signal_group.signal_names)
                ))

    return sig_group

def _is_extended_mux_needed(messages):
    """Check for messages with more than one mux signal or signals with
    more than one multiplexer value.

    """

    for message in messages:
        multiplexers = [
            signal.name
            for signal in message.signals
            if signal.is_multiplexer
        ]

        if len(multiplexers) > 1:
            return True

        for signal in message.signals:
            if signal.multiplexer_ids:
                if len(signal.multiplexer_ids) > 1:
                    return True

    return False

def _create_mux_ranges(multiplexer_ids):
    """Create a list of ranges based on a list of single values.

    Example:
        Input:  [1, 2, 3, 5,      7, 8, 9]
        Output: [[1, 3], [5, 5], [7, 9]]

    """

    ordered = sorted(multiplexer_ids)
    # Anything but ordered[0] - 1
    prev_value = ordered[0]
    ranges = []

    for value in ordered:
        if value == prev_value + 1:
            ranges[-1][1] = value
        else:
            ranges.append([value, value])

        prev_value = value

    return ranges

def _dump_signal_mux_values(database):
    """Create multiplex entries ("SG_MUL_VAL_") if extended multiplexing
    is used.

    """

    if not _is_extended_mux_needed(database.messages):
        return []

    sig_mux_values = []

    for message in database.messages:
        for signal in message.signals:
            if not signal.multiplexer_ids:
                continue

            ranges = ', '.join([
                '{}-{}'.format(minimum, maximum)
                for minimum, maximum in _create_mux_ranges(signal.multiplexer_ids)
            ])

            sig_mux_values.append(
                'SG_MUL_VAL_ {frame_id} {name} {multiplexer} {ranges};'.format(
                    frame_id=get_dbc_frame_id(message),
                    name=signal.name,
                    multiplexer=signal.multiplexer_signal,
                    ranges=ranges))

    return sig_mux_values

def _load_comments(tokens):
    comments = defaultdict(dict)

    for comment in tokens.get('CM_', []):
        if not isinstance(comment[1], list):
            comments['database']['bus'] = comment[1]
            continue

        item = comment[1]
        kind = item[0]

        if kind == 'SG_':
            frame_id = int(item[1])

            if 'signal' not in comments[frame_id]:
                comments[frame_id]['signal'] = {}

            comments[frame_id]['signal'][item[2]] = item[3]
        elif kind == 'BO_':
            frame_id = int(item[1])
            comments[frame_id]['message'] = item[2]
        elif kind == 'BU_':
            node_name = item[1]
            comments[node_name] = item[2]
        elif kind == 'EV_':
            environment_variable_name = item[1]
            comments[environment_variable_name] = item[2]

    return comments

def _load_attribute_definitions(tokens):
    return tokens.get('BA_DEF_', [])

def _load_attribute_definition_defaults(tokens):
    defaults = odict()

    for default_attr in tokens.get('BA_DEF_DEF_', []):
        defaults[default_attr[1]] = default_attr[2]

    return defaults

def _load_attributes(tokens, definitions):
    attributes = odict()
    attributes['node'] = odict()

    def to_object(attribute):
        value = attribute[3]

        definition = definitions[attribute[1]]

        if definition.type_name in ['INT', 'HEX', 'ENUM']:
            value = to_int(value)
        elif definition.type_name == 'FLOAT':
            value = Decimal(value)

        return Attribute(value=value,
                         definition=definition)

    for attribute in tokens.get('BA_', []):
        name = attribute[1]

        if len(attribute[2]) > 0:
            item = attribute[2][0]
            kind = item[0]

            if kind == 'SG_':
                frame_id_dbc = int(item[1])
                signal = item[2]

                if frame_id_dbc not in attributes:
                    attributes[frame_id_dbc] = {}
                    attributes[frame_id_dbc]['message'] = odict()

                if 'signal' not in attributes[frame_id_dbc]:
                    attributes[frame_id_dbc]['signal'] = odict()

                if signal not in attributes[frame_id_dbc]['signal']:
                    attributes[frame_id_dbc]['signal'][signal] = odict()

                attributes[frame_id_dbc]['signal'][signal][name] = to_object(attribute)
            elif kind == 'BO_':
                frame_id_dbc = int(item[1])

                if frame_id_dbc not in attributes:
                    attributes[frame_id_dbc] = {}
                    attributes[frame_id_dbc]['message'] = odict()

                attributes[frame_id_dbc]['message'][name] = to_object(attribute)
            elif kind == 'BU_':
                node = item[1]

                if node not in attributes['node']:
                    attributes['node'][node] = odict()

                attributes['node'][node][name] = to_object(attribute)
            elif kind == 'EV_':
                envvar = item[1]

                if 'envvar' not in attributes:
                    attributes['envvar'] = odict()

                if envvar not in attributes['envvar']:
                    attributes['envvar'][envvar] = odict()

                attributes['envvar'][envvar][name] = to_object(attribute)
        else:
            if 'database' not in attributes:
                attributes['database'] = odict()

            attributes['database'][name] = to_object(attribute)

    return attributes

def _load_value_tables(tokens):
    """Load value tables, that is, choice definitions.

    """

    value_tables = odict()

    for value_table in tokens.get('VAL_TABLE_', []):
        name = value_table[1]
        choices = {int(number): text for number, text in value_table[2]}
        value_tables[name] = choices

    return value_tables

def _load_environment_variables(tokens, comments, attributes):
    environment_variables = odict()

    for env_var in tokens.get('EV_', []):
        name = _get_environment_variable_name(attributes, env_var[1])
        environment_variables[name] = EnvironmentVariable(
            name=name,
            env_type=int(env_var[3]),
            minimum=num(env_var[5]),
            maximum=num(env_var[7]),
            unit=env_var[9],
            initial_value=num(env_var[10]),
            env_id=int(env_var[11]),
            access_type=env_var[12],
            access_node=env_var[13],
            comment=comments.get(env_var[1], None))

    return environment_variables

def _load_choices(tokens):
    choices = defaultdict(dict)

    for choice in tokens.get('VAL_', []):
        if len(choice[1]) == 0:
            continue

        od = odict((int(''.join(v[0])), v[1]) for v in choice[3])

        if len(od) == 0:
            continue

        frame_id = int(choice[1][0])
        choices[frame_id][choice[2]] = od

    return choices

def _load_message_senders(tokens, attributes):
    """Load additional message senders.

    """

    message_senders = defaultdict(list)

    for senders in tokens.get('BO_TX_BU_', []):
        frame_id = int(senders[1])
        message_senders[frame_id] += [
            _get_node_name(attributes, sender) for sender in senders[3]
        ]

    return message_senders

def _load_signal_types(tokens):
    """Load signal types.

    """

    signal_types = defaultdict(dict)

    for signal_type in tokens.get('SIG_VALTYPE_', []):
        frame_id = int(signal_type[1])
        signal_name = signal_type[2]
        signal_types[frame_id][signal_name] = int(signal_type[4])

    return signal_types

def _load_signal_multiplexer_values(tokens):
    """Load additional signal multiplexer values.

    """

    signal_multiplexer_values = defaultdict(dict)

    for signal_multiplexer_value in tokens.get('SG_MUL_VAL_', []):
        frame_id = int(signal_multiplexer_value[1])
        signal_name = signal_multiplexer_value[2]
        multiplexer_signal = signal_multiplexer_value[3]
        multiplexer_ids = []

        for lower, upper in signal_multiplexer_value[4]:
            lower = int(lower)
            upper = int(upper[1:])
            # ToDo: Probably store ranges as tuples to not run out of
            #       memory on huge ranges.
            multiplexer_ids.extend(range(lower, upper + 1))

        if multiplexer_signal not in signal_multiplexer_values[frame_id]:
            signal_multiplexer_values[frame_id][multiplexer_signal] = {}

        multiplexer_signal = signal_multiplexer_values[frame_id][multiplexer_signal]
        multiplexer_signal[signal_name] = multiplexer_ids

    return signal_multiplexer_values

def _load_signal_groups(tokens):
    """Load signal groups.

    """

    signal_groups = defaultdict(list)

    for signal_group in tokens.get('SIG_GROUP_',[]):
        frame_id = int(signal_group[1])
        signal_groups[frame_id].append(SignalGroup(name=signal_group[2],
                                                   repetitions=int(signal_group[3]),
                                                   signal_names=signal_group[5]))

    return signal_groups

def _load_signals(tokens,
                  comments,
                  attributes,
                  definitions,
                  choices,
                  signal_types,
                  signal_multiplexer_values,
                  frame_id_dbc,
                  multiplexer_signal):
    signal_to_multiplexer = {}

    try:
        signal_multiplexer_values = signal_multiplexer_values[frame_id_dbc]

        for multiplexer_name, items in signal_multiplexer_values.items():
            for name in items:
                signal_to_multiplexer[name] = multiplexer_name
    except KeyError:
        pass

    def get_attributes(frame_id_dbc, signal):
        """Get attributes for given signal.

        """

        try:
            return attributes[frame_id_dbc]['signal'][signal]
        except KeyError:
            return None

    def get_comment(frame_id_dbc, signal):
        """Get comment for given signal.

        """

        try:
            return comments[frame_id_dbc]['signal'][signal]
        except:
            return None

    def get_choices(frame_id_dbc, signal):
        """Get choices for given signal.

        """

        try:
            return choices[frame_id_dbc][signal]
        except KeyError:
            return None

    def get_is_multiplexer(signal):
        if len(signal[1]) == 2:
            return signal[1][1].endswith('M')
        else:
            return False

    def get_multiplexer_ids(signal, multiplexer_signal):
        ids = []

        if multiplexer_signal is not None:
            if len(signal) == 2 and not signal[1].endswith('M'):
                value = signal[1][1:].rstrip('M')
                ids.append(int(value))
        else:
            multiplexer_signal = get_multiplexer_signal(signal,
                                                        multiplexer_signal)

        try:
            ids.extend(
                signal_multiplexer_values[multiplexer_signal][signal[0]])
        except KeyError:
            pass

        if ids:
            return list(set(ids))

    def get_multiplexer_signal(signal, multiplexer_signal):
        if len(signal) != 2:
            return

        if multiplexer_signal is None:
            try:
                return signal_to_multiplexer[signal[0]]
            except KeyError:
                pass
        elif signal[0] != multiplexer_signal:
            return multiplexer_signal

    def get_receivers(receivers):
        if receivers == ['Vector__XXX']:
            receivers = []

        return [_get_node_name(attributes, receiver) for receiver in receivers]

    def get_minimum(minimum, maximum):
        if minimum == maximum == '0':
            return None
        else:
            return num(minimum)

    def get_maximum(minimum, maximum):
        if minimum == maximum == '0':
            return None
        else:
            return num(maximum)

    def get_minimum_decimal(minimum, maximum):
        if minimum == maximum == '0':
            return None
        else:
            return Decimal(minimum)

    def get_maximum_decimal(minimum, maximum):
        if minimum == maximum == '0':
            return None
        else:
            return Decimal(maximum)

    def get_is_float(frame_id_dbc, signal):
        """Get is_float for given signal.

        """

        try:
            return signal_types[frame_id_dbc][signal] in FLOAT_SIGNAL_TYPES
        except KeyError:
            return False

    def get_signal_name(frame_id_dbc, name):
        signal_attributes = get_attributes(frame_id_dbc, name)

        try:
            return signal_attributes['SystemSignalLongSymbol'].value
        except (KeyError, TypeError):
            return name

    def get_signal_initial_value(frame_id_dbc, name):
        signal_attributes = get_attributes(frame_id_dbc, name)

        try:
            return signal_attributes['GenSigStartValue'].value
        except (KeyError, TypeError):
            return None

    def get_signal_spn(frame_id_dbc, name):
        signal_attributes = get_attributes(frame_id_dbc, name)

        try:
            return signal_attributes['SPN'].value
        except (KeyError, TypeError):
            return None

    signals = []

    for signal in tokens:
        signals.append(
            Signal(name=get_signal_name(frame_id_dbc, signal[1][0]),
                   start=int(signal[3]),
                   length=int(signal[5]),
                   receivers=get_receivers(signal[20]),
                   byte_order=('big_endian'
                               if signal[7] == '0'
                               else 'little_endian'),
                   is_signed=(signal[8] == '-'),
                   initial=get_signal_initial_value(frame_id_dbc, signal[1][0]),
                   scale=num(signal[10]),
                   offset=num(signal[12]),
                   minimum=get_minimum(signal[15], signal[17]),
                   maximum=get_maximum(signal[15], signal[17]),
                   decimal=SignalDecimal(Decimal(signal[10]),
                                         Decimal(signal[12]),
                                         get_minimum_decimal(signal[15],
                                                             signal[17]),
                                         get_maximum_decimal(signal[15],
                                                             signal[17])),
                   unit=(None if signal[19] == '' else signal[19]),
                   spn=get_signal_spn(frame_id_dbc, signal[1][0]),
                   choices=get_choices(frame_id_dbc,
                                       signal[1][0]),
                   dbc_specifics=DbcSpecifics(get_attributes(frame_id_dbc, signal[1][0]),
                                              definitions),
                   comment=get_comment(frame_id_dbc,
                                       signal[1][0]),
                   is_multiplexer=get_is_multiplexer(signal),
                   multiplexer_ids=get_multiplexer_ids(signal[1],
                                                       multiplexer_signal),
                   multiplexer_signal=get_multiplexer_signal(signal[1],
                                                             multiplexer_signal),
                   is_float=get_is_float(frame_id_dbc, signal[1][0])))

    return signals

def _load_messages(tokens,
                   comments,
                   attributes,
                   definitions,
                   choices,
                   message_senders,
                   signal_types,
                   signal_multiplexer_values,
                   strict,
                   bus_name,
                   signal_groups):
    """Load messages.

    """

    def get_attributes(frame_id_dbc):
        """Get attributes for given message.

        """
        from collections import OrderedDict

        attrib = OrderedDict()

        for k, v in definitions.items():
            attrib[k] = Attribute(v.default_value, v)

        if frame_id_dbc in attributes and 'message' in attributes[frame_id_dbc]:
            for k, v in attributes[frame_id_dbc]['message'].items():
                attrib[k] = v

        return attrib

    def get_comment(frame_id_dbc):
        """Get comment for given message.

        """

        try:
            return comments[frame_id_dbc]['message']
        except:
            return None

    def get_send_type(frame_id_dbc):
        """Get send type for a given message.

        """

        message_attributes = get_attributes(frame_id_dbc)

        if 'GenMsgSendType' in message_attributes:
            result = message_attributes['GenMsgSendType'].value
            # Resolve ENUM index to ENUM text
            result = message_attributes['GenMsgSendType'].definition.choices[int(result)]
        else:
            result = None

        return result

    def get_cycle_time(frame_id_dbc):
        """Get cycle time for a given message.

        """

        message_attributes = get_attributes(frame_id_dbc)

        if 'GenMsgCycleTime' in message_attributes:
            result = int(message_attributes['GenMsgCycleTime'].value)
        else:
            result = None

        return result

    def get_frame_format(frame_id_dbc):
        """Gets the frame format for a given message

        """
        message_attributes = get_attributes(frame_id_dbc)

        if 'VFrameFormat' in message_attributes:
            frame_format = message_attributes['VFrameFormat'].value
            frame_format = message_attributes['VFrameFormat'].definition.choices[frame_format]
        else:
            frame_format = None

        return frame_format

    def get_protocol(frame_id_dbc):
        """Get protocol for a given message.

        """

        message_attributes = get_attributes(frame_id_dbc)

        if 'ProtocolType' in message_attributes:
            protocol_type = message_attributes['ProtocolType'].value
        else:
            protocol_type = None

        return protocol_type

    def get_message_name(frame_id_dbc, name):
        message_attributes = get_attributes(frame_id_dbc)

        if 'SystemMessageLongSymbol' in message_attributes:
            result = message_attributes['SystemMessageLongSymbol'].value
        else:
            result = name

        return result

    def get_signal_groups(frame_id_dbc):
        try:
            return signal_groups[frame_id_dbc]
        except:
            return None

    messages = []

    for message in tokens.get('BO_', []):
        # Any message named VECTOR__INDEPENDENT_SIG_MSG contains
        # signals not assigned to any message. Cantools does not yet
        # support unassigned signals. Discard them for now.
        if message[2] == 'VECTOR__INDEPENDENT_SIG_MSG':
            continue

        # Frame id.
        frame_id_dbc = int(message[1])
        frame_id = frame_id_dbc & 0x7fffffff
        is_extended_frame = bool(frame_id_dbc & 0x80000000)

        # Senders.
        senders = [_get_node_name(attributes, message[5])]

        for node in message_senders.get(frame_id_dbc, []):
            if node not in senders:
                senders.append(node)

        if senders == ['Vector__XXX']:
            senders = []

        # Signal multiplexing.
        multiplexer_signal = None

        for signal in message[6]:
            if len(signal[1]) == 2:
                if signal[1][1].endswith('M'):
                    if multiplexer_signal is None:
                        multiplexer_signal = signal[1][0]
                    else:
                        multiplexer_signal = None
                        break

        signals = _load_signals(message[6],
                                comments,
                                attributes,
                                definitions,
                                choices,
                                signal_types,
                                signal_multiplexer_values,
                                frame_id_dbc,
                                multiplexer_signal)

        messages.append(
            Message(frame_id=frame_id,
                    is_extended_frame=is_extended_frame,
                    name=get_message_name(frame_id_dbc, message[2]),
                    length=int(message[4], 0),
                    senders=senders,
                    send_type=get_send_type(frame_id_dbc),
                    cycle_time=get_cycle_time(frame_id_dbc),
                    dbc_specifics=DbcSpecifics(get_attributes(frame_id_dbc),
                                               definitions),
                    signals=signals,
                    comment=get_comment(frame_id_dbc),
                    strict=strict,
                    protocol=get_protocol(frame_id_dbc),
                    frame_format=get_frame_format(frame_id_dbc),
                    bus_name=bus_name,
                    signal_groups=get_signal_groups(frame_id_dbc)))

    return messages

def _load_version(tokens):
    return tokens.get('VERSION', [[None, None]])[0][1]

def _load_bus(attributes, comments):
    try:
        bus_name = attributes['database']['DBName'].value
    except KeyError:
        return None

    try:
        bus_baudrate = attributes['database']['Baudrate'].value
    except KeyError:
        bus_baudrate = None

    try:
        bus_comment = comments['database']['bus']
    except KeyError:
        bus_comment = None

    return Bus(bus_name, baudrate=bus_baudrate, comment=bus_comment)

def _load_nodes(tokens, comments, attributes, definitions):
    nodes = None

    for token in tokens.get('BU_', []):
        nodes = [Node(name=_get_node_name(attributes, node),
                      comment=comments.get(node, None),
                      dbc_specifics=DbcSpecifics(attributes['node'].get(node, None),
                                                 definitions))
                 for node in token[2]]

    return nodes

def get_attribute_definition(database, name, default):
    if database.dbc is None:
        database.dbc = DbcSpecifics()

    if name not in database.dbc.attribute_definitions:
        database.dbc.attribute_definitions[name] = default

    return database.dbc.attribute_definitions[name]

def get_long_node_name_attribute_definition(database):
    return get_attribute_definition(database,
                                    'SystemNodeLongSymbol',
                                    ATTRIBUTE_DEFINITION_LONG_NODE_NAME)

def get_long_message_name_attribute_definition(database):
    return get_attribute_definition(database,
                                    'SystemMessageLongSymbol',
                                    ATTRIBUTE_DEFINITION_LONG_MESSAGE_NAME)

def get_long_signal_name_attribute_definition(database):
    return get_attribute_definition(database,
                                    'SystemSignalLongSymbol',
                                    ATTRIBUTE_DEFINITION_LONG_SIGNAL_NAME)

def try_remove_attribute(dbc, name):
    try:
        dbc.attributes.pop(name)
    except (KeyError, AttributeError):
        pass

def make_node_names_unique(database):
    converter = LongNamesConverter(database)

    for node in database.nodes:
        name = converter.convert(node.name)
        try_remove_attribute(node.dbc, 'SystemNodeLongSymbol')

        if name is None:
            continue

        for message in database.messages:
            for index, sender in enumerate(message.senders):
                if sender == node.name:
                    message.senders[index] = name

            for signal in message.signals:
                for index, receiver in enumerate(signal.receivers):
                    if receiver == node.name:
                        signal.receivers[index] = name

        if node.dbc is None:
            node.dbc = DbcSpecifics()

        node.dbc.attributes['SystemNodeLongSymbol'] = Attribute(
            node.name,
            get_long_node_name_attribute_definition(database))
        node.name = name

def make_message_names_unique(database):
    converter = LongNamesConverter(database)

    for message in database.messages:
        name = converter.convert(message.name)
        try_remove_attribute(message.dbc, 'SystemMessageLongSymbol')

        if name is None:
            continue

        if message.dbc is None:
            message.dbc = DbcSpecifics()

        message.dbc.attributes['SystemMessageLongSymbol'] = Attribute(
            message.name,
            get_long_message_name_attribute_definition(database))
        message.name = name

def make_signal_names_unique(database):
    converter = LongNamesConverter(database)

    for message in database.messages:
        for signal in message.signals:
            name = converter.convert(signal.name)
            try_remove_attribute(signal.dbc, 'SystemSignalLongSymbol')

            if name is None:
                continue

            if signal.dbc is None:
                signal.dbc = DbcSpecifics()

            signal.dbc.attributes['SystemSignalLongSymbol'] = Attribute(
                signal.name,
                get_long_signal_name_attribute_definition(database))
            signal.name = name

def make_names_unique(database):
    """Make message, signal and node names unique and add attributes for
    their long names.

    """

    make_node_names_unique(database)
    make_message_names_unique(database)
    make_signal_names_unique(database)

    return database

def dump_string(database):
    """Format database in DBC file format.

    """

    # Make a deep copy of the database as names and attributes will be
    # modified for items with long names.
    database = deepcopy(database)

    database = make_names_unique(database)
    bu = _dump_nodes(database)
    val_table = _dump_value_tables(database)
    bo = _dump_messages(database)
    bo_tx_bu = _dump_senders(database)
    cm = _dump_comments(database)
    signal_types = _dump_signal_types(database)
    ba_def = _dump_attribute_definitions(database)
    ba_def_def = _dump_attribute_definition_defaults(database)
    ba = _dump_attributes(database)
    val = _dump_choices(database)
    sig_group = _dump_signal_groups(database)
    sig_mux_values = _dump_signal_mux_values(database)

    return DBC_FMT.format(version=_dump_version(database),
                          bu=' '.join(bu),
                          val_table='\r\n'.join(val_table),
                          bo='\r\n\r\n'.join(bo),
                          bo_tx_bu='\r\n'.join(bo_tx_bu),
                          cm='\r\n'.join(cm),
                          signal_types='\r\n'.join(signal_types),
                          ba_def='\r\n'.join(ba_def),
                          ba_def_def='\r\n'.join(ba_def_def),
                          ba='\r\n'.join(ba),
                          val='\r\n'.join(val),
                          sig_group='\r\n'.join(sig_group),
                          sig_mux_values='\r\n'.join(sig_mux_values))

def get_definitions_dict(definitions, defaults):
    result = odict()

    def convert_value(definition, value):
        if definition.type_name in ['INT', 'HEX']:
            value = to_int(value)
        elif definition.type_name == 'FLOAT':
            value = Decimal(value)

        return value

    for item in definitions:
        if len(item[1]) > 0:
            kind = item[1][0]
        else:
            kind = None

        definition = AttributeDefinition(name=item[2],
                                         kind=kind,
                                         type_name=item[3])
        values = item[4][0]

        if len(values) > 0:
            if definition.type_name == "ENUM":
                definition.choices = values
            elif definition.type_name in ['INT', 'FLOAT', 'HEX']:
                definition.minimum = convert_value(definition, values[0])
                definition.maximum = convert_value(definition, values[1])

        try:
            value = defaults[definition.name]
            definition.default_value = convert_value(definition, value)
        except KeyError:
            definition.default_value = None

        result[definition.name] = definition

    return result

def load_string(string, strict=True):
    """Parse given string.

    """

    tokens = Parser().parse(string)

    comments = _load_comments(tokens)
    definitions = _load_attribute_definitions(tokens)
    defaults = _load_attribute_definition_defaults(tokens)
    attribute_definitions = get_definitions_dict(definitions, defaults)
    attributes = _load_attributes(tokens, attribute_definitions)
    bus = _load_bus(attributes, comments)
    value_tables = _load_value_tables(tokens)
    choices = _load_choices(tokens)
    message_senders = _load_message_senders(tokens, attributes)
    signal_types = _load_signal_types(tokens)
    signal_multiplexer_values = _load_signal_multiplexer_values(tokens)
    signal_groups = _load_signal_groups(tokens)
    messages = _load_messages(tokens,
                              comments,
                              attributes,
                              attribute_definitions,
                              choices,
                              message_senders,
                              signal_types,
                              signal_multiplexer_values,
                              strict,
                              bus.name if bus else None,
                              signal_groups)
    nodes = _load_nodes(tokens, comments, attributes, attribute_definitions)
    version = _load_version(tokens)
    environment_variables = _load_environment_variables(tokens, comments, attributes)
    dbc_specifics = DbcSpecifics(attributes.get('database', None),
                                 attribute_definitions,
                                 environment_variables,
                                 value_tables)

    return InternalDatabase(messages,
                            nodes,
                            [bus] if bus else [],
                            version,
                            dbc_specifics)

database.py (click to expand)
import logging

from .formats import arxml
from .formats import dbc
from .formats import kcd
from .formats import sym
from .internal_database import InternalDatabase
from ...compat import fopen

LOGGER = logging.getLogger(__name__)

class Database(object):
    """This class contains all messages, signals and definitions of a CAN
    network.

    The factory functions :func:`load()<cantools.database.load()>`,
    :func:`load_file()<cantools.database.load_file()>` and
    :func:`load_string()<cantools.database.load_string()>` returns
    instances of this class.

    If `strict` is ``True`` an exception is raised if any signals are
    overlapping or if they don't fit in their message.

    """

    def __init__(self,
                 messages=None,
                 nodes=None,
                 buses=None,
                 version=None,
                 dbc_specifics=None,
                 frame_id_mask=None,
                 strict=True):
        self._messages = messages if messages else []
        self._nodes = nodes if nodes else []
        self._buses = buses if buses else []
        self._name_to_message = {}
        self._frame_id_to_message = {}
        self._version = version
        self._dbc = dbc_specifics

        if frame_id_mask is None:
            frame_id_mask = 0xffffffff

        self._frame_id_mask = frame_id_mask
        self._strict = strict
        self.refresh()

    @property
    def messages(self):
        """A list of messages in the database.

        Use :meth:`.get_message_by_frame_id()` or
        :meth:`.get_message_by_name()` to find a message by its frame
        id or name.

        """

        return self._messages

    @property
    def nodes(self):
        """A list of nodes in the database.

        """

        return self._nodes

    @property
    def buses(self):
        """A list of CAN buses in the database.

        """

        return self._buses

    @property
    def version(self):
        """The database version, or ``None`` if unavailable.

        """

        return self._version

    @version.setter
    def version(self, value):
        self._version = value

    @property
    def dbc(self):
        """An object containing dbc specific properties like e.g. attributes.

        """

        return self._dbc

    @dbc.setter
    def dbc(self, value):
        self._dbc = value

    def add_arxml(self, fp):
        """Read and parse ARXML data from given file-like object and add the
        parsed data to the database.

        """

        self.add_arxml_string(fp.read())

    def add_arxml_file(self, filename, encoding='utf-8'):
        """Open, read and parse ARXML data from given file and add the parsed
        data to the database.

        `encoding` specifies the file encoding.

        """

        with fopen(filename, 'r', encoding=encoding) as fin:
            self.add_arxml(fin)

    def add_arxml_string(self, string):
        """Parse given ARXML data string and add the parsed data to the
        database.

        """

        database = arxml.load_string(string, self._strict)

        self._messages += database.messages
        self._nodes = database.nodes
        self._buses = database.buses
        self._version = database.version
        self._dbc = database.dbc
        self.refresh()

    def add_dbc(self, fp):
        """Read and parse DBC data from given file-like object and add the
        parsed data to the database.

        >>> db = cantools.database.Database()
        >>> with open ('foo.dbc', 'r') as fin:
        ...     db.add_dbc(fin)

        """

        self.add_dbc_string(fp.read())

    def add_dbc_file(self, filename, encoding='cp1252'):
        """Open, read and parse DBC data from given file and add the parsed
        data to the database.

        `encoding` specifies the file encoding.

        >>> db = cantools.database.Database()
        >>> db.add_dbc_file('foo.dbc')

        """

        with fopen(filename, 'r', encoding=encoding) as fin:
            self.add_dbc(fin)

    def add_dbc_string(self, string):
        """Parse given DBC data string and add the parsed data to the
        database.

        >>> db = cantools.database.Database()
        >>> with open ('foo.dbc', 'r') as fin:
        ...     db.add_dbc_string(fin.read())

        """

        database = dbc.load_string(string, self._strict)

        self._messages += database.messages
        self._nodes = database.nodes
        self._buses = database.buses
        self._version = database.version
        self._dbc = database.dbc
        self.refresh()

    def add_kcd(self, fp):
        """Read and parse KCD data from given file-like object and add the
        parsed data to the database.

        """

        self.add_kcd_string(fp.read())

    def add_kcd_file(self, filename, encoding='utf-8'):
        """Open, read and parse KCD data from given file and add the parsed
        data to the database.

        `encoding` specifies the file encoding.

        """

        with fopen(filename, 'r', encoding=encoding) as fin:
            self.add_kcd(fin)

    def add_kcd_string(self, string):
        """Parse given KCD data string and add the parsed data to the
        database.

        """

        database = kcd.load_string(string, self._strict)

        self._messages += database.messages
        self._nodes = database.nodes
        self._buses = database.buses
        self._version = database.version
        self._dbc = database.dbc
        self.refresh()

    def add_sym(self, fp):
        """Read and parse SYM data from given file-like object and add the
        parsed data to the database.

        """

        self.add_sym_string(fp.read())

    def add_sym_file(self, filename, encoding='utf-8'):
        """Open, read and parse SYM data from given file and add the parsed
        data to the database.

        `encoding` specifies the file encoding.

        """

        with fopen(filename, 'r', encoding=encoding) as fin:
            self.add_sym(fin)

    def add_sym_string(self, string):
        """Parse given SYM data string and add the parsed data to the
        database.

        """

        database = sym.load_string(string, self._strict)

        self._messages += database.messages
        self._nodes = database.nodes
        self._buses = database.buses
        self._version = database.version
        self._dbc = database.dbc
        self.refresh()

    def _add_message(self, message):
        """Add given message to the database.

        """

        if message.name in self._name_to_message:
            LOGGER.warning("Overwriting message '%s' with '%s' in the "
                           "name to message dictionary.",
                           self._name_to_message[message.name].name,
                           message.name)

        masked_frame_id = (message.frame_id & self._frame_id_mask)

        if masked_frame_id in self._frame_id_to_message:
            LOGGER.warning(
                "Overwriting message '%s' with '%s' in the frame id to message "
                "dictionary because they have identical masked frame ids 0x%x.",
                self._frame_id_to_message[masked_frame_id].name,
                message.name,
                masked_frame_id)

        self._name_to_message[message.name] = message
        self._frame_id_to_message[masked_frame_id] = message

    def as_dbc_string(self):
        """Return the database as a string formatted as a DBC file.

        """

        return dbc.dump_string(InternalDatabase(self._messages,
                                                self._nodes,
                                                self._buses,
                                                self._version,
                                                self._dbc))

    def as_kcd_string(self):
        """Return the database as a string formatted as a KCD file.

        """

        return kcd.dump_string(InternalDatabase(self._messages,
                                                self._nodes,
                                                self._buses,
                                                self._version,
                                                self._dbc))

    def get_message_by_name(self, name):
        """Find the message object for given name `name`.

        """

        return self._name_to_message[name]

    def get_message_by_frame_id(self, frame_id):
        """Find the message object for given frame id `frame_id`.

        """

        if frame_id & self._frame_id_mask in self._frame_id_to_message:
            return self._frame_id_to_message[frame_id & self._frame_id_mask]

        for message in self.messages:
            if (
                'UseGMParameterIDs' in message.dbc.attributes and
                message.dbc.attributes['UseGMParameterIDs'].value == 1
            ):
                # convert to GM Parameter Id
                id_ = (frame_id >> 13 & 0x1FFF) << 13

                if message.frame_id == id_:
                    return message

            elif (
                message.frame_format is not None and
                message.frame_format.startswith('J1939')
            ):
                # convert to J1939 paramter group
                id_ = ((frame_id >> 8) & 0x3FFFF) << 8
                if message.frame_id == id_:
                    return message

        raise KeyError(frame_id)

    def get_node_by_name(self, name):
        """Find the node object for given name `name`.

        """

        for node in self._nodes:
            if node.name == name:
                return node

        raise KeyError(name)

    def get_bus_by_name(self, name):
        """Find the bus object for given name `name`.

        """

        for bus in self._buses:
            if bus.name == name:
                return bus

        raise KeyError(name)

    def encode_message(self,
                       frame_id_or_name,
                       data,
                       scaling=True,
                       padding=False,
                       strict=True):
        """Encode given signal data `data` as a message of given frame id or
        name `frame_id_or_name`. `data` is a dictionary of signal
        name-value entries.

        If `scaling` is ``False`` no scaling of signals is performed.

        If `padding` is ``True`` unused bits are encoded as 1.

        If `strict` is ``True`` all signal values must be within their
        allowed ranges, or an exception is raised.

        >>> db.encode_message(158, {'Bar': 1, 'Fum': 5.0})
        b'\\x01\\x45\\x23\\x00\\x11'
        >>> db.encode_message('Foo', {'Bar': 1, 'Fum': 5.0})
        b'\\x01\\x45\\x23\\x00\\x11'

        """

        try:
            message = self.get_message_by_frame_id(frame_id_or_name)
        except KeyError:
            message = self.get_message_by_name(frame_id_or_name)

        return message.encode(data, scaling, padding, strict)

    def decode_message(self,
                       frame_id_or_name,
                       data,
                       decode_choices=True,
                       scaling=True):
        """Decode given signal data `data` as a message of given frame id or
        name `frame_id_or_name`. Returns a dictionary of signal
        name-value entries.

        If `decode_choices` is ``False`` scaled values are not
        converted to choice strings (if available).

        If `scaling` is ``False`` no scaling of signals is performed.

        >>> db.decode_message(158, b'\\x01\\x45\\x23\\x00\\x11')
        {'Bar': 1, 'Fum': 5.0}
        >>> db.decode_message('Foo', b'\\x01\\x45\\x23\\x00\\x11')
        {'Bar': 1, 'Fum': 5.0}

        """

        try:
            message = self.get_message_by_frame_id(frame_id_or_name)
        except KeyError:
            message = self.get_message_by_name(frame_id_or_name)

        return message.decode(data, decode_choices, scaling)

    def refresh(self):
        """Refresh the internal database state.

        This method must be called after modifying any message in the
        database to refresh the internal lookup tables used when
        encoding and decoding messages.

        """

        self._name_to_message = {}
        self._frame_id_to_message = {}

        for message in self._messages:
            message.refresh(self._strict)
            self._add_message(message)

    def __repr__(self):
        lines = []

        lines.append("version('{}')".format(self._version))
        lines.append('')

        if self._nodes:
            for node in self._nodes:
                lines.append(repr(node))

            lines.append('')

        for message in self._messages:
            lines.append(repr(message))

            for signal in message.signals:
                lines.append('  ' + repr(signal))

            lines.append('')

        return '\n'.join(lines)

Please create a draft PR to make it easier for me to see what was changed. Thanks =)

I can do that, give me 10 minutes or so to get it done.

Thanks!
First of all, is GMLAN public? I can't find it in my CANdb++ installation. Is there any configuration missing?
In the PR the frame id is masked based on frame format found in the dbc attribute. This violates the database, message and signal classes design as file format agnostic. The dbc specific attribute should not be accessed in these classes, except as storage of DBC specific information. We could add more attributes to these classes if needed to make frame format masking possible if needed.
There is already a frame_id_mask parameter in the database constructor to ignore given bits. It might not be that convenient, but might be useful in you case.
Could you provide an example script to showcase how you want to use cantools? It would give an even better understanding of which changes are needed.

I am still in the process of trying to understand  how the damned attributes work.
if I use this code the option to set the message id as a GM Extended id is available
BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_  "UseGMParameterIDs" 1;
BA_ "UseGMParameterIDs" 1;

if I use this code the option to set the message id as a GM Extended id is available
BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_  "UseGMParameterIDs" 0;
BA_ "UseGMParameterIDs" 1;

if I use this code the option to set the message id as a GM Extended id is available
BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_  "UseGMParameterIDs" 1;

if I use this code the option to set the message id as a GM Extended id is NOT available
BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_  "UseGMParameterIDs" 1;
BA_ "UseGMParameterIDs" 0;

if I use this code the option to set the message id as a GM Extended id is NOT available
BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_  "UseGMParameterIDs" 0;
BA_ "UseGMParameterIDs" 0;

if I use this code the option to set the message id as a GM Extended id is NOT available
BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_  "UseGMParameterIDs" 0;
``'

Now I am not sure as to why I can set the default to 0 and set a network attribute as a 1 and it effects the message

I am not able to do this I get an error when loading the file in CANdb++

BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_  "UseGMParameterIDs" 0;
BA_ "UseGMParameterIDs" BO_ 1 1;

it appears as tho the last line syntax is selective in what it works on.

Well, the syntax for defining and using attributes in a DBC is a little bit complicated.
I'll try to explain it.

use BA_DEF to define an attribute:
BA_DEF_ <object type> "<attribute name>" <type> <min> <max>;
use BA_DEF_DEF to set the default value:
BA_DEF_DEF "<attribute name>" <default value>;
BA_DEF_ and BA_DEF_DEF_ are mandatory! It's not allowed to omit one of them.
use BA_ (optionally) to set specific values for a certain object:
BA_ "<attribute name>" <object type> <object identifier> <specific value>;

Keep in mind that an attribute is always defined for a specific object type:

BO_ for messages
SG_ for signals
BU_ for nodes
EV_ for environement variables
'' (empty string/no qualifier) for the database/the network itself (this seems true for "useGMParameterIDs").

I guess the parameter "useGMParameterIDs" lets CANdb++ know how to present the message dialog (to split the 29 bit CAN id into the GM specific logic and let the user change the "GM ID" only). However, this attribute does not have different values for different objects (as it is defined as a databases attribute, not a message attribute).
Applying this logic your examples seem to be fine:
If you want the database attribute "UseGMParameterIDs" to be 1, you can do one of the following:

set the default value (BA_DEF_DEF_) to 1, and DON'T set a different value with BA_
set the default value to any value, and set the resulting value to 1

I haven't worked much with database attributes. But maybe they should have one instance only (the dbc itself). For that it might be confusing to set the default value and then set a different value with an extra BA_ line.
The syntax makes more sense for attributes of other types. Let's take messages:
BA_DEF_ BO_  "GenMsgCycleTime" INT 0 50000;
BA_DEF_DEF_  "GenMsgCycleTime" 0;
BA_ "GenMsgCycleTime" BO_ 2147483705 1000;

--> The message 2147483705 will have cycle time 1000, all others 50000.
For more details, you could ask Vector to give you their DBC format specifications. As it's confidential, I can't upload it. But you should receive it for internal use. The attribute syntax is specified in that document. However, I had many questions with Vector, because the spec lacks many aspects ...

what I would like to know is what the pre defined attribute names are and which ones work what ways.
It seems that the VFrameFormat really doesn't do anything with respect to adding additional dialogs for J1939 and the frame id. So I am going to guess that there is another attribute like the UseGMParameterIds variable except for J1939 something like UseJ1939ParamaterGroupNumbers or UserJ1939PGNs.
So now my question is also this
What does this do
BA_DEF_ BO_  "GenMsgCycleTime" INT 0 50000;
that is different from this?
BA_DEF_ "GenMsgCycleTime" INT 0 50000;
I am going to assume that it is to set the attribute to messages. and if that is the case then how come this works?
BA_DEF_DEF_  "GenSigStartValue" 0;
BA_ "GenSigStartValue" SG_ 2015 Odometer 0;
You would think it would have to be
BA_DEF_DEF_ SG_  "GenSigStartValue" 0;
BA_ "GenSigStartValue" SG_ 2015 Odometer 0;
then working under that assumption this should work as well (haven't tried it yet)
BA_DEF_ BO_ "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_  "UseGMParameterIDs" 0;
BA_ "UseGMParameterIDs" BO_ 1 1;

then if I do this
BA_DEF_ "GenMsgCycleTime" INT 0 50000;
BA_DEF_DEF_  "GenMsgCycleTime" 0

do all objects in the network have the parameter GenMsgCycleTime with a default of 50000 or only the network it's self?
There also does not seem to be a separation in cantools for the variable definitions,
BA_DEF_ BO_ "GenMsgCycleTime" INT 0 50000;

that should be placed into definitions['messages'] right now all of the definitions are lumped together. if defining an attribute this way means that all messages have the attribute then that attribute should be able to be see in Message.attributes, this does not seem to be the case, or maybe I am missing it somewhere?

ok so in the tests there is test_issue_167_long_names_from_scratch
in the test it uses this code to create a database
db = cantools.database.Database(
            messages=[
                can.message.Message(
                    frame_id=1,
                    name='MSG456789_123456789_123456789_ABC',
                    length=8,
                    signals=[
                        can.signal.Signal(name='SIG456789_123456789_123456789_ABC',
                                          start=9,
                                          length=8)
                    ],
                    senders=['NODE56789_abcdefghi_ABCDEFGHI_XYZ']),
                can.message.Message(
                    frame_id=2,
                    name='MSG_short',
                    length=8,
                    signals=[
                        can.signal.Signal(name='SIG_short', start=1, length=8)
                    ],
                    senders=['NODE_short'])
            ],
            nodes=[
                can.node.Node('NODE_short', None),
                can.node.Node('NODE56789_abcdefghi_ABCDEFGHI_XYZ', None)
            ],
            version='')

and this is the database output as a string
VERSION ""

NS_ :
	NS_DESC_
	CM_
	BA_DEF_
	BA_
	VAL_
	CAT_DEF_
	CAT_
	FILTER
	BA_DEF_DEF_
	EV_DATA_
	ENVVAR_DATA_
	SGTYPE_
	SGTYPE_VAL_
	BA_DEF_SGTYPE_
	BA_SGTYPE_
	SIG_TYPE_REF_
	VAL_TABLE_
	SIG_GROUP_
	SIG_VALTYPE_
	SIGTYPE_VALTYPE_
	BO_TX_BU_
	BA_DEF_REL_
	BA_REL_
	BA_DEF_DEF_REL_
	BU_SG_REL_
	BU_EV_REL_
	BU_BO_REL_
	SG_MUL_VAL_

BS_:

BU_: NODE_short NODE56789_abcdefghi_ABCDEFGHI_XY

BO_ 1 MSG456789_123456789_123456789_AB: 8 NODE56789_abcdefghi_ABCDEFGHI_XY
 SG_ SIG456789_123456789_123456789_AB : 9|8@1+ (1,0) [0|0] "" Vector__XXX

BO_ 2 MSG_short: 8 NODE_short
 SG_ SIG_short : 1|8@1+ (1,0) [0|0] "" Vector__XXX

BA_DEF_ BU_  "SystemNodeLongSymbol" STRING ;
BA_DEF_ BO_  "SystemMessageLongSymbol" STRING ;
BA_DEF_ SG_  "SystemSignalLongSymbol" STRING ;
BA_DEF_DEF_  "SystemNodeLongSymbol" "";
BA_DEF_DEF_  "SystemMessageLongSymbol" "";
BA_DEF_DEF_  "SystemSignalLongSymbol" "";
BA_ "SystemNodeLongSymbol" BU_ NODE56789_abcdefghi_ABCDEFGHI_XY "NODE56789_abcdefghi_ABCDEFGHI_XYZ";
BA_ "SystemMessageLongSymbol" BO_ 1 "MSG456789_123456789_123456789_ABC";
BA_ "SystemSignalLongSymbol" SG_ 1 SIG456789_123456789_123456789_AB "SIG456789_123456789_123456789_ABC";

Now based on what we have discussed above that would mean that the message MSG_short has a name of ""
and that message MSG456789_123456789_123456789_AB has a name of "MSG456789_123456789_123456789_ABC"
This would be because the attribute SystemMessageLongSymbol has been applied to all messages using the BO_ identifier and a default value of "" was specified for that attribute and there is no specific setting for the attribute for message 2 (MSG_short) so it is going to use the default value of ""
if this is not the case then Vectors API really isn't tickling my pickle if ya get my meaning. It seems like when the  API was added to and consideration for how the API was initially supposed to work was tossed right out the window.

what I would like to know is what the pre defined attribute names are and which ones work what ways.
Unfortunately that is not described in the DBC specification document from Vector.

It seems that the VFrameFormat really doesn't do anything with respect to adding additional dialogs for J1939 and the frame id. So I am going to guess that there is another attribute like the UseGMParameterIds variable except for J1939 something like UseJ1939ParamaterGroupNumbers or UseJ1939PGNs.
The VFrameFormat J1939 is also evaluated by other tools. Knowing that a Frame is defined as PG (and not as Standard or Extended CAN Frame), these tools do the transport protocol stuff for you. That means you can use these PGs (even with a "DLC" > 8) as it would be a single message. And then just send it. The tool care's about the protocol stuff, i. e. it creates a sequence of different frames (CAN extended frames with each 8 data bytes). You don't have to care about that yourself. Same for receiving PGs > 8 databytes.

So now my question is also this
What does this do
BA_DEF_ BO_ "GenMsgCycleTime" INT 0 50000;
--> this defines an attribute called GenMsgCycleTime for message objects; type = INT, valid range from 0 to 50000. As far as I know this is one of the (undocumented) J1939 specific attributes. I. e. using a J1939 database, this attribute is used for specifying message cycle times (and not the built-in cycle time attribute).

that is different from this?
BA_DEF_ "GenMsgCycleTime" INT 0 50000;
--> yes, that is different! This line omits the "BO_" qualifier and would therefor define a cycle time attribute for the whole database.

I am going to assume that it is to set the attribute to messages. and if that is the case then how come this works?
BA_DEF_DEF_ "GenSigStartValue" 0;
BA_ "GenSigStartValue" SG_ 2015 Odometer 0;
--> This is a different attribut (start value, not cycle time). The two lines list the default value and a specific value for the signal "Odometer" in the message with ID 2015. There should by a third line defining the attribute itself (e. g. BA_DEF_ SG_ "GenSigStartValue" INT 0 65535;. If such a line does not exist, I expect the DBC to be invalid.

You would think it would have to be
BA_DEF_DEF_ SG_ "GenSigStartValue" 0;
BA_ "GenSigStartValue" SG_ 2015 Odometer 0;
--> not exactly. The point is, that BA_DEF_ needs the object qualifier ("SG_" in this case), but BA_DEF_DEF_ does not even allwo it. The reference is just made by the name of the attribute.

then working under that assumption this should work as well (haven't tried it yet)
BA_DEF_ BO_ "UseGMParameterIDs" INT 0 0;
BA_DEF_DEF_ "UseGMParameterIDs" 0;
BA_ "UseGMParameterIDs" BO_ 1 1;
--> Yes, that looks correct (I'm not sure about INT 0 0. This could mean there's no explicit definition of min/max.
But one question:
Is "UseGMParameterIDs" something you want to define per message? I. e. you have some messages in GM style and others in a non GM style? If yes, that snippet above makes sense

then if I do this
BA_DEF_ "GenMsgCycleTime" INT 0 50000;
BA_DEF_DEF_  "GenMsgCycleTime" 0

do all objects in the network have the parameter GenMsgCycleTime with a default of 50000 or only the network it's self?

No. The BA_DEF_ does not have a type qualifier. That means it is an attribute on database level. It cannot be used for messages/signals/nodes/envvars!
Defining a "Database attribute" does not mean to set the default value for all objects in this database!!! The database itself is the (single) object of this type. If you want to set default values for all messages, use BO_ in the BA_DEF_ statement for example.

There also does not seem to be a separation in cantools for the variable definitions,
BA_DEF_ BO_ "GenMsgCycleTime" INT 0 50000;

that should be placed into definitions['messages'] right now all of the definitions are lumped together. if defining an attribute this way means that all messages have the attribute then that attribute should be able to be see in Message.attributes, this does not seem to be the case, or maybe I am missing it somewhere?

Yeah, that's maybe a problem. The concept of the DBC file format is to have two line for defining the attribute (and its default value), and then one extra line for every object that does not use the default value.
When reading such a file with cantools, this results in "empty" attribute lists for those objects that use the default values.

BU_: NODE_short NODE56789_abcdefghi_ABCDEFGHI_XY
BO_ 1 MSG456789_123456789_123456789_AB: 8 NODE56789_abcdefghi_ABCDEFGHI_XY
SG_ SIG456789_123456789_123456789_AB : 9|8@1+ (1,0) [0|0] "" Vector__XXX
BO_ 2 MSG_short: 8 NODE_short
SG_ SIG_short : 1|8@1+ (1,0) [0|0] "" Vector__XXX
BA_DEF_ BU_  "SystemNodeLongSymbol" STRING ;
BA_DEF_ BO_  "SystemMessageLongSymbol" STRING ;
BA_DEF_ SG_  "SystemSignalLongSymbol" STRING ;
BA_DEF_DEF_  "SystemNodeLongSymbol" "";
BA_DEF_DEF_  "SystemMessageLongSymbol" "";
BA_DEF_DEF_  "SystemSignalLongSymbol" "";
BA_ "SystemNodeLongSymbol" BU_ NODE56789_abcdefghi_ABCDEFGHI_XY "NODE56789_abcdefghi_ABCDEFGHI_XYZ";
BA_ "SystemMessageLongSymbol" BO_ 1 "MSG456789_123456789_123456789_ABC";
BA_ "SystemSignalLongSymbol" SG_ 1 SIG456789_123456789_123456789_AB "SIG456789_123456789_123456789_ABC";

Now based on what we have discussed above that would mean that the message `MSG_short` has a name of ""
and that message `MSG456789_123456789_123456789_AB` has a name of "MSG456789_123456789_123456789_ABC"

This would be because the attribute `SystemMessageLongSymbol` has been applied to all messages using the `BO_` identifier and a default value of "" was specified for that attribute and there is no specific setting for the attribute for message 2 (`MSG_short`) so it is going to use the default value of ""

Yes and no :-)
DBCs can treat two names of an object. A short name (up to 32 chars, mandatory), and a long name (afaik up to 128 chars, optionally).
The snippet above just says, that message 2 has an empty long name. But of course the short name still exists: MSG_short.
The short name is a kind of compatibility mode. I guess it's neccessary to support some old tools which are limited to 32 chars per object. However, any up to date tool should support the long names. At first, they read the short name. Then they check, if there's a attribute for the long name. If yes, the present the long name to the user, otherwise the short name. This is completely done in the background (also if you change the name from a short to a long string and save the DBC again ...).
This is one of the 'exceptions' that are in fact defined in the DBC specifications in detail.
I hope, this helps :-)

OK so when talking about the long names.
I have modified the cantools code so it properly populates the various objects with the default attributes. which in the case of the long name example above cantools is not handling the empty default attribute properly. I have to track down where that problem is. I wanted to confirm my understand to make sure that I do the right thing with respect to that portion of it.
Now with respect to the defining of an attribute, I am going to use the GenSigStartValue as an example.
In my project I have a DBC file. in that DBC file are all of the bits needed to decode an incoming OBDII CAN message. decoding a message does not need to make use of GenSigStartValue. but to send a message it does. I wanted to make everything work nice and easy so using a multiplexed setup along with GenSigStartValue I am able to have the the frame data constructed without needing to know anything  other then the name of the message and the name of the signal. by using the GenSigStartValue it sets the default of 0 for all signals including the signals that are only used for decoding. Is there a way to unset an attribute for a specific signal??

I just did a commit and push to #245 It has address all of the above mentioned topics we discussed, I have also updated the tests and all tests run successfully with the changes I have made.
The attributes are now set into their respective objects correctly and get output when calling as_dbc_str if the attributes value differs from the default set for that object type.
The attribute output of as_dbc_string is now more organized.

authors: eerimoq geo4711 kdschlosser

repo: eerimoq/cantools