As tokenization does involves some languagespecific processing, its use here somewhat undermines the objective set out for this project.
we report the accuracy achieved for both untokenized and tokenized Japanese tweets.