While the name bzip3 itself would be okay for reasons others stated, I still don't like it because it is so easy to confuse with bzip2 and may indicate a false relationship with bzip2 regardless of intents. A spiritual successor doesn't have to be named confusingly similar, after all.
If they have to be incompatible then it's better to not conceal that. Generalized file formats require you to implement more stuff to support them, and we can't tell the format by looking at the file name.
Please, no. What's next ? BZip4 and BZip5, each one incompatible with each other ?