The title of the song is a kind of joke/proverb. For instance, many rock bands that are well past their prime continue to be "Big in Japan" long after they cannot draw a large audience in the UK or US.
However, this song is mostly about a couple, who tries to get off a heroin addiction. The lead signer Marian Gold explains this this way:
"I must admit the only ones who understood the meaning of the song correctly were Bernhard (Lloyd) and our then publisher Andy Budde. Bernhard recognized the quality of the song and Andy the ability of its marketing. 'Big In Japan' tells about a couple of lovers trying to get off Heroin. They both imagine how great it would be to love without the drug: no steal, no clients, no ice age in the pupil, real emotions, true worlds. Till nowadays Berlin station Zoo is an important meeting place for junkies. That's why this place became a venue of the song. In 1977 I was quite regular in a club in Kreuzberg, the SO36. I hoped to meet David Bowie. There was the rumor he would be in Berlin for withdrawal treatment in that period. There was a fellow, who sold stuff of independent bands from US and UK. I bought the album of a British band named Big In Japan. "Big in Japan" meant so much as if you are nothing in your environs, you can be big somewhere else. You can be a king in another world. And if you aren't that, you can tell it at home. Japan is so far away. Anyway that statement fit perfectly to the story of the couple of lovers. So I used it for the refrain of the song. It was strange: as we released the song later, Frankie Goes To Hollywood was the #1 in the charts with "Relax" and we needed some weeks to edge them out. And Holly Johnson, the singer of the band, was the singer of this band "Big in Japan," who I stole the line some years ago, before he came to Frankie Goes To Hollywood."