L. L. Bean is in Hibiya! (near Ginza)

But it's "Japan Fit" (sized for the Japanese consumer), so ironically it probably won't fit me very well. I have the same trouble with Eddie Bauer - in Japan they only carry women's Petite sizes, where the sleeves and torso length are a tad too short for me. Ah well. Zara, Gap and Banana Republic fit fine.