According to Songfacts, When I'm With You was inspired by singer-guitarist Bethany Cosentino's memories of growing up in Los Angeles. "I listen to a lot of '60s beach music," she explained to Spin Magazine: "It sounds happy and innocent, but it was made during a time that wasn't happy or innocent at all."
The world is lazy
But me and you
We're just crazy
So when I'm with you I have fun