Anything is possible I think. At this point in history it seems like it would be more difficult than before because almost all learning aids other than Pimsleur or other CD/audio based only learning systems have reading involved. So if you didnt use anything else like websites, apps, books, textbooks, etc then someone may be able to manage it. But it would almost be like avoiding any written Japanese.
Most children learn to speak before they can read, and back in the day there were spoken languages only, not written, or even only a couple hundred years (maybe more recent than that even) reading was only a higher learning type of thing and lots of people only spoke languages. Learning to read at that time was less common than it is today.
With all that being said, I think it would possible, but I feel like even looking up words or anything would be difficult because I feel like you would almost have to read something at some point. If you only learned hiragana and katakana you wouldnt find a ton of media written out in that way, it would almost always use kanji as well. It is an interesting thing to think about thoughbut man it sounds really difficult and limiting, like I would only get to know a portion of what would be possible, or it would take a really long timehaha!