This is where the game officially begins. Once you are allowed to leave the capsule, take a left straight through the archway and then another left at the massive statue into a small chapel. Here must you enter the tiny room to the right to pick up the game's very first Voxophone.
That will be all for the collectibles in this area so head toward the congregation to proceed with the baptism. Press Up on the d-pad if you need help with directions; the objective marker will show you where to go. It is of course recommended to explore the rest of the area for money and items.
After the baptism and brief dream sequence, Booker conveniently awakens inside the city of Columbia. You'll want to completely ignore the objective marker for the moment and focus on exploring, before you meet any form of resistance, which can no doubt make checking things out difficult. At New Eden Square, keep an eye out for a boutique called Hudson's. There is a Voxophone behind the counter inside that shop.
Also don't forget to use the Kinetoscope machine next to the front counter as well, as it counts towards your progress for the Sightseer achievement.
Head past the aerial parade gate next. There is a Kinetoscope next to a The False Shepherd poster near the grocery store. After that, look for a shop called Chas A. White Supply from which you can head to the nearby garden to find a Voxophone on the table under the umbrella.
Moving on to the Fairgrounds after Booker receives the telegram, look to your left for a Telescope. Once again, this counts toward the Sightseer achievement's progress. Another Kinetoscope is at the gate near the beginning of the street where the carnival game stalls are at.
Also, do take the time to try and claim all the prizes in all of the carnival games. You will need all the cash you can get. Plus these games give you the opportunity to practice with guns (Carbine and Shotgun) as well as Vigors. When you're done, grab the Possession Vigor from the woman at the ticket machine then use your new ability on the machine so you can pass without buying a ticket.
At the Path to the Scrolls, follow the objective marker when you're done scavenging for loot, and you'll eventually be told to look at something. On the bench beneath the statue where this sequence occurs is a Voxophone. Proceed to the Raffle Park now to start the next chapter.