He is especially challenging because of his speed and his weapon's wide range so when fighting him, you have to keep on the defensive and attack him whenever there's an opening. This battle will be easier once you have enough stamina to evade several times and land some attacks. Don't bother blocking his attacks if your character is not efficient in handling shields since it will consume a lot of stamina. When you see him readying for a charge, move towards him and evade to his right side (to the left of your character). This will ensure that you'll evade his upward slash and attack him from behind.
There are ballistas here that can fatally wound him and allow you for a quick kill but lining it up and timing your fire can be challenging, not to mention that the ballistas can be destroyed. Your best hope is to evade and attack as soon as his combos stop. When you have a stronger weapon and higher stats, you should be able to defeat him and survive his attacks.
If you examine the nest in the area behind the Pursuer, you'll be taken by a bird to Lost Bastille. Fortunately, there's a bonfire from where you'll land that you can use to travel back if you selected it by mistake.
In the same area, carefully drop down the broken part of the wall to find a corpse. Loot it to find a Drangleic Sword, Drangleic Shield and a Drangleic armor set. The Drangleic Shield provides a 100% damage reduction rate, perfect for some upcoming boss battles. Its low STR requirement makes it a very good shield for almost any melee-based class early on.