Shortly, around 9 a.m., on July 4, Union signal corpsmen spotted the westward movements of Confederate wagons moving down the Fairfield Road. These were the wagons of Lt. Gen. Ewell’s Corps, who were to follow behind Major Harman’s reserve train through Monterey Pass over South Mountain. They left Gettysburg from Oak Ridge, making their way down toward Black Horse Tavern, which was next to the Fairfield Road.

Upon receiving this information, Maj. Gen. Meade had limited options for an all out Union pursuit. His army was hungry and in rags. Many were shoeless and ill-equipped for an aggressive pursuit. His cavalry units at Gettysburg were in the same situation. Major General Meade, knowing that most of his cavalry was at Westminster, Maryland guarding his supply wagons, could only use what cavalry he had on the Gettysburg battlefield.

Major General Alfred Pleasanton was ordered to pursue the Confederate army’s wagon trains moving westward. He ordered Brigadier General Judson Kilpatrick’s Third Cavalry Division to locate which South Mountain pass the Confederates were using, and harass the retreating columns of wagons. Brigadier General Kilpatrick was also ordered to disrupt their line of communications.

Leaving Gettysburg around 10:00 a.m., Brig. Gen. Kilpatrick, with the brigades of Brigadier General George A. Custer and Colonel Nathaniel Richmond, moved to Emmitsburg, Maryland, arriving there at noon. Once there, Brig. Gen. Kilpatrick was reinforced by Colonel Pennock Huey’s brigade from Brigadier General David Gregg’s Second Cavalry Division. Brigadier General Kilpatrick’s force now consisted of 5,000 mounted horsemen, sixteen pieces of rifled artillery and at least one mile worth of wagons of supplies.

As the day continued at Gettysburg, both armies look upon each other from opposite sides of the battlefield. Knowing he had 4,000 Union prisoners that could slow his march back to Virginia, General Lee sent a dispatch to Maj. Gen. Meade asking for a prisoner exchange. Lee’s request was refused. Earlier on July 3, the Confederate army paroled 1,500 Union prisoners. Major General Meade knew that an army in retreat would be slowed with prisoners being escorted.

While the Army of the Potomac took care of burial details and the wounded, Maj. Gen. Meade ordered Major General William French in Frederick, Maryland to occupy and reinforce the South Mountain gaps of Turner’s Gap, Fox’s Gap and Crampton’s Gap to protect Washington and Baltimore. Major General Meade also ordered Major General Darius Couch’s Department of the Susquehanna to send troops to Cashtown. Major General William Smith’s division of New York State National Guard and Pennsylvania Militia were ordered to move down the South Mountain ridge near Carlisle to Cashtown Gap. From there, they could reinforce Maj. Gen. Meade’s army at Gettysburg, or proceed to pursue those retreating columns of the Confederate army into Maryland.

By nightfall on July 4, with heavy rain falling, the Confederate army began marching out onto the Fairfield Road. Lieutenant General A.P. Hill’s Corps moved out first. Following behind was Lt. Gen. Longstreet’s Corps. Once Lt. Gen. Longstreet’s Corps was on the road, then Lt. Gen. Ewell would close up the line and bring up the rear.

By 9:00 p.m. at Monterey Pass, the Union cavalry commanded by Brig. Gen. Kilpatrick attacked the retreating columns of Confederate wagons. As fighting escalated, more Confederate reinforcements arrived on scene, and by dawn of July 5, had secured Monterey Pass for the rest of the Confederate retreat. Brigadier General Kilpatrick moved to Ringgold, Maryland and halted just after daybreak. Taking inventory of prisoners and captured supplies, Brig. Gen. Kilpatrick moved to Smithsburg, within range of support from Maj. Gen. French. The Confederates lost 1,300 men taken prisoner, a dozen wounded, and several killed. Kilpatrick lost over 100 men that were killed, captured, or wounded.

Over at Cashtown, Brig. Gen. Imboden assembled the columns of wagons, including the wounded from Gettysburg. He encountered some minor skirmishing along the way during the night and early morning hours of July 5. Brigadier General Imboden was saddened as he heard the screams of wounded soldiers, begging for their lives to end. At Greencastle, some of the wagons were attacked by civilians, who, with axes, began cutting the spokes from the wheels.

By the morning of July 5, Maj. Gen. Meade knew that General Lee was on the move. He ordered the VI Corps to pursue him into South Mountain, while the rest of the Union army marched southward toward Frederick, and then turned westward toward Middletown. With the heavy rains, medical attention required for the wounded, burial details, and condition of his troops from the battle, his orders would be delayed by one day.

By late afternoon, Maj. Gen. Stuart’s cavalry, after moving through Emmitsburg during the early morning, was now moving through the Catoctin Mountain. Once he arrived at South Mountain at Raven Rock, he ran into Brig. Gen. Kilpatrick’s skirmishers just at the entrance of South Mountain, near Smithsburg. Union and Confederate cavalry skirmished for several hours. Artillery from both sides opened, keeping each checked. By nightfall, Kilpatrick falls back to Boonsboro, Maryland. Kilpatrick lost 5 men wounded and 3 men missing, while Stuart had 1 man killed and 3 men wounded.

On July 6, fearing the Confederate army was fortified in South Mountain near Monterey Pass and Fairfield Gap, Maj. Gen. Meade ordered the VI Corps to move directly to Emmitsburg for the pursuit. VI Corps commander, Major General John Sedgwick detached Brigadier General Thomas Neill’s Brigade and Colonel John McIntosh’s cavalry brigade. Their orders were to continue following the Confederate army without committing themselves to an all out fight. They marched all the way to Boonsboro, linking up with the Army of the Potomac a few days before the Confederate army retreated beyond the Potomac River into West Virginia.

During the morning at Boonsboro, Brig. Gen. Kilpatrick and Brigadier General John Buford coordinated two attacks with one another. Brigadier General Buford would attack the Confederate positions at Williamsport, while Brig. Gen. Kilpatrick attacked Hagerstown. Both commanders moved out of Boonsboro during the morning.

Nearing Hagerstown, Colonel Nathaniel Richmond’s brigade was the first to advance. Fighting quickly broke out on the city streets. Colonel Richmond’s brigade made it almost to the city hub where it was barricaded. Then troopers began fighting on other streets. Fighting from yard to yard and house to house took place. Within hours, Maj. Gen. Stuart’s cavalry was concentrated in Hagerstown, supported by artillery and infantry. By nightfall, the Battle of Hagerstown was over. Kilpatrick lost 21 killed, 59 wounded, and 220 missing. The Confederates had 11 killed, 1 mortally wounded, 50 wounded, and 38 missing.

As the attack on Hagerstown was erupting, Brig. Gen. Buford’s division, along with Custer’s brigade of Kilpatrick’s division began their advance on Williamsport. Brigadier General Imboden was made aware of the Union advance and began deploying his brigade, supported heavily by field artillery. He also had wagoneers and wounded men join in the defensive line.

By the afternoon, as several cannon from Buford’s division came out of the woods and deployed, they opened up on the Confederate position. The Confederate artillery responded. During the bombardment, the Confederate artillery began to run low on ammunition, which had to be ferried from the West Virginia shore to the Maryland side of the Potomac River.

Between 4 & 5 p.m., Confederate reinforcements had arrived on the field. By dusk, Kilpatrick’s men began to give way on the right flank at Hagerstown, which forced Buford to fall back toward Boonsboro. Night quickly covered the ground and Imboden had won the day. Buford had 1 killed, 3 mortally wounded, 5 wounded, and 172 missing. Imboden had 14 killed, 117 wounded, and 47 missing.