A bit complicated, so bear with me...
During the 1960s, American foreign policy was dominated by three unquestioned dogmas:
Containment, The Domino Theory & The Communist Monolith. These would combine to catastrophic effect...
Containment (articulated in the 1940s by George Kennan) maintained that the west could defeat the Soviet Union by countering their movements to expand. This manifested itself in a series of alliances designed to rein in the Soviet Empire (NATO, ANZUS, SEATO, etc).
The Domino Theory maintained that if one country in a region fell to communism the other nations in that region would all likely be threatened and would fall like a series of dominoes.
And finally, the Communist Monolith posited that international communism was a singular entity, all run effectively out of the Kremlin. It denied the overarching influence of historical, nationalistic and cultural factors which might make cooperation amongst communist nations difficult if not impossible to achieve and sustain in the long run.
An unquestioned acceptance of these three paradigms convinced Lyndon Johnson that American involvement in securing South Vietnam was imperative.
Initially, the USA became involved on the basis of Secretary of Defence Robert McNamara's rosy assessment that the war could be won by 1966. However, he hadn't counted on massive aid to North Vietnam pouring in from the Soviet Union. The Soviets actually wanted the USA involved in Vietnam in the same way that the USA wanted the Soviets to invade Afghanistan in 1979. (Zbigniew Brzezinski boasted how he euchred the Soviets into that war).
Why did the Soviets want this? Because since the late 1950s a crisis in foreign policy in China was threatening Sino-Soviet relations. There was a debate raging in Beijing over which course of modernisation to take. One group wanted greater ties with the Soviet Union, and one group simply didn't trust the Soviets (for historical reasons) and wanted to strengthen trade and cooperation with western nations. The Soviets wanted to stop this at all costs -- hence their increased aid to the North Vietnamese -- which killed two birds with one stone: it kept America busy and distracted, AND it drove a wedge between China and the West.
Meanwhile, Back in the USA...
Increased aid to the North Vietnamese, coupled with a severe underestimation of popular resistance from the NVA and VC, required a recalculation of victory on McNamara's part. Even with projected build ups to counter the North, McNamara calculated in late 1966 that the war could not be won until AFTER the 1968 presidential election. Lyndon Johnson only approved actions after he's been assured that victory could be achieved BEFORE 1968, and now he had to swallow the bitter pill. At that point, Johnson personally redirected American efforts away from victory and towards a negotiated stalemate.
America could have won the war, IF America had been willing to commit the resources and manpower necessary to see it through. They didn't do that, partially because victory couldn't be achieved before 1968, and partly because of the fear that an all out invasion of North Vietnam would have brought China into the war (Which it very well may have done).
There's actually more to this saga, but I think you get the overall drift... America was not willing to invest what it took to win, and the North Vietnamese were willing to risk everything to insure they didn't lose.