Hm... There's not a detailed answer here yet, so I guess I'll jump in.

First thing to realize: in reach retina, there's over 100 million photoreceptors, but only 1 million axons in the optic nerve. How's that work? The answer, is that it's completely accurate to say that the earliest part of visual processing is in the eye. There's 5 layers to the retina. The million ganglion cells sending the optic nerve to central command, a middle 'processing' layer with three main types of interneurons, and the actual photoreceptor cell layer (the other two layers are 'in between' layers made up of the cabling connecting these three cell body layers).

There's actually about 20 different kinds of ganglion cells, so you can look at it as there being 20 different image filters sent in to central. Some with blue/yellow contrast, others for pure luminance contrast, etc. Some of these 'views' have very tiny 'receptive fields' (the part of the retina their signal contains information for). So-called 'P' ganglions for example can have one single cone that causes them to depolarize and fire, though they all have surrounding parts/colors that inhibits firing. Each ganglion cell actually has a fairly complex 'optimal' pattern that makes them fire... Usually either 'light in the center, dark on the edges', the reverse of that, or a color version (blue light but no yellow light, for example). So even by the time the signal's leaving the eye, you're already getting various size views containing different kinds of contrast information. It should be said too, the closer to the fovea (center of the view) the smaller the receptive field gets, so the more detail you can perceive... So the 'resolution' of your view is actually not consistent even just across the eye.

Anyway. So: the optic nerve. These million axons aren't sending pixel information, they're sending 20 pictures, each with larger or smaller receptive fields depending on type and distance from the fovea, and different activation patterns they're 'looking for' to fire. This tract splits off on the way to the central switchboard (the LGN in the thalamus). For the left hemisphere of the LGN (say), you get the right eye's ear half of the view, and the left eye's nose half of the view. Opposite that for the right hemisphere.

So, the left hemisphere switchboard gets input from the right half of the visual field. But these two views don't perfectly line up... Maybe 80% of that view does, but the most peripheral part of the view only gets a signal from the outside edge, not the nose-side edge, so your peripheral vision doesn't have a binocular signal to put together in the first place.

Input from different eyes is totally separate still in this central switchboard. Each hemisphere's LGN has 6 main layers, 3 from each eye, with thin middle layers in between carrying the comparatively small amount of color information (koniocellular layers), also still separated by eye.

Among other places (pupil and eye muscle autonomic circuits in particular) this tract then gets sent to the primary visual cortex in V1, at the very back of your head. Here, it's still separated by eye. You've got these stripes running from the back to the front, with alternating left eye, right eye input. Perpendicular to that, you've got simple edge detectors going through different orientations, and mixed into all that, you've got these barrel shaped blobs of cells that respond to color information. You can see a picture of what I mean here. One 'cycle' of left eye/right eye, and 180 degrees of orientation preference for edges marks out a roughly 1mm x 1mm 'hypercolumn', that takes in input from a chunk of the retina. These hypercolumn are the basic processing unit here in V1, and they tile over the visual field. The receptive field here is larger than a single cone or rod certainly, but it's still fairly small. You can see that the same parts of the visual field are at least nearby now though, that's what those stripes are... Lined up regions from the same part of the visual field from different eyes.

Once it gets to V2, the next layer of processing, this is where you start to have 'binocular integration', neurons that selectively fire based on visual input from both eyes.

As you climb up in levels, you'll see larger and larger receptive fields. By the time you get deep into the ventral visual stream (very loosely speaking, ventral stream is identifying things you're seeing, dorsal stream is for helping to guide hands and stuff for grabbing things and so on) you're seeing cells that selectively fire given complex input from anywhere in the visual field. A 'Jennifer Aniston' neuron for example that might fire anytime you're seeing her face... Or her name written, or a drawing of her, or so on, from anywhere in the visual field.

But anyway. You get the gist. The full complex view of even the early visual system is hilariously intricate, and there's no really simple description that captures all the detail. But maybe a partway answer that's close enough... Peripheral vision has no binocular component, since only one eye captures that outside edge. For the bulk of your visual field though, yes... Things get tied together eventually, but it's not until V2... After many layers of processing in the retina, LGN, and V1. By then, you're talking about receptive fields with hundreds of input photoreceptors, and you're already talking about integrating signals for fairly high level information... Direction of movement, edge orientations, color information and so on, all in separate parallel feeds, to be integrated into even more high level, abstract tracts of information with even larger receptive fields as you continue climbing up towards the dorsal (hand eye coordination) and ventral (object recognition) tracts.

Note too: every step here is more tightly interconnected than I'm describing. In particular, there's 10x connections coming BACKWARDS from downstream than there are coming in from upstream towards the retina, so you probably do have some binocular signal affecting neuron firing even before you get to the binocular integration part in V2. Those incredibly numerous backpropagating connections aren't well understood, but it does definitely complicate the question of where you can start pointing to neurons influenced by input from both eyes in the same part of the visual field.

So anyway... There you go, haha. This is largely cobbled together from Kandel's 'principles of Neuroscience, 6th edition', there's a half dozen chapters in the low 20's going through how the brain processes visual information in pretty heavy detail.