Conversations are a basic unit of analysis in studies of human interaction. These units are conventionally distinguished by reference to the set of ratified participants who take part, often by appeal to their physical proximity/orientation. We show that within such conversational units there are distinct dialogue contexts which are more fine-grained than this. We present experimental evidence that demonstrates how these dialogue contexts are distinguished not in terms of topic but in terms of the participants who are actively contributing. More specifically, we show that changes in the set of primary participants alters how easily participants can access a prior turn in the conversation using a context dependent elliptical expression. So even within small groups of fully ratified participants, distinct dialogue contexts, indexed to specific sets of participants, can emerge. This leads to the idea that in multi-party dialogue, context/adjacency has a tree like structure, so that under certain conditions, presented here, utterances on either side of a sub-dialogue (a stretch of talk during which the set of primary participants remains unchanged) can be regarded as 'pragmatically next' to each other. This has implications for our understanding of how conversations are structured and for how models of dialogue track what is salient for whom at any given point in conversation.