Reliability, Validity and Ethics

ES EN

Soft Exclusion Coding Protocol · Monte Carlo Research Lab

Methodological safeguards supporting consistent coding, cautious interpretation, and traceable and ethical documentation of soft spatial exclusion mechanisms.

Protocol navigation:
Overview · Theory · Taxonomy · Coding method · Empirical validation · Reliability and validity · Research materials

Application status and scope of this page

The current SECP corpus release contains final results, reviewed and ratified by the responsible researcher. Internal review involves a single coder. Intercoder reliability has not been estimated for this release.

This page distinguishes the internal controls documented for the current corpus from procedures recommended for applications involving multiple coders and from approaches to validity assessment. Including a procedure in the protocol does not mean that it was implemented in this release.

The final status of corpus decisions is not equivalent to an independent reliability estimate. Likewise, empirical application and operational review do not, by themselves, establish all the dimensions of validity described below.

Internal controls documented for the current corpus

Application of explicit definitions and rules from the SECP codebook.
Internal review of positive, negative, uncertain and borderline decisions.
Clarification of category boundaries and review of alternative explanations.
Separation of evidence confidence from restrictive intensity.
Traceability between observation opportunity, evidence and final decision.

These controls describe the operational review of the corpus. They do not constitute a measurement of intercoder agreement or a formal estimate of intracoder reliability through recoding. Their scope is documented on the empirical validation and final results page.

Recommended intercoder reliability procedure

The following procedure is proposed for applications involving two or more coders. It is not presented as a procedure implemented in the current corpus release.

Develop and maintain the coding manual.
Train coders using positive, negative and ambiguous examples.
Conduct photographic and documentary calibration exercises.
Complete a shared pilot exercise, separate from the sample used to estimate independent agreement.
Define the sample, units, variables and treatment of non-assessable observations.
Independently code a common sample.
Calculate agreement before any discussion or adjudication.
Preserve original coding decisions and maintain a disagreement register.
Review definitions when recurring interpretation problems emerge.
If the manual changes, evaluate the revised version using a new independently coded subsample.

Reliability measures for applications with multiple coders

The following measures are methodological options. This page does not report coefficients calculated for the current corpus. The choice must be justified according to the variable, number of coders, sampling design and treatment of missing data.

Data type	Possible measure	Purpose and conditions
Binary presence	Percentage agreement and Cohen’s kappa	Describe agreement on presence or absence. Kappa accounts for expected chance agreement in a two-coder design. Uncertain or non-assessable observations must not automatically be treated as absences.
Nominal categories	Cohen’s kappa or Krippendorff’s alpha	Assess assignment consistency according to the design. Where multiple mechanisms may coexist, define assessment by category without imposing a single mutually exclusive category.
Ordinal intensity scores	Weighted kappa	Distinguish discrepancies of different magnitudes in two-coder designs, reporting the weights used.
Multiple coders or incomplete observations	Krippendorff’s alpha	Estimate agreement using a distance function appropriate to the measurement level and sufficient shared information. Missing data are not automatically replaced with zeros.
Continuous or aggregated spatial-unit scores	Intraclass correlation coefficient	Consider this measure where the scale and design support it, specifying the model, unit of estimation and whether consistency or absolute agreement is assessed.

Criteria for interpreting reliability

No universal numerical scale is applied interchangeably to kappa, alpha and intraclass correlation. Acceptance criteria must be justified before analysis for the selected measure and the intended use of the results.

Aspect	Information to document
Design	Number of coders, independence of decisions, sample selection and units assessed.
Measure	Coefficient, variant, weights or model, and rationale for the choice.
Category distribution	Prevalence, imbalances, rare categories and non-assessable observations.
Precision	Sample size and estimation uncertainty, where this can be calculated appropriately.
Disagreements	Discrepancy patterns and their implications for interpretation and manual revision.
Methodological decision	Previously justified criteria for accepting, revising or limiting the use of a variable.

A coefficient must be interpreted alongside the empirical design and observed disagreements. These criteria do not represent reliability results obtained for the current corpus.

Recommended disagreement procedure

This procedure applies to studies involving multiple coders. Reconsideration by the responsible researcher in the current corpus constitutes internal review, not disagreement between independent assessors.

Preserve the original decisions of all participating coders.
Calculate and record agreement on those independent decisions before adjudication.
Determine whether disagreement concerns observation, interpretation or application of a rule.
Review the evidence and reapply the manual and decision tree.
Record the adjudicated decision and its rationale separately from the original coding.
Revise a definition when disagreement reveals a recurring problem.
If assessing the effect of a manual revision, use new independent coding and document it as a separate assessment.

Consensus reached after adjudication must not be presented as independent reliability: it is the outcome of resolving disagreements.

Scope of validity assessments

The following sections describe dimensions of validity and criteria for examining them. They do not assert that all dimensions have been assessed through separate studies. Every validation claim must be linked to the procedure, evidence and results actually documented.

Content validity

Content validity examines whether the SECP taxonomy adequately represents the forms of indirect spatial filtering included in the construct of soft spatial exclusion.

Relevant expert perspectives for an assessment:

urban sociology;
tourism studies;
architecture and urban design;
accessibility;
observational methodology;
luxury tourism territories;
specific spatial knowledge of Monaco.

Assessment should focus on conceptual coverage, operational boundaries and relevant mechanisms that cannot be adequately classified through SE1–SE12. This list identifies relevant perspectives; it does not establish the participation of an external expert panel in the current corpus.

Construct validity

Construct validity examines whether categories and variables coherently represent the theoretical phenomenon they are intended to measure.

Assessment should examine whether:

categories are related but remain distinguishable;
intensity provides information distinct from the number of mechanisms;
evidence confidence remains separate from restrictive intensity;
soft exclusion and direct exclusion are analytically differentiated;
mechanism configurations provide information beyond individual categories.

The corpus boundary rules contribute to operational coherence. They do not, by themselves, constitute comprehensive construct validation.

Convergent validity

Convergent validity examines relationships with conceptually related indicators without requiring identical results.

Potential comparison sources:

Visibility–Accessibility Matrix classifications;
observed behavioural responses;
route-use patterns;
documented access conditions;
qualitative evidence on perceived exclusion;
specific measures of spatial exclusion where appropriate.

These sources describe possible assessments. Listing them does not imply that convergent analyses have been conducted using all of them.

SECP and VAM are complementary, not equivalent. Low VAM accessibility does not, by itself, constitute evidence of soft spatial exclusion. The presence of an SECP mechanism does not automatically determine an accessibility score.

Discriminant validity

Discriminant validity assessment should examine the distinction between soft spatial exclusion and:

low accessibility;
high commercial prices;
direct prohibition;
private ownership;
security presence;
luxury aesthetics;
architectural monumentality;
ordinary service differentiation.

None of these elements is sufficient on its own to confirm soft spatial exclusion. The relationship between a mechanism and an affected spatial practice must be documented.

This rule operates as a coding safeguard. Its application is not equivalent to an independent study of discriminant validity.

Known-case validity

This assessment would examine whether the protocol distinguishes configurations with different conditions of access, staying, circulation or participation.

Comparative settings may include:

an open public square;
a public pedestrian route;
a luxury shopping arcade;
a restaurant terrace;
a luxury hotel lobby;
a private residential entrance;
a marina hospitality area;
a temporary event enclosure.

Expected differences must be established using prior theoretical and operational criteria, without retrospectively adjusting expectations to the results. The type of setting does not automatically determine the presence of exclusion.

This section describes a comparison design; it does not report a separate completed known-case study.

Temporal validity

Temporal assessment examines the conditions under which observations persist, vary or change over time.

Comparisons may include:

ordinary weekdays;
weekends;
high-demand periods;
event preparation;
event operation;
event dismantling;
post-event restoration.

Comparison is particularly relevant to SE10 · Temporal Exclusion and SE12 · Temporary Privatisation and Event Transformation. The periods actually documented must be identified. Having observations from different dates does not, by itself, demonstrate temporal stability or test–retest reliability.

Alternative explanations

An observation must not be classified as Confirmed soft exclusion when a non-exclusionary explanation is better supported and there is insufficient evidence of an indirect restrictive function.

Explanatory group	Examples
Safety	Emergency management, pedestrian protection and security alerts.
Accessibility	Inclusive routes, ramps and protective barriers.
Operations	Service provision, deliveries, maintenance and construction work.
Legal requirements	Capacity limits, fire safety requirements and event-related obligations.
Environmental conditions	Weather, material protection and environmental risks.
Privacy	Protection of residents, guests or confidential activities.
Planning	Traffic management and urban planning requirements.

These explanations must be assessed using contextual evidence. Merely invoking them neither confirms nor rules out a mechanism.

Negative-case register

The register supports documentation of observations initially compatible with an SECP mechanism whose review does not support confirmation, for example because an alternative explanation has stronger evidential support.

Assessed absence, insufficient evidence and lack of observation opportunity must remain distinct. An uncertain or non-assessable situation does not automatically become a case of confirmed absence.

Recommended register fields:

case identifier;
category assessed;
initial interpretation;
alternative explanation;
contradictory evidence;
available behavioural evidence;
final decision and rationale;
implications for the coding manual;
implications for the theoretical framework.

Negative cases contribute to critical examination of the protocol and help reduce confirmation bias. Their review does not replace an independent reliability estimate or, by itself, demonstrate discriminant validity.

Ethical protocol

The following rules guide observation, evidence handling and publication of materials.

Observation

Focus observation on spatial processes rather than personal identities.
Avoid collecting identifying information unnecessary for the research.
Observe only under ethically and legally appropriate conditions.
Do not interfere with access, services or staff activities.

Photography

Prioritise entrances, routes, signage, barriers and spatial configurations.
Minimise the presence of identifiable people where unnecessary for the evidence.
Anonymise or exclude identifiable people where appropriate.
Store original research files securely.

Interactional evidence

Record observable behaviour without attributing unsubstantiated intentions.
Do not publish names or unnecessary physical descriptions.
Do not classify an interaction as discriminatory without sufficient evidence.
Preserve the context needed to assess alternative explanations.

Sensitive characteristics

Do not infer from appearance:

social class;
income;
ethnicity;
nationality;
religion;
disability;
sexual orientation;
political identity;
other protected characteristics.

Data management

Use anonymised case and mechanism identifiers.
Keep original evidence in access-controlled locations.
Distinguish original files from published materials.
Apply data minimisation principles.
Establish retention and deletion procedures where appropriate.
Remove unnecessary personal information before publication.

Cases requiring additional review

Certain decisions require further examination because of their intensity, ambiguity or conceptual relevance. In the current corpus, this review is internal and undertaken by the responsible researcher. It does not imply the involvement of a second independent coder.

Restrictive intensity coded at level 4.
Explicitness coded at level 3.
Simultaneous direct exclusion and soft exclusion within the same sequence.
Uncertainty about the primary category.
Equally plausible alternative explanations.
Evidence based mainly on subjective interpretation.
Unverified public, semi-public or private access status.

In applications involving multiple coders, a disagreement exceeding one intensity point should also trigger review. This is a rule for that design, not a description of disagreements recorded in the current corpus.

Connection with empirical validation

The empirical validation page presents final results and the documented scope of corpus review. This page distinguishes those internal controls from recommended procedures for estimating independent agreement and from other dimensions of validity.

References to reviewed or validated results must be understood within the explicitly documented procedure. They do not imply the existence of agreement coefficients, external assessors or separate validity studies that have not been reported.

Explore the SECP Empirical Validation & Final Results →

Continue to the research materials →

Reliability, Validity and Ethics · Soft Exclusion Coding Protocol