Skip to main content

Similarity Detection

The system uses artificial intelligence to automatically detect and group similar grievances, facilitating identification of recurring issues and batch processing.

Overview​

┌─────────────────────────────────────────────────────────────────────────┐
│                   SIMILARITY PIPELINE                                    │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  GRIEVANCES       VECTORIZATION      CLUSTERING       GROUPS            │
│  ┌──────────┐     ┌──────────┐      ┌──────────┐     ┌──────────┐       │
│  │ Text     │ ──▶ │ TF-IDF   │ ──▶  │ Cosine   │ ──▶ │ Similar  │       │
│  │ grievance│     │ Cohere   │      │ Similarity│     │ groups   │       │
│  └──────────┘     └──────────┘      └──────────┘     └──────────┘       │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

Technologies used​

The system combines two vectorization approaches:

MethodDescriptionAdvantages
TF-IDFTerm Frequency-Inverse Document FrequencyFast, works offline
Cohere EmbeddingsSemantic embeddings via Cohere APIDeep contextual understanding

Access​

Similarity detection is visible in:

  1. Grievance list: Grouping icon on similar grievances
  2. Grievance detail: "Similar grievances" section at bottom of page
  3. Reports: Grievance cluster analysis

Similar groups​

Display in list​

Grouped grievances are displayed with:

ElementDescription
Group iconIndicates group membership
Similar countBadge with number of linked grievances
Similarity scoreSimilarity percentage (70-100%)

Expand/Collapse groups​

In the grievance list:

  1. Click the ▶ icon to expand a group
  2. Similar grievances appear as a sub-list
  3. Click ▼ to collapse the group

Similarity threshold​

The system considers two grievances as similar if their score exceeds 70%.

ScoreInterpretation
90-100%Very similar (possible duplicate)
80-89%Similar (same issue)
70-79%Potentially related
< 70%Not grouped

Use cases​

Identify recurring issues​

  1. Access the grievance list
  2. Filter by category or region
  3. Observe similarity groups
  4. Identify recurring patterns

Batch processing​

For very similar grievances:

  1. Identify the group
  2. Process the main grievance
  3. Apply the same resolution to linked grievances
  4. Document cross-reference in each grievance

Detect duplicates​

Grievances with a score > 90% may be duplicates:

  1. Verify complainant information
  2. Compare creation dates
  3. If duplicate confirmed, merge or close one of the grievances

Configuration​

Clustering activation​

Clustering is enabled by default. To configure:

Environment variables:

VariableDescriptionDefault
COHERE_API_KEYCohere API key for embeddings-
SIMILARITY_THRESHOLDSimilarity threshold (0-1)0.7
ENABLE_SIMILARITYEnable/disabletrue

Degraded mode​

If Cohere API is unavailable:

  • The system uses TF-IDF only
  • Quality remains good for similar texts
  • Less performant on rephrased content

Technical operation​

Processing pipeline​

  1. Extraction: Grievance text (subject + description) is extracted
  2. Cleaning: Punctuation removal, normalization
  3. Vectorization: Transformation into numeric vector
  4. Similarity calculation: Cosine distance between vectors
  5. Clustering: Grouping by threshold

Automatic update​

Clustering is recalculated:

  • On creation of a new grievance
  • When description is modified
  • Via daily scheduled task (batch)

Statistics​

In reports​

The Reports > Grievances tab displays:

IndicatorDescription
Number of clustersTotal identified groups
Average sizeAverage grievances per group
Top clustersMost important groups
EvolutionRecurring issue trends

Analysis by category​

Clusters can be analyzed by:

  • Grievance category
  • Geographic region
  • Time period
  • Priority

Best practices​

:::tip Recommendations

  1. Examine large clusters: They indicate a systemic issue
  2. Verify duplicates: Score > 90% = manual verification
  3. Document links: Reference linked grievances in comments
  4. Process as group: Similar grievances often deserve a coordinated response :::

Required permissions​

ActionPermission
View similar groupsplaintes.view
Configure clusteringparametres.edit

Limitations​

  • Clustering works better with detailed descriptions
  • Very short grievances may be misclassified
  • Mixed languages (Portuguese/Creole) may affect accuracy

See also​