Understanding the Incident Lifecycle and the Genesis of Duplicates

Before we can fix duplicates, it’s crucial to understand what an incident is and how it fits into the broader service management landscape. As defined, an incident is a sudden interruption to a service or a reduction in the quality of a service. For an employee, it’s that moment when their laptop freezes, the printer stops working, or they can’t access a critical application. Their immediate recourse is to reach out for support by creating an incident record.

Now, let’s consider how duplicates arise. Often, it’s not a malicious act but a consequence of user behavior, system limitations, or communication breakdowns:

• User Frustration and Multiple Attempts: An employee facing a persistent issue might try reporting it multiple times through different channels (e.g., email, phone, self-service portal) before receiving a confirmation or seeing their issue being addressed.
• System Glitches or Integrations: Sometimes, automated systems or integrations that create incidents might fire multiple times due to network hiccups or faulty logic, resulting in identical or near-identical records.
• Lack of Clear Identification: Without robust mechanisms to identify unique issues, users might inadvertently create new incidents for problems that are already being worked on.
• Confusion with Problems: The line between an incident and a problem can sometimes blur, especially for end-users. An employee experiencing a recurring issue might keep creating new “incidents” when a more appropriate approach would be to report it as a recurring issue that needs to be investigated as a problem. As we know, a problem is the underlying cause of one or more incidents, and if the same issue repeatedly happens to the same employee, it’s a problem. If it’s happening to multiple people, it’s often treated as a major incident that might trigger a problem investigation.

Effectively managing these situations requires a multi-pronged approach, focusing on prevention, detection, and remediation.

Preventing the Influx: Strategies for Proactive Duplicate Management

The best way to deal with duplicates is to stop them from happening in the first place. This involves a combination of user education, system configurations, and intelligent automation.

1. Enhancing the User Experience for Incident Creation

A smooth and intuitive incident submission process is your first line of defense. If users can easily find existing information or get clear guidance, they are less likely to create unnecessary duplicates.

• Robust Knowledge Base Integration: Ensure your self-service portal prominently features a well-maintained knowledge base. When a user starts typing a description, the system should proactively suggest relevant articles that might already address their issue. This diverts them from creating a new incident if a solution already exists.
• Intelligent Search and Suggestions: Implement search capabilities that can identify similar existing incidents based on keywords, affected CIs, or user descriptions. Prompt users with “Are you trying to report this issue?” before they proceed with creating a new record.
• Clear Communication Channels: Provide clear instructions on how and where to report issues. Emphasize the importance of checking for existing reports before submitting a new one.

2. Leveraging System Configurations and Automations

Your ITSM platform, like ServiceNow, offers powerful tools to prevent duplicates programmatically.

• Duplicate Detection Rules (e.g., using Business Rules): This is where scripting and intelligent logic come into play. You can configure business rules to run *before* an incident is inserted or updated. These rules can scan for existing incidents that share similar characteristics (e.g., same caller, same affected CI, similar short description within a certain timeframe).
• Record Producers with Smart Logic: If you use Record Producers for incident creation, embed logic within them. For instance, before allowing submission, a Record Producer can query for existing open incidents matching key criteria. If a match is found, it can guide the user to the existing incident or prompt them to add details to it.
• Email Ingestion Rules: When incidents are created via email, implement sophisticated parsing and matching. If an email subject and body closely resemble an open incident, the system should ideally link it to the existing one or notify the user to check for an existing ticket.

3. Differentiating Incidents, Problems, and Changes

Misunderstanding the purpose of these record types is a common cause of duplicate and inappropriate record creation. Clear definitions and training are essential.

• Incident: A single occurrence of a service disruption. (Reference 19)
• Problem: The underlying cause of one or more incidents. If the same issue happens repeatedly, it’s a problem. As noted, if the same problem is happening to multiple people, it might be treated as a major incident, and a problem record will be created for investigation. (Reference 20)
• Change Request: A formal process to manage modifications to services or the IT infrastructure. Support engineers might raise a change request from an incident if they identify a need for a system alteration to resolve the issue permanently. (Reference 22)

By educating users and support staff on these distinctions, you reduce the likelihood of creating duplicate incidents when a problem investigation or a change request is more appropriate.

The Technical Arsenal: Scripting Solutions for Duplicate Detection and Resolution

While preventative measures are paramount, sometimes duplicates slip through. This is where robust technical solutions become indispensable. We’ll explore how to implement these using ServiceNow’s scripting capabilities, drawing upon the provided references.

3.1. Before Business Rule for Real-time Duplicate Detection

A “before” business rule is your most powerful tool for preventing duplicates *before* they hit the database. This rule will trigger just before an `insert` or `update` operation and can check for existing similar records.

Scenario: An employee submits a new incident. We want to check if another open incident exists with the same caller and a very similar short description.

Implementation (ServiceNow – Before Business Rule):

/**
 * Business Rule: Prevent Duplicate Incident Creation
 * Table: Incident
 * When: Before
 * Insert: true
 * Update: false (usually handled by other rules or specific duplicate scenarios)
 *
 * This rule aims to detect and prevent the creation of duplicate incidents
 * by checking for existing open incidents with the same caller and a similar
 * short description.
 */

(function executeRule(current, previous /*, grmember, state, system */ ) {

    // Get the caller and short description from the current incident
    var callerSysId = current.getValue('caller_id');
    var shortDescription = current.getValue('short_description');

    // If caller or short description is empty, we can't perform a meaningful check
    if (!callerSysId || !shortDescription) {
        gs.debug("Incident duplicate check skipped: Caller or Short Description is missing.");
        return;
    }

    // Normalize the short description for better matching (e.g., lowercase, remove common words)
    // This is a basic normalization; more advanced techniques might involve NLP libraries.
    var normalizedShortDescription = shortDescription.toLowerCase().replace(/[^a-z0-9\s]/g, '').trim();
    var keywords = normalizedShortDescription.split(/\s+/);

    // Limit the number of keywords to check to avoid performance issues and false positives
    var maxKeywordsToCheck = 5;
    var descriptionKeywords = keywords.slice(0, maxKeywordsToCheck);

    // Create a GlideRecord to search for existing open incidents
    var grIncident = new GlideRecord('incident');
    grIncident.addActiveQuery(); // Only consider active incidents
    grIncident.addQuery('caller_id', callerSysId);

    // We need to be smart about matching the short description.
    // Exact matches are rare. We can use 'LIKE' or build a more complex query.
    // For simplicity and effectiveness, we'll add a few key terms.
    // A more robust approach might involve full-text search or similarity scoring.

    // Add query for keywords in short description. This is a simplified approach.
    // In a real-world scenario, you might want to construct a dynamic query
    // or use advanced search features if available.
    var orQuery = grIncident.addOR();
    for (var i = 0; i < descriptionKeywords.length; i++) {
        if (descriptionKeywords[i].length > 3) { // Ignore very short words
            orQuery.add('short_description', 'CONTAINS', descriptionKeywords[i]);
        }
    }
    // Ensure the OR condition actually has clauses if none of the keywords were long enough.
    if (orQuery.getOperator() === 'OR' && orQuery.getQuery().indexOf('short_description') === -1) {
        grIncident.addQuery('short_description', 'DOES NOT EXIST'); // effectively a no-op if no keywords found
    }


    grIncident.query();

    var duplicateFound = false;
    var existingIncidentSysId = '';
    var existingIncidentNumber = '';

    while (grIncident.next()) {
        // Double-check the description similarity if multiple matches are found
        // This is a simplified similarity check. For production, consider Levenshtein distance or other algorithms.
        var existingShortDesc = grIncident.getValue('short_description').toLowerCase().replace(/[^a-z0-9\s]/g, '').trim();
        var similarityScore = calculateStringSimilarity(normalizedShortDescription, existingShortDesc); // Helper function needed

        // Adjust the threshold for similarity as needed. 0.7 (70%) is a starting point.
        if (similarityScore > 0.7) {
            duplicateFound = true;
            existingIncidentSysId = grIncident.getUniqueValue();
            existingIncidentNumber = grIncident.getValue('number');
            break; // Found a strong duplicate, no need to check further
        }
    }

    if (duplicateFound) {
        // Add an error message to the user
        gs.addErrorMessage('A similar incident (' + existingIncidentNumber + ') already exists for this issue. Please check ticket ' + existingIncidentNumber + ' for updates.');

        // Abort the action to prevent the duplicate from being created
        current.setAbortAction(true);
    }

})(current, previous);

/**
 * Helper function to calculate string similarity (e.g., Jaccard index or simple overlap).
 * This is a basic example; for production, consider a more robust algorithm.
 * Example using simple word overlap:
 */
function calculateStringSimilarity(str1, str2) {
    var set1 = new Set(str1.split(/\s+/).filter(word => word.length > 0));
    var set2 = new Set(str2.split(/\s+/).filter(word => word.length > 0));

    if (set1.size === 0 || set2.size === 0) {
        return 0;
    }

    var intersection = new Set([...set1].filter(x => set2.has(x)));
    var union = new Set([...set1, ...set2]);

    return intersection.size / union.size;
}
        

Troubleshooting this Rule:

• False Positives: If the rule flags legitimate new incidents, refine the similarity calculation or the keywords used. Ensure that `addQuery(‘short_description’, ‘CONTAINS’, …)` is not too broad. You might need to exclude common words or implement more advanced text analysis.
• False Negatives: If duplicates are still being created, the matching logic might be too strict. Consider expanding the search criteria, using fuzzy matching algorithms, or increasing the number of keywords checked.
• Performance: Complex string comparisons or very broad queries can impact performance. Test thoroughly and optimize the `addQuery` statements. Consider indexing relevant fields.

3.2. Cleaning Up Existing Duplicates: An Automated Approach

For duplicates already in the system, a scheduled job combined with a script is often the most efficient way to clean them up.

Scenario: Periodically run a script that identifies and closes duplicate incidents that share the same caller, affected CI, and have a very similar short description, while ensuring we keep the oldest one.

Implementation (ServiceNow – Scheduled Job):

First, create a Script Include to house your duplicate detection and resolution logic. This promotes reusability and cleaner code.

/**
 * Script Include: IncidentDuplicateManager
 * Description: Provides methods for identifying and resolving duplicate incidents.
 */
var IncidentDuplicateManager = Class.create();
IncidentDuplicateManager.prototype = {
    initialize: function() {
        // Initialization logic if needed
    },

    /**
     * Finds and potentially closes duplicate incidents.
     * Keeps the oldest incident and closes the newer ones.
     * @returns {number} The number of duplicate incidents closed.
     */
    findAndCloseDuplicates: function() {
        var closedCount = 0;
        var processedIncidentSysIds = new Set(); // To avoid processing the same incident multiple times

        var grIncident = new GlideRecord('incident');
        grIncident.addActiveQuery(); // Only consider active incidents
        grIncident.orderBy('sys_created_on'); // Process older incidents first
        grIncident.query();

        while (grIncident.next()) {
            var currentIncidentSysId = grIncident.getUniqueValue();

            // Skip if this incident has already been processed as a duplicate
            if (processedIncidentSysIds.has(currentIncidentSysId)) {
                continue;
            }

            var callerSysId = grIncident.getValue('caller_id');
            var shortDescription = grIncident.getValue('short_description');
            var affectedCI = grIncident.getValue('cmdb_ci'); // Also consider affected CI

            if (!callerSysId || !shortDescription) {
                continue; // Cannot determine duplicates without key info
            }

            var normalizedShortDescription = shortDescription.toLowerCase().replace(/[^a-z0-9\s]/g, '').trim();
            var descriptionKeywords = normalizedShortDescription.split(/\s+/).filter(word => word.length > 3).slice(0, 5); // Limit keywords

            var grDuplicateSearch = new GlideRecord('incident');
            grDuplicateSearch.addActiveQuery();
            grDuplicateSearch.addQuery('caller_id', callerSysId);

            // Add query for affected CI if it exists
            if (affectedCI) {
                grDuplicateSearch.addQuery('cmdb_ci', affectedCI);
            } else {
                // If CI is not mandatory, we might broaden the search or exclude this criteria.
                // For now, we'll assume if CI is a strong indicator, we require it for matching.
                // Or, we can use a less strict query if CI is missing.
            }

            // Add description keywords - similar logic to the before rule
            var orQuery = grDuplicateSearch.addOR();
            for (var i = 0; i < descriptionKeywords.length; i++) {
                orQuery.add('short_description', 'CONTAINS', descriptionKeywords[i]);
            }
             if (orQuery.getOperator() === 'OR' && orQuery.getQuery().indexOf('short_description') === -1) {
                grDuplicateSearch.addQuery('short_description', 'DOES NOT EXIST');
            }

            // Exclude the current incident from the search
            grDuplicateSearch.addNotInQuery('sys_id', currentIncidentSysId);

            grDuplicateSearch.orderBy('sys_created_on'); // Find the oldest potential duplicate
            grDuplicateSearch.query();

            while (grDuplicateSearch.next()) {
                var potentialDuplicateSysId = grDuplicateSearch.getUniqueValue();
                var potentialDuplicateShortDesc = grDuplicateSearch.getValue('short_description').toLowerCase().replace(/[^a-z0-9\s]/g, '').trim();
                var similarityScore = calculateStringSimilarity(normalizedShortDescription, potentialDuplicateShortDesc); // Re-use helper

                // If a strong duplicate is found (and it's newer than currentIncidentSysId, which is guaranteed by orderby)
                if (similarityScore > 0.7) {
                    gs.info("Found duplicate: Keeping " + grIncident.getValue('number') + " (Created: " + grIncident.sys_created_on + "), Closing " + grDuplicateSearch.getValue('number') + " (Created: " + grDuplicateSearch.sys_created_on + ")");

                    // Close the duplicate incident
                    var grToClose = new GlideRecord('incident');
                    if (grToClose.get(potentialDuplicateSysId)) {
                        grToClose.state = 7; // Assuming 7 is the 'Closed' state value
                        grToClose.work_notes = "Duplicate incident. Automatically closed by scheduled job. Original incident: " + grIncident.getValue('number');
                        grToClose.update();
                        closedCount++;
                        processedIncidentSysIds.add(potentialDuplicateSysId); // Mark this as processed
                    }
                }
            }
        }
        return closedCount;
    },

    type: 'IncidentDuplicateManager'
};

/**
 * Helper function to calculate string similarity (same as before).
 */
function calculateStringSimilarity(str1, str2) {
    var set1 = new Set(str1.split(/\s+/).filter(word => word.length > 0));
    var set2 = new Set(str2.split(/\s+/).filter(word => word.length > 0));

    if (set1.size === 0 || set2.size === 0) {
        return 0;
    }

    var intersection = new Set([...set1].filter(x => set2.has(x)));
    var union = new Set([...set1, ...set2]);

    return intersection.size / union.size;
}
        

Then, create a Scheduled Job:

• Name: Daily Incident Duplicate Cleanup
• Run: Daily (e.g., at 2 AM)
• Script:

  
  var duplicateManager = new IncidentDuplicateManager();
  var closedIncidents = duplicateManager.findAndCloseDuplicates();
  gs.info("Daily Incident Duplicate Cleanup: Closed " + closedIncidents + " duplicate incidents.");
                  

Troubleshooting this Scheduled Job:

• Incorrect Closures: Review the `calculateStringSimilarity` function and the `0.7` threshold. Adjust it based on your data. Also, ensure that the criteria for matching (caller, description, CI) are appropriate. If CI is not always present, the logic needs to be more flexible.
• Performance Issues: For very large incident volumes, the `query()` and nested `while` loops can be slow. Consider optimizing queries, processing in batches, or using more advanced indexing. Limiting the number of keywords searched is crucial.
• No Incidents Closed: Verify that your similarity threshold is too high or that the matching criteria are too strict. Check that the `state` value `7` is indeed the ‘Closed’ state in your instance.
• Skipping Records: Ensure the `processedIncidentSysIds` set is working correctly to prevent re-processing.

3.3. Handling Incident Tasks and State Transitions

It’s critical to ensure that related records are handled correctly during incident closure. References 26 and 27 provide excellent examples.

Closing Parent Incidents (Reference 26):

An “after” business rule ensures that when a parent incident is closed, all its child incidents are also closed.

// Business Rule: Auto-close child incidents
// Table: Incident
// When: After
// Update: true
// Condition: current.state.changesTo(7) && current.parent == '' // Assuming state 7 is Closed

if (current.state == 7 && current.parent == '') { // Check if current incident is closed and is a parent
    var grChild = new GlideRecord('incident');
    grChild.addQuery('parent', current.sys_id);
    grChild.query();

    while (grChild.next()) {
        if (grChild.state != 7) { // Only update if not already closed
            grChild.state = 7; // Set the state to Closed
            grChild.work_notes = "Parent incident " + current.number + " has been closed.";
            grChild.update();
        }
    }
}
        

Preventing Incident Closure with Open Tasks (Reference 27):

A “before” business rule prevents an incident from being closed if it has open incident tasks.

// Business Rule: Prevent closing incident with open tasks
// Table: Incident
// When: Before
// Update: true
// Condition: current.state.changesTo(7) // Assuming state 7 is Closed

var grTask = new GlideRecord('incident_task');
grTask.addQuery('incident', current.sys_id);
grTask.addQuery('state', '!=', 3); // Assuming 3 is the state value for 'Closed'
grTask.query();

if (grTask.hasNext()) {
    gs.addErrorMessage('Cannot close the incident because there are open incident tasks. Please resolve all associated tasks first.');
    current.setAbortAction(true);
}
        

The logic for preventing closure with open tasks also applies similarly to Problems and Change Requests, ensuring that all related child work is completed before the parent record is finalized.

3.4. The Problem-Incident Relationship (Reference 28)

When a problem is resolved, its associated incidents should also be closed. This is typically handled by an “after” business rule on the `problem` table.

// Business Rule: Auto-close associated incidents when problem is closed
// Table: Problem
// When: After
// Update: true
// Condition: current.state.changesTo(7) // Assuming state 7 is Closed

if (current.state == 7) { // If the problem is closed
    var grIncident = new GlideRecord('incident');
    grIncident.addQuery('problem_id', current.sys_id); // Link incident to this problem
    grIncident.addQuery('state', '!=', 7); // Find incidents that are NOT yet closed
    grIncident.query();

    while (grIncident.next()) {
        grIncident.state = 7; // Set the state to Closed
        grIncident.work_notes = "Associated Problem " + current.number + " has been resolved and closed. This incident is now closed.";
        grIncident.update();
    }
}
        

Interview Relevance: What to Expect and How to Prepare

Understanding and managing duplicate incidents is a common responsibility for IT Service Management professionals, especially those working with platforms like ServiceNow. In an interview, you might be asked about:

• Your understanding of incident, problem, and change management: Be ready to explain the distinct roles of each (References 19, 20, 29) and how they relate.
• Strategies for preventing duplicate incident creation: Discuss user education, knowledge base utilization, and system configurations.
• Technical solutions for duplicate detection: Explain the use of “before” business rules, scripting with GlideRecord, and how you’d implement duplicate checks (referencing techniques used in Section 3.1).
• Handling existing duplicates: Describe scheduled jobs, scripting for cleanup, and the importance of keeping the oldest record (referencing Section 3.2).
• Business Rule logic: Be prepared to walk through the logic of business rules for state transitions (References 26, 28) and validation (Reference 27).
• Scripting examples: You might be asked to write a simple GlideRecord query or explain how to create a record using `new GlideRecord(‘incident’)` (Reference 23).
• Troubleshooting scenarios: How would you debug a business rule that’s not firing or a script that’s not cleaning up duplicates correctly?

Demonstrate a practical understanding of how to translate these concepts into working solutions within an ITSM framework.

Conclusion: A Cleaner System, Happier Users

Taming duplicate incident creation is not just about system hygiene; it directly impacts the efficiency of your support team and the satisfaction of your users. By implementing a combination of proactive preventive measures, intelligent system configurations, and robust automated cleanup processes, you can significantly reduce the occurrence of duplicates.

Leveraging the power of your ITSM platform’s scripting capabilities, as demonstrated with ServiceNow examples, allows for sophisticated detection and resolution. Remember to always test your scripts thoroughly, monitor their performance, and iterate on your logic based on real-world observations. A well-managed incident process, free from the clutter of duplicates, leads to faster resolutions, more accurate reporting, and a more streamlined IT service.