Archive Scheduling: Automate Your Document Archiving Process

Archive Scheduling

In the realm of IT Service Management (ITSM) and enterprise systems, particularly within platforms like BMC Remedy or BMC Helix, efficient data management is paramount. As systems grow, so does the volume of data. Storing all historical data indefinitely can lead to performance degradation, increased storage costs, and difficulties in retrieving current, relevant information. This is where Archive Scheduling comes into play. It’s not just about deleting old data; it’s a strategic process for intelligently managing the lifecycle of your information, ensuring optimal system performance and data accessibility.

At its core, archive scheduling is about identifying data that is no longer actively used but still needs to be retained for compliance, auditing, or historical analysis, and then systematically moving it to a more cost-effective, long-term storage solution. This process is deeply intertwined with the platform’s workflow engine, which dictates how and when these archiving tasks are executed.

Understanding the Foundation: Workflow in BMC AR System

Before diving into archive scheduling, it’s crucial to grasp the fundamental concepts of workflow within BMC’s AR System (and by extension, BMC Helix). Workflow defines the set of processes that drive a company’s operations. In BMC AR System, workflow automates these processes through a combination of intelligent components that trigger actions in response to predefined conditions or events. These components are:

1. Active Links

Active links are the workhorses of client-side automation. They execute based on user interactions or information displayed on the current screen. Think of them as event listeners for user actions. When a user clicks a button, fills a field, or navigates through the interface, an active link can be triggered to perform a specific operation.

Trigger: Client operations or information on the current screen.
Where Performed: Client, current form window.
Key Characteristics:
- Cannot be triggered via API.
- Duplicate names are not allowed (a `_c` suffix is appended).
- Must have at least one ‘Add’ action.
- When saving a form with a new name, active links are not copied. Renaming a form, however, *will* preserve its associated active links.
Common Actions: Change Field, Close Window, Commit Changes, Open Window, Push Fields, Set Fields, Notify, Message.

2. Filters

Filters are server-side workflow objects that intercept and process data transactions as they move through the AR System server. They act as gatekeepers, checking requests against defined qualifications and performing actions if those qualifications are met. Filters are essential for enforcing business logic and data integrity before data is committed to the database.

Trigger: Server-side operations on form transactions (Submit, Modify, Delete, Get Entry, Merge, Service).
Where Performed: AR System server.
Key Characteristics:
- Interact with the database to retrieve or modify data.
- Execute with administrator permissions.
- Operate at the form level.
- Have an execution order (default is 500).
Execution Options: Submit, Modify, Delete, Get Entry, Merge, Service.
Common Actions: Push Fields, Set Fields, Run Process, Log to File, Goto, Direct SQL, Service, Commit Changes.

3. Escalations

Escalations are also server-side workflow objects, similar to filters, but with a distinct trigger mechanism: time. While filters react to specific user or system events, escalations are scheduled to run at predetermined time intervals or after a defined duration. When an escalation runs, it scans the database for all records that meet its defined qualifications and then executes associated actions.

Trigger: Predetermined time interval or after a defined time.
Where Performed: AR System server.
Key Characteristics:
- Interact with the database.
- Executed by the ar_escalator user.
- Unlike filters acting on the current request, escalations act on *all* requests matching qualifications when they run.
Common Actions: Push Fields, Set Fields, Run Process, Log to File, Goto, Direct SQL, Service, Commit Changes.

The Role of Archive Scheduling

Archive scheduling leverages these workflow components, primarily escalations and filters, to automate the process of moving data from active, high-performance databases to slower, more cost-effective archive storage. This isn’t a one-size-fits-all approach; the specific implementation depends heavily on the system’s architecture and business requirements.

Why Archive?

Performance Optimization: Reducing the volume of active data improves query response times and overall system responsiveness.
Cost Reduction: Archival storage is typically less expensive than primary production storage.
Compliance and Auditing: Many industries have regulatory requirements for data retention, necessitating long-term storage.
Simplified Data Management: Easier to manage and access historical records when they are segregated.
Disaster Recovery: Archived data can serve as a valuable component of a comprehensive disaster recovery strategy.

Implementing Archive Scheduling with Workflow

The typical pattern for archive scheduling involves using a combination of filters and escalations. Here’s a conceptual breakdown:

Phase 1: Identification and Staging (often using Filters)

First, you need to identify records that are candidates for archiving. This often involves a filter that runs periodically or is triggered by specific events.

Trigger: Could be a scheduled filter (though less common for initial identification) or, more practically, a filter triggered by a process that marks records as “ready for archive” (e.g., after a certain number of days since closure, or when a record reaches a final status).
Qualification: Criteria that define an “archive candidate.” This might include a status field (e.g., ‘Closed’, ‘Resolved’), a timestamp field (e.g., ‘Date Closed’ older than X months), or a combination of factors.
Action:
- Set Fields: Update a flag on the record (e.g., ‘Archive Status’ set to ‘Ready’).
- Push Fields: If the archiving process involves moving data to a separate archive form or database, this action could initiate the transfer.

Phase 2: Archiving Execution (primarily using Escalations)

Once records are identified and potentially staged, escalations are ideal for systematically moving them to the archive.

Trigger: A scheduled interval (e.g., daily, weekly).
Qualification: Targets records marked as “Ready for Archive” or directly checks the criteria for archiving. For example:
- 'Archive Status' = "Ready"
- 'Status' = "Closed" AND 'Last Modified Date' < (CURRENT_DATE() - 180) (for records closed more than 180 days ago)
Action:
- Push Fields: Transfer the selected fields from the active record to a dedicated archive form or a staging area. This is a common and robust method.
- Run Process: If your archiving strategy involves external scripts or tools that manage data movement to specialized archive databases (e.g., SQL Server, Oracle), the Run Process action can be used to invoke these external processes.
- Direct SQL: For more advanced scenarios or direct database manipulation, Direct SQL can be used to execute SQL statements for data movement or deletion. This requires careful handling and deep database knowledge.
- Set Fields: After successfully pushing fields or executing the archive process, this action can update the original record (e.g., set 'Archive Status' to 'Archived', or even delete the record if the archive process is a copy-then-delete mechanism).
- Delete: If the Push Fields action creates a copy in the archive, the original record can then be deleted. This needs to be carefully sequenced.

Key Workflow Actions for Archiving

Let's elaborate on the workflow actions most relevant to archive scheduling, drawing from the provided reference:

1. Push Fields

This is the most versatile and commonly used action for archiving. It allows you to copy data from selected fields in one request to another. This can be used to:

Copy data to a dedicated Archive Form.
Transfer data to a staging table before further processing.

Reference: "Push Fields action enables you to automate updates to the database. You can transfer values from selected fields in the current request to another request in a different form or in the same form."

Use Cases: Push selected fields from the active Incident form to an Incident_Archive form.

2. Run Process

When archiving involves more complex logic or integration with external systems, Run Process is invaluable.

Trigger external archiving utilities.
Execute scripts that manage data transformation or database moves.

Reference: "Run Process action to run an independent process on a client computer or an AR System server."

Use Cases: Execute a Python script that moves data to a separate archival database or initiates a database snapshot.

3. Direct SQL

For direct database manipulation, Direct SQL can be used, though it comes with higher risk and requires expert knowledge.

Perform bulk inserts into archive tables.
Execute custom deletion scripts.

Reference: "Direct SQL action to submit any legal SQL command to a non-AR System database." (Note: It can also interact with the AR System database directly).

Use Cases: Inserting large batches of data into a pre-defined archive table in a SQL Server instance. Be extremely cautious with this action as it bypasses much of the AR System's built-in safety mechanisms.

4. Set Fields

Crucial for updating the status of records after archiving.

Mark original records as 'Archived'.
Set a flag indicating successful archival.

Reference: "Set Fields... used in Active links, Filters, Escalations."

Use Cases: After pushing fields to Incident_Archive, use Set Fields on the original Incident form to update 'Archive Status' = "Archived".

5. Delete

Used judiciously after successful archival to remove data from the active system.

Clean up the production database after data has been safely archived.

Reference: The delete audit field value is 8. This indicates that delete operations can be tracked.

Use Cases: Delete the original Incident record after it has been successfully copied to Incident_Archive.

Audit Fields and Triggers

Understanding audit fields helps in tracking actions related to data manipulation, including archiving.

1 GET ENTRY: Records retrieved.
2-Set: Fields updated.
4-Create: New records created.
8-Delete: Records deleted.
16-Merge: Records merged.

When implementing archive scheduling, you'll often be dealing with actions that involve 'Set' (to update status) and potentially 'Delete' if the original records are removed. Tracking these actions via auditing is vital for validation and troubleshooting.

Cache Modes and Their Impact

The cache mode of your AR System server can significantly impact the execution of workflow, including archive scheduling. Understanding these modes is crucial for predicting performance and avoiding unexpected delays.

1. Production Cache Mode (Default)

In this mode, administrative operations (like cache refreshes or certain workflow executions that might temporarily lock resources) are performed on a copy of the server's cache. This allows regular application users to continue their operations without interruption. It's ideal for production environments with high user concurrency.

Impact on Archiving: Generally less impact on interactive users. Archive escalations running in the background are less likely to be blocked by user activity.

2. Development Cache Mode

In this mode, administrative operations can lock the shared cache, temporarily preventing other users from accessing it until the operation is complete. Potentially long-running tasks like escalations, integration jobs, or complex queries are incompatible with administrative changes in this mode and can lead to significant delays.

Impact on Archiving: Archive escalations, if they are long-running, could be delayed or even interfere with administrative tasks if not carefully managed. It's often best to schedule archiving during off-peak hours when using this mode.

Tip: For production systems, always strive to use Production Cache Mode for ongoing operations. Development or maintenance windows might be the only times to consider Development Cache Mode for specific tasks.

`arsignal` Utility for Cache and Definition Reloading

The arsignal utility is a powerful command-line tool used to signal the AR System server to reload or recache various types of definitions. This is crucial after making changes to workflow, configurations, or when dealing with data that might have become stale.

Key options relevant to archive scheduling:

-e: Recache and reload escalation definitions. If you've made changes to your archive escalations, running arsignal -e serverName ensures these changes are picked up by the server.
-r: Recache definitions from the database. A more general reload that can be useful if you suspect stale workflow definitions.
-c: Reload information from its configuration file (ar.conf or ar.cfg). Less directly related to workflow but important for server-level configurations.

Example: To force the AR System server named 'MyRemedyServer' to reload its escalation definitions, you would run: arsignal -e MyRemedyServer.

Troubleshooting Archive Scheduling

Archive scheduling can present a few common challenges:

1. Archiving Not Happening

Check Escalation Status: Ensure the escalation is enabled and not disabled.
Review Qualifications: Are the qualifications accurately selecting the intended records? Use the Log to File action within the escalation to log the number of matching records before and after any filtering to verify.
Execution Order: If the escalation relies on a preceding filter to mark records, ensure their execution orders are correct.
Permissions: Does the ar_escalator user have the necessary permissions to read and write to the target archive form or database?
Server Load: Is the server heavily loaded? Long-running escalations might be timing out or being deferred.
ar.cfg Settings: Check parameters like EscalationPollInterval to ensure the escalation is scheduled to run frequently enough.

2. Data Not Appearing in Archive

Push Fields Mapping: Verify that all necessary fields are correctly mapped between the source and destination.
Destination Form/Table Issues: Is the archive form or table accessible and correctly configured? Check for any errors during the Push Fields operation in the server's API logs or the archive process logs.
Data Integrity: Are there any data type mismatches or constraints in the archive destination that prevent successful pushes?

3. Original Records Not Deleted (if applicable)

Sequence of Actions: Ensure the Delete action occurs *after* a successful Push Fields or the completion of the archiving process invoked by Run Process.
Error Handling: If the Push Fields or Run Process action fails, the Delete action should ideally not execute. Implement error handling or rollback mechanisms.
Dependencies: Are there any other workflows or business rules that depend on the original record being present?

Interview Relevance

Understanding archive scheduling is a valuable asset for IT professionals, especially those working with ITSM platforms. In an interview, you might be asked:

"Describe a situation where you implemented data archiving. What workflow components did you use and why?"
"How would you ensure that archiving doesn't impact end-user performance?" (Discuss cache modes, off-peak scheduling).
"What are the key differences between using Push Fields and Run Process for archiving?"
"How do you troubleshoot an escalation that isn't firing?"
"What are the risks associated with using Direct SQL for data archival?"
"Explain the role of escalations in automated data lifecycle management."

Demonstrating knowledge of workflow objects, their interactions, and practical considerations like performance and troubleshooting will make you a strong candidate.

Conclusion

Archive scheduling is an essential discipline for maintaining the health, performance, and compliance of any robust IT system. By leveraging the powerful workflow capabilities of BMC AR System and BMC Helix – specifically filters, escalations, and a suite of carefully chosen actions like Push Fields, Run Process, and Set Fields – organizations can implement automated, efficient, and reliable data archival processes. This proactive approach ensures that valuable historical data is preserved while keeping the active system lean, responsive, and cost-effective.

Remember to always test your archiving workflows thoroughly in a non-production environment before implementing them in production. Monitor logs, audit trails, and system performance closely after implementation to ensure everything is running as expected.

Further Reading: