Unpacking Audit Trails: A Deep Dive into BMC Remedy AR System Data Integrity

In the complex world of enterprise IT Service Management (ITSM), maintaining the integrity and traceability of data is paramount. This is where the concept of “audit forms” or, more broadly, audit trails and data tracking mechanisms, comes into play. While the term “audit forms” might sound straightforward, in practice, it often refers to the underlying systems and configurations that capture changes, track user activity, and provide a historical record of data modifications. Today, we’ll explore how BMC Remedy Action Request System (AR System) facilitates these crucial audit functions, delving into its architecture and features that contribute to a robust audit trail.

The Backbone of Data Integrity: BMC Remedy AR System Architecture

To understand how BMC Remedy AR System (ARS) handles data tracking and auditing, we must first grasp its fundamental architecture. ARS is designed as a multi-tiered system, where each tier plays a specific role in managing and delivering data and services.

Server Tier: The Brains of the Operation

At the heart of ARS lies the Server tier. This is where the magic happens. The AR System server is the central control unit. It orchestrates workflow processes, managing how data flows, how it’s accessed, and how changes are processed. Crucially, this tier also hosts essential server-side applications like the Approval Server, Email Engine, and various plug-in servers (C and Oracle Java). These components interact with the data tier to execute complex operations, many of which involve logging and tracking.

Data Tier: The Vault of Information

Supporting the server tier is the Data tier. This layer houses your databases (e.g., Oracle, SQL Server, DB2) and other data sources. The database server acts as the primary engine for storing, retrieving, and managing all the information within ARS. The AR System server communicates with this tier to read and write data, and it’s within these databases that the actual audit records are typically stored.

Understanding ARS Database Structure and Table Creation

When you install AR System, a specific sequence of database tables is created to house the system’s core configurations and metadata. Understanding these tables can provide insight into how data is organized and managed:

1. control: The foundational table, often the very first one created, containing essential system-level control information.
2. controlRecordIds: Manages unique record identifiers, crucial for maintaining data integrity and relationships.
3. arschema: This is a critical table that stores the schema definitions for all forms. It essentially describes the structure of each form, including its fields, properties, and relationships.
4. schema_index: Handles indexing for schemas, optimizing data retrieval performance.
5. schema_group_ids: Manages group associations with schemas, impacting access control and permissions.

These core tables lay the groundwork for how ARS manages its metadata, which is fundamental to tracking changes and understanding system behavior. For instance, changes to form structures or field properties would be logged and managed within these foundational schema-related tables.

Configuration and Data Storage: The ar.cfg File

The ar.cfg file (or its equivalent on different operating systems) is the central configuration hub for the AR System server. When you make configuration changes, whether it’s tuning performance, setting up email integrations, or defining security parameters, these settings are stored here. Notably, sensitive information like passwords can be saved in this configuration file, which, while convenient, also highlights the importance of securing these configuration files, as they directly impact system access and data security.

Database Table Naming Conventions

To track specific ARS components within the database, BMC follows a consistent naming convention. This makes it easier to locate and understand the data associated with different workflows:

• Active Links: Tables are prefixed with dbo.actlink and further differentiated by appending the action name (e.g., dbo.actlink_show).
• Filters: Similar to active links, filter tables are prefixed with dbo.filter (e.g., dbo.filter_setfield).
• Escalations: These are identified by tables prefixed with dbo.escalation.

This structured approach to table naming is a form of implicit auditing, allowing administrators to easily identify and analyze the data related to specific workflow types.

Version Information and Overlay Groups: Managing Customizations

In any dynamic system like ARS, customizations are inevitable. Understanding how these customizations are managed and how they relate to the “out-of-the-box” functionality is key to maintaining system stability and enabling effective audits. The Share: application properties form is your go-to place for obtaining comprehensive version information about your ARS environment.

Overlay Groups: Controlling Customization Modes

The concept of Overlay Groups is fundamental to how ARS manages customizations, especially in newer versions. It allows administrators to define different levels of access and modification capabilities for users:

• Overlay Group field set to 1: Users are restricted to working with overlay and custom mode objects. In BMC Remedy Developer Studio, this means they operate in Best Practice Customization mode. This is crucial for developers making specific enhancements.
• Overlay Group field set to 0: Users are restricted to working with base mode objects. In Developer Studio, this places them in Base Developer mode. This is important for users who need to see and work with the original, untouched forms and components.
• Overlay Group set to (clear): This provides the most flexibility, with no restrictions on access to base, overlay, or custom objects. This is typically for administrators or highly trusted personnel.

These settings ensure that customizations are managed in a controlled manner, preventing accidental overwrites of core functionality and providing a clear audit trail of who is making what changes in which mode.

Understanding ARS Groups: Permissions and Access Control

Group management in ARS is a critical aspect of access control and, by extension, auditing. Groups define who can access what within the system. ARS categorizes groups in several ways:

Group ID Ranges

Each group is assigned an Integer ID, with specific ranges allocated for different purposes:

• 0-1000: Reserved for AR System internal groups and current AR System applications.
• 1000-13004 and 13007-14999: For regular and computed groups.
• 13005-13006: Reserved for CMDB groups.
• 14999-59999: For future AR System application use.
• 60000-60999: Allocated for dynamic groups.

These ranges help in organizing and identifying the purpose of different groups, which can be referenced in audit logs.

Types of Groups

ARS offers two primary types of groups, each with implications for how user access is managed and audited:

• Explicit Groups: These are groups to which users are manually assigned. When a user is added to an explicit group, they inherit all permissions granted to that group. Examples include admin, sub-admin, customize, and struct admin. It’s important to note that explicit groups are often server-specific. If you move objects to a new server, you might need to re-map or resolve permission conflicts. Deployable applications, using role-based permissions, offer a more flexible approach here.
• Implicit Groups: Users belong to these groups based on specific conditions or circumstances, rather than direct assignment. This often relates to the content of special fields within a request. You don’t directly add users to implicit groups. Dynamic groups you create also fall under this category. Examples include Arsystem, public, assignee, and submitter.

Understanding these group types is crucial for auditing access. For instance, if a specific record is accessed by a user, an audit log might indicate they accessed it as a member of the assignee group (implicit) rather than a directly assigned group.

Core Fields and Data Storage of Date/Time Values

ARS utilizes a set of 9 core fields that are fundamental to every record. These fields provide essential metadata:

1. Request ID (1)
2. Submitter (2)
3. Create Date (3)
4. Assigned To (4)
5. Last Modified By (5)
6. Modified Date (6)
7. Status (7)
8. SD (8)
9. Status History (Hidden, 15)

The Create Date and Modified Date fields are prime examples of auditable data points. They provide a timestamp for when a record was created or last altered.

How AR System Stores Date/Time Values

ARS stores date/time values in a very specific and efficient manner: as the integer number of seconds since 00:00:00 GMT, January 1, 1970. This is commonly known as Unix time or epoch time. This format allows for precise temporal tracking and calculations. The system currently supports dates from January 1, 1970, through January 18, 2038, effectively covering a significant period.

Troubleshooting Tip: If you encounter issues with date/time calculations or display, always verify that the server’s system clock is accurate and synchronized. Incorrect server time can lead to misinterpretation of timestamps in audit logs.

Character Fields vs. Diary Fields: A Tale of Two Data Types

When it comes to storing textual information, ARS offers different field types, each with distinct behavior relevant to auditing:

• Character Fields: These are designed to hold alphanumeric characters. A key characteristic is that they can be overwritten. If you modify data in a character field, the old data is replaced.
• Diary Fields: These fields are designed for appending information over time. When new data is entered into a diary field, it is appended along with the timestamp and the user who made the modification. This preserves a historical log of all entries within that single field, making it invaluable for tracking the evolution of specific notes or comments.

Interview Relevance: In an interview, understanding this distinction is vital. A common question might be about how to track the complete history of comments on a ticket. The answer, leveraging diary fields, showcases practical ARS knowledge.

Note: While character fields can be overwritten, Web reports may not fully support diary fields, attachment fields, or attachment pools. This limitation should be considered when designing reporting strategies.

Menus and Their Association with Fields

Menus in ARS provide users with predefined options for character fields, enhancing data entry consistency. They can be static or dynamic:

• Static Menus: These are predefined lists that do not change unless manually edited. Examples include Character, Form data dictionary, and Field data dictionary.
• Dynamic Menus: These menus retrieve their data from external sources, such as Search results, SQL queries, or files.

File menus can be of both static and dynamic types.

Menu Refresh Rates

For dynamic menus, a refresh rate determines how often the menu’s contents are updated:

• On Connect: Retrieves the menu when the user opens the menu after selecting the form. Requires re-opening the form to update.
• On Open: Retrieves the menu every time the user opens it. This ensures the most up-to-date data but can impact performance if done frequently.
• On 15 Minute Interval: A balanced approach. Retrieves the menu on first open and then every 15 minutes.

Character menus are static and therefore do not have a refresh rate.

Important Note: When a menu-associated field or form is deleted, the menu itself does not automatically get deleted. This ensures that menu definitions remain intact even if their associated field is removed.

Form Types and Database Storage

ARS organizes forms into different types, each with a specific purpose and corresponding database designation:

• Regular Forms (Form ID: 1): Standard forms that hold transactional data.
• Display Forms (Form ID: 4): Forms used for presenting data without direct modification.
• Join Forms (Form ID: 2): Combine data from multiple regular forms based on join criteria. BMC recommends joining up to 6 forms for optimal performance.
• View Forms (Form ID: 3): Essentially a type of form that allows creating views of other forms for different user perspectives.
• Vendor Forms (Form ID: 5): Allow ARS to access arbitrary external data sources, acting as an interface to data residing outside the primary ARS database.
• Archive Forms: Used for storing historical data that is no longer actively needed in the primary forms.
• Audit Forms: Specifically designed for auditing purposes.

The creation of these forms directly leads to the creation of corresponding tables in the database. For instance, after saving a form, you’ll typically find three associated tables:

• T (Transaction Table): Stores the actual data of the form.
• B (Binary Table): Stores binary data like attachments, and potentially active link scripts.
• H (History Table): Stores status history and other historical tracking information.

Attachments and Their Storage

ARS supports file attachments, which are crucial for many ITSM processes. Attachment data is stored in a structured manner:

• Attachment Pool Data Type: This refers to the field type that holds the attachment.
• Attachment Field: The specific field configured to accept attachments.

When an attachment is stored, ARS creates two primary database entries:

1. BSchema_id: This table manages the association between schema IDs and binary data.
2. Battachpoolid: This table links request IDs with the binary data of the attachments.

Every attachment typically creates three columns in the relevant binary table of the transaction:

• C: Depicts the path of the file.
• CO: Depicts the original size of the file.
• CC: Depicts the compressed size of the file.

These are stored in bytes, providing granular information about attached files.

Workflow Automation: Active Links, Filters, and Escalations

Workflows are the engine that drives automation in ARS, and their behavior is inherently auditable. Workflows are sets of processes that automate company operations triggered by specific conditions.

Workflow Objects and Their Functions:

• Active Links: These execute based on client operations or information currently displayed on the screen. They are triggered by user interactions within the AR System client (e.g., opening a form, modifying a field). Actions can include displaying messages, changing field colors, or pushing data to other forms.
• Filters: Filters monitor form transactions as they are processed by the server. They are triggered by events like submitting, modifying, or deleting a record. Filters perform actions on the data itself, such as data validation, workflow execution, or updating related records.
• Escalations: These workflows operate based on predetermined time intervals. They periodically check requests in the database for specific conditions and then execute actions. Think of automated ticket reassignments after a certain period of inactivity or overdue notifications.

The execution of these workflows is logged, providing an audit trail of automated actions and system responses. For example, an audit log might show that a filter automatically updated the status of a ticket after it was modified.

Interview Tip: Be prepared to explain the difference between Active Links, Filters, and Escalations, providing real-world examples for each. This demonstrates a solid understanding of ARS workflow capabilities.

Audit Trails in Action: Tracking Status History and Data Modifications

The concept of “audit forms” is most tangible when we look at how specific data changes are recorded. ARS provides mechanisms to track these changes:

Status History

The Status History field (15), often hidden by default, is a direct audit mechanism. By modifying the Status field on a form (e.g., from “New” to “Assigned”), an entry is created in this history field. This entry includes the status value, the timestamp, and the user who made the change. You can view this history through the AR System User tool by navigating to the View menu and selecting “Status History,” or by using the shortcut Ctrl+H while the form is in modify mode.

Diary Field Values in the Database

As discussed, diary fields append data. In the database, the format typically looks like: [timestamp in seconds] -| [User] -| [Data]. If a modification occurs, new data is appended in the same format to the existing entry, creating a clear, chronological audit trail within that field.

Status Updates Reflected in Database Rows

When you update a ticket’s status, the changes are reflected within the same row in the database tables, associated with the core fields like Submitter, Create Date, Assigned To, and Modified Date. This ensures that all related information for a specific ticket remains consolidated and auditable.

Granular Overlays: Fine-Tuning Customization Audits

More advanced customization management in ARS involves Granular Overlays. This allows you to choose specific subcomponents of an object and apply different overlay types to them. This precision in customization is inherently beneficial for auditing, as it isolates the changes made.

Types of Granular Overlays

• Additive Overlay (Extensions Overlay): Used to add custom information to an origin object. If the origin object changes, these additions are appended. This is like adding notes to an existing document.
• Overwrite Overlay: The entire overlaid object is used, and the origin object is ignored. This is useful for scenarios where you need to replace default behavior entirely, such as removing permissions.
• No Overlay (Inheritance Overlay): The default for untouched objects. Properties are inherited from the origin object without changes. This ensures that if the origin object is updated, the overlaid object benefits from those updates unless explicitly overridden.

The granular nature of these overlays makes it easier to pinpoint exactly what has been customized and how, simplifying the audit process and reconciliation after upgrades.

Permissions and Auditing

Access control and permissions are intrinsically linked to auditing. When it comes to menus, you cannot directly assign permissions to menus themselves. However, the permissions applied to the character field to which the menu is attached will be inherited by the menu. This means auditing who can access a menu is done by auditing the permissions on its associated character field.

To grant common permissions to multiple forms:

1. Identify the group to which you want to grant permissions.
2. Right-click on the group name.
3. Select “Assign Permissions.”
4. Add the desired forms to the group’s permissions.

This centralized permission management simplifies administration and provides a clear audit trail of who has access to which forms.

Troubleshooting and Best Practices for Auditing in ARS

Maintaining effective audit trails in ARS involves proactive measures and understanding potential issues.

Key Considerations for Auditability:

• Diary Fields for History: For fields where a chronological history of changes is critical (e.g., comments, resolution notes), always use Diary fields.
• Status History Field: Ensure the Status History field (15) is enabled and visible for relevant forms where status changes are tracked.
• Workflow Logging: Configure workflow logging (if applicable and supported by your version) for critical workflows to capture their execution details.
• Regular Database Backups: Implement a robust backup strategy for your ARS database. Backups are the ultimate safety net for data recovery and historical analysis.
• Secure Configuration Files: Protect sensitive configuration files like ar.cfg to prevent unauthorized system access and data manipulation.
• User and Group Auditing: Regularly review user accounts and group memberships to ensure appropriate access levels are maintained.

Troubleshooting Common Audit-Related Issues:

• Missing Audit Entries: If audit entries are not appearing, check the field types (e.g., character vs. diary), workflow configurations (filters might be suppressed or not firing), and system event logs for errors.
• Performance Degradation: Overly complex workflows or excessive logging can impact performance. Optimize workflows and audit logging levels.
• Data Corruption: In rare cases, data corruption can occur. This is where regular backups and database integrity checks are vital.
• Attachment Issues: Verify attachment pool configuration, disk space, and file path permissions if attachments are not storing or retrieving correctly.

Troubleshooting: If you suspect data inconsistency and need to investigate the database directly, remember that the ARS database structure can be complex. Always proceed with caution and ideally have an experienced ARS administrator or DBA assist you. Using SQL queries to directly inspect tables like arschema, actlink, and filter can provide raw data for analysis.

Conclusion: Building Trust Through Traceability

While “audit forms” might not be a single, distinct ARS object, the system provides a comprehensive suite of features that enable robust data tracking and auditing. From the core architecture and database table structures to workflow automation, group permissions, and specialized fields like diary fields, BMC Remedy AR System empowers organizations to maintain data integrity, ensure compliance, and build trust through complete traceability. By understanding and leveraging these capabilities, IT teams can create a more secure, reliable, and auditable enterprise environment.