Writing Custom Parsing Rules in McAfee ESM

McAfee McA fee Enterprise Security Manager Ma nager Writing Custom Parsing Rules in McAf McAfee ee ESM September 19, 2016

The information contained in this document is confidential and proprietary. Please do not redistribute without permission. permission.

Writing Custom Parsing Rules in McAfee ESM

Page 1 of 21

Table of Contents 1

Introduction ..................................................................... ............................................................................................................................................... ............................................................................................................3 ..................................3

2

Prerequisites .................................................................... .............................................................................................................................................. ............................................................................................................3 ..................................3

3

Creating a Custom C ustom ASP Rule Rul e ......................................................................................................................................... ................................................................................................................................................4 .......4

4

3.1

Create a New Data Da ta Source ......................................................................................................... ...........................................................................................................................................4 ..................................4

3.2

Create a Custom Parsing Rule for ASP ..................................................................................................................5 ..................................................................................................................5

3.3

Enable the New ASP Rule in Policy ...................................................................................................................... ...................................................................................................................... 15

Creating a New Custom Type ......................................................................... ............................................................................................................................................ ...................................................................16 4.1

5

Testing the New ASP A SP Rule R ule ............................................................................................................................... .................................................................................................................................................. ................... 18 5.1

6

Create a Custom Type ............................................................................. ................................................................................................................................................ ...................................................................16

Importing a Log Sample ................................................................................................. ............................................................................................................................................ ........................................... 18

Best Practices ........................................................................................................................................................................... ........................................................................................................................................................................... 20 6.1

Know Your Data ............................................................................................................................................................ ............................................................................................................................................................20

6.2

Regular Expressions .................................................................................................................................................... ....................................................................................................................................................20

6.3

Content .................................................................... .............................................................................................................................................. ......................................................................................................... ...............................21

The information contained in this document is confidential and proprietary. Please do not redistribute without permission. permission.


Page 2 of 21

1 Introduction This guide details how to create and deploy a custom advanced syslog parser (ASP) rule in the McAfee ESM. A custom ASP rule could be necessary if there is an unknown/unparsed log in the McAfee ESM. This guide also includes some “Best Practices” to follow when writing ASP rules.

2 Prerequisites The following will be required, or useful, in creating custom parsing rules: 







A sample of the log to parse. Account privileges to add data sources. In some cases, administrator privileges, such as the ability to define custom types may also be needed. Vendor documentation can be helpful to determine the meaning of each field in the logs desired to be parsed. A working knowledge of PCRE (Perl Compatible Regular Expressions).

The information contained in this document is confidential and proprietary. Please do not redistribute without permission.


Page 3 of 21

3 Creating a Custom ASP Rule Follow the steps below to build a new SYSLOG data source and create an ASP parsing rule.

3.1 Create a New Data Source This step is optional if a data source already exists and is collecting the logs needing to be parsed. 1.

Select the Receiver, and then click on the “+” icon to add a data source.

2.

The Add Data Source window will open, set the Data Source Vendor as Generic and the Data Source Model as Advanced Syslog Parser. Give the new data source a name, add the proper IP address, and ensure that the Support Generic Syslogs field is left as Do nothing.

3.

Select OK, and then apply the new data source settings by confirming in the next window. This will create a new data source with an empty rule policy. The information contained in this document is confidential and proprietary. Please do not redistribute without permission.


Page 4 of 21

3.2 Create a Custom Parsing Rule for ASP 1.

From the McAfee ESM console, select the data source and open the Policy Editor.

2.

The Policy Editor window should appear. In the menu on the left, expand Receiver by selecting the “+” symbol, and select Advanced Syslog Parser. Initially this list will be blank while looking at the policy for the new data source.

3.

Select New > Advanced Syslog Parser Rule.



Page 5 of 21

4.

The Advanced Syslog Parser Rule window will appear. Within the General tab, assign information as displayed below:

Name – Assign a name, making it as descriptive and unique as possible. This is the event message that will show in the ESM views when the rule matches a log, unless the message is mapped directly from the log text within the rule. Tags – Assign tags to the rule. It is best practice to assign one or more tags to which this rule belongs. This helps in finding and grouping sets of rules created for a given device or application within the policy editor. Any tags added to a rule, will cause ESM to automatically include the rule in any policy that has enabled the given tagged rule set. Default Normalized ID – The normalization or category is a key step as many views, correlation rules, and reports use this field as a filter. It is very important to select the most relevant value in the taxonomy for the parsing rule in order to get the greatest value from within the McAfee ESM. Default Severity – If the log message does not contain a value for the severity, then the event will be assigned the value that is set here (the default is 25, with a scale of 1-100, 1 being the lowest and 100 being the highest). Rule Assignment Type – Rules can be grouped together; this pull-down menu provides a list of supported products to group the parsing rules by, separating the events from other data sources. This allows the event to be reported for a specific product. Description – Give the new rule a clear human-readable description. It should be a clear and complete description of the rule with complementary information to what was given in the rule’s name. The information contained in this document is confidential and proprietary. Please do not redistribute without permission.


Page 6 of 21

5.

Click on the next tab, Parsing, assign information as displayed below:

Process Name – This is an optional filter that acts similar to the content string, but only applies to the process name found within the SYSLOG header. Due to various syslog header formats used, and no guarantee that a syslog header will be present in the log, it is recommended to use content strings instead of the process name match. Content String – If a fixed string is always going to be found in the log then add it as a content string. The content string(s) of an ASP rule should uniquely identify each log if possible. Content strings are very important as they will not only make matches more accurate, but will speed up the execution of the rules in the running policy. It is highly recommended to have one or more content strings per ASP rule. This is a pre-filter for optimization – only logs that match the given content strings will be considered for matching and parsing by the regular expressions. The log must contain all given content strings, otherwise the logs will not be parsed using the regular expressions. Regular Expression – The first regular expression determines if the ASP rule will match the log. Any additional expressions will be used to capture values from the log that may or may not be found in all logs. Named Captures – Named captures can be used in regular expressions to more easily identify capture groups. The label used for the named capture can consist of alphanumeric and underscore characters but cannot begin with a number or include a space. The regular expression syntax for a named capture is: (?Pregular expression capture). Here is an example of a named capture where hostname is the name assigned to the capture group: Host\x3d(?P\S+). When using named captures the policy editor will display the capture name instead of the capture number, in the right-hand side of the Parsing tab as shown below. The information contained in this document is confidential and proprietary. Please do not redistribute without permission.


Page 7 of 21

Targeted Regular Expressions – Targeted regular expressions allow for applying an additional regular expression to the resulting value from another capture. For example, a regular expression could be written to pull a user’s full name from a log. Referencing that capture as the target, two targeted regular expressions could then be written to pull out the user’s first and last name . This data can then be mapped to different database fields. Follow the steps below to create a targeted regular expression: 1. 2.

On the Parsing tab, add a regular expression. Add another regular expression and select the capture (name or number) from the Target key drop-down list that references the first expression. The second regular expression should contain an expression to capture additional information from the captured value of the first regular expression.

In the image below, the named capture group full_name has a value of “Sam Johnson”. This named capture is being used as the Target Key for the first_name and last_name targeted regular expressions. The regular expression for first_name and last_name will be evaluated against the value of the full_name named capture. The result shows the key of first_name:1 with a value of “Sam” and last_name:1 with a value of “Johnson”. This specific example could be done within a single regular expression, and is a simplified example to illustrate the purpose of the Target Key field used with regular expressions. In general, it is recommended to run as few regular expressions as possible, but that may vary with specific use cases.



Page 8 of 21

Sample Log Data – In this field a sample log can be added and will be saved with the rule. The parts of the log matching defined regular expressions will be highlighted blue. Format – ASP can pre-process certain logging formats to simplify the mapping of data. The following formats are available: Generic – This is the default and should be used if the log does not match the other available formats. CEF (Common Event Format) – This eliminates the need to create a regular expression for each capture, and will allow the data to be mapped using the CEF key names found in the log. JSON – Similar to CEF, this eliminates the need to create a regular expression for each capture, and will allow data to be mapped using the JSON key names found in the log. XML - Basic, Simple, or Positional – This will allow ASP to parse logs that are in XML format and assign parsed data. The XML format choice will depend upon the type of XML that is within the logs. XML – Basic expects XML without any repeated elements. XML – Simple expects XML with either a single node with attributes, or a single set of non-repeated elements without nesting. XML – Positional expects XML that can have multiple nodes with attributes and multiple repeated elements with nesting.



Page 9 of 21

Parsed Values – The Key/Value fields on the right displays what is being parsed from the log sample(s) by the regular expression(s). The Key will display two numbers, divided by a colon. The first number indicates the regular expression being used, and the second number indicates the capture group used within that regular expression. If a captured value is the 4 th capture in the 3 rd regular expression defined, the key would display “3:4”. 6.

Add applicable Content Strings by selecting Edit. These are very important and can drastically affect performance of the created rule. Any content strings added will need to be a string that is guaranteed to be within the log, as these act as a type of “pre filter”. Multiple unique content strings can be added to increase the speed and efficiency of the rule.

7.

Select applicable regular expression options by adding checks within the boxes add one or more Regular Expressions by selecting the “ +” symbol.

Only use regular expressions for parsing purposes – Checking this box will remove the requirement for the first regular expression to match the log. The parser will rely upon the Content String for matching purposes, and will only use regular expressions to parse data when it finds a matching pattern. Case Insensitive – If the log may contain either upper or lower case letters in some fields, it may be simpler to write the expression in the same case and then use this option. This enables the case insensitivity option for all regular expressions defined in the parsing rule. Trigger when data doesn’t match – This option will make the rule trigger when the regular expression does not match the log.

a.

8.

The Edit Regular Expression window will appear. Add the expression and select OK.

With the expression added, the parsed values should be highlighted in blue, with the Group and Value shown to the right. Verify that each value is parsed out correctly and that all of the log line samples match. Make sure to know the meaning (from the vendor device documentation or own knowledge) of all the fields within the logs and capture the values that are useful. The information contained in this document is confidential and proprietary. Please do not redistribute without permission.


Page 10 of 21

9.

After all the regular expressions are written to extract out the needed values from the log, select the Field Assignment tab at the top of the window.

10. Available fields will be displayed, along with the Group and Value section as shown in the previous Parsing tab.

11. Drag and drop the Values from the right hand side to the Expression column next to the Field on the left. a.

For more complicated log file formats, multiple regular expressions may be used to capture the same data. In such cases, a field such as First Time might come from one of several potential several expressions. McAfee ESM allows the ability to specify the regular expression values in order of preference, so that if one does not match, it will use the second, or third, or so on. An example of this is shown below:

12. If the field is not displayed that is needed, click “+” above the Sample Value column, to display all custom type fields.

13. The Custom Types window will now be displayed. Select the desired field and then select OK.



Page 11 of 21

14. If a custom type does not exist for the type of data being mapped, the McAfee ESM allows for user-defined custom types to be added. Once a new custom type has been created, it will be listed within the Custom Types window and be available for use to map the values to. Please see Section 4 for details on creating a new custom type. 15. Click on the final tab, Mapping. This section will allow parsing the date format, mapping the action, and mapping the severity found within the log message.



Page 12 of 21

Time Format – The date/timestamp of a log message can be parsed using the variables defined in these fields. Many standard date/timestamps are recognized automatically by the McAfee ESM, however there may be a format that is either not recognized or that would be preferred to display differently. This section will allow for formatting the time to show up in the proper format when parsed. Action Mapping – Use this option if there is an action found within the log to be mapped to an available McAfee ESM action. Severity Mapping – The severity mapping allows for a value within the log to be mapped to a severity from 1-100. For example, a vendor might define their severity as either “Low”, “Medium”, or “High” in their logs. With the Severity Map section the severity value can map “Low” as 25, “Medium” as 50, and “High” as 75. 16. To add a Time Format, select the “+” symbol in the upper-right of the time format section.

a.

The Add Time Format window will appear.

b.

Enter a custom time format. As the values are entered, a small window will appear to assist in creating the custom time. The values can either be typed or selected from the given list.

c.

Select the fields that this custom format will apply to. It is best practice to apply it to both First Time and Last Time.

d.

Select OK to save.

17. To add an Action Map, find an applicable Action Value by scrolling through the list and then click inside the Action Key column. Enter the value that will be parsed from the device logs. If this value is parsed and assigned to “ Action”, then it will map to the assigned Action Value.



Page 13 of 21

Below the Action Mapping table, the option will be given to use a default action when one is not specified. This is generally a good option to set in order to insure that all logs receive an action type. Depending on the device, the default action should be different. To add the default action map, place a check next to Use the following action for the default if one is not specified , and then select a default action from the drop down menu.

18. To add a Severity Mapping, select the “+” symbol in the upper-right of the Severity Mapping section.

a.

The Add Severity windows will appear.

b.

Within the Severity Mapping field, enter the value that will be parsed from the device logs. If this value is parsed and assigned to “ Severity”, then it will map to the assigned Severity Value.

c.

Move the sliding scale to the desired Severity Value.

d.

Select OK to save.

19. Below the Severity Mapping section, the option to add a default severity will be given. This is generally a good option to set in order to insure that all logs receive a severity value. Depending on the device, the default severity should be different. To add the default severity, place a check next to Use the following severity for the default if one is not specified, and then enter a default value.

20. With the previous steps complete, the new ASP rule should be finished. Select Finish, located in the bottom-right hand corner to finish and save. The information contained in this document is confidential and proprietary. Please do not redistribute without permission.


Page 14 of 21

3.3 Enable the New ASP Rule in Policy Once the new rule has been created, it will need to be enabled and added to the policy in order for the receiver to begin parsing the associated logs. 1.

Open the Policy Editor for the data source that the new ASP rule was created for, and beneath Receiver, select Advanced Syslog Parser. The newly created ASP rule(s) will be listed with the Action column displaying “disabled”. Click on disabled and then change the value to enabled.

2.

With the new rule set as enabled, click on the Rollout icon displayed in the upper-right hand corner of the Policy Editor window.

3.

If the rule didn’t already save previously, it will prompt to save. Select Yes to save.

4.

Once saved, the Rollout window will appear and will display a list of devices to roll policy out to. Ensure that the applicable data source is set to “Roll this policy out now.” and select OK to do so.



Page 15 of 21

4 Creating a New Custom Type 4.1 Create a Custom Type To create a custom type, login with a user that has system administrator privileges and then select the System Properties icon.

1.

Select the Custom Types option from the menu on the left.

2.

Select Add from the options shown on the right.

Note: Select the Edit option to edit a user defined pre-existing custom type if needed. 3.

The Add Custom Type window will appear. Setup the new custom type as desired and select OK.



Page 16 of 21

Name – This will be the name for the new Custom Type that will be displayed within McAfee ESM. Data Type – This will be the type of data that the McAfee ESM will expect to be populated into this field. Selection of the correct data type is important. Commonly used data types include: Random String, IP Address, Decimal, Integer, and GUID. Events Field – These fields are for Events. There are a specific number of fields that can be used to have data parsed into. If two different custom types occupy the same field, only one can be used. When creating custom types, make sure to place it within a field that is not already being used by another important field that will be used. Flows Field – These fields are for Flows. There are a specific number of fields that can be used to have data parsed into. If two different custom types occupy the same field, only one can be used. When creating custom types, make sure to place it within a field that is not already being used by another important field that will be used. Index Data – This option will determine on whether or not the data that is parsed is indexed within the database. Indexed data is searchable within the McAfee ESM. Description – This will be a simple description to explain what type of data the newly created custom type contains. 4.

With the desired custom type setting entered, select OK.



Page 17 of 21

5 Testing the New ASP Rule 5.1 Importing a Log Sample Once the new ASP rule has been created and added to the current policy, it is a good idea to test it to ensure that it is parsing as expected. 1.

Obtain sample logs from the device that the new rule was created for and save these to a text file.

2.

Select the data source to be tested and click Properties.

3.

The Data Source Properties window will appear, select the Upload button.

4.

Navigate to the log sample file and select it to upload. Select the Upload button and then a confirmation window will be displayed showing the file was successfully uploaded. Select Close to close out the window.



Page 18 of 21

5.

To initiate the log and flow gathering process manually, click on the Get Events and Flows button.

6.

The Get Events and Flows window will appear. Select Events and then click the Start button to collect the events.

7.

Once the events are collected, find the events in the dashboard and verify the newly created ASP rule is parsing as expected. If any adjustments need to be made, make the changes and then send the sample logs through again to verify that the rule is parsing as expected.



Page 19 of 21

6 Best Practices The following are given as best practices that can help with creating ASP rules that work well within the McAfee ESM. Following these recommendations will help ensure reporting is accurate, not overly complex, and will help parsers and Receivers maintain intended performance and efficiency.

6.1 Know Your Data It is important to understand the logs that are being sent to the SIEM, and whether or not they have value in your organizations reporting needs. It is recommended to only parse logs that meet your organization’s reporting goals. 







Ensure that captured values mapped to ESM database fields align with the intended use of the specific custom type fields. Avoid indexing fields that contain unique and random or high cardinality data such as URLs. Ensure rules mapping event messages directly from the log are not mapping unique and random or high cardinality strings as messages. The ESM creates a data source rule for each unique event message, and ESM performance can be negatively impacted if there are a high rate of unique strings being mapped to message. Categorize events by adding a normalized category to the rule. Data source rules, generated by parsing rules, inherit the normalization assigned to the main parsing rule. This means that if the main parsing rule is left normalized to “Uncategorized” , then the parsed events will also be normalized as “Uncategorized”, making a search for “Uncategorized” events to find unparsed events in accurate.

6.2 Regular Expressions By default, the first regular expression determines if the ASP rule will match the log and will need to match a pattern that is always present in the log. Additional regular expressions can be written to capture values from the log and mapped to custom types within the McAfee ESM. Subsequent regular expressions do not determine the rule match, and are used for parsing only. While it is possible to test regular expression results on a few log lines within the McAfee ESM console itself, we recommend using a graphical tool. There are many free web based tools that can be used in addition to standalone installable tools. Optionally, another useful tool would be a text editor that supports regular expression searches. Any tools used to test regular expressions need to support pcre expressions. Make sure that regular expressions are efficiently written because poorly written expressions can take more system resources to run, and can affect the parsing performance of a specific data source or even the entire Receiver.



Page 20 of 21

6.3 Content Ensure there is at least one value in the content field section. Content strings should be at least 3 characters in length and should be as unique as possible for the specific event. It is advised to include enough content matches to uniquely identify the log. Using one or more content fields within the ASP rule will significantly speed up the matching and parsing process on the Receiver.

The table below shows an example of how to use the content field to uniquely identify a log sample. Syslog Sample

Content Fields

<180>Jan 1 00:00:00 testhost ftpd[4325]: FTP LOGIN FROM test.org [192.168.1.1], anonymous

“ftpd” “FTP LOGIN FROM”

<180>Jan 1 00:00:00 testhost ftpd[4325]: FTP LOGIN REFUSED (ftp not in /etc/passwd) FROM 192.168.1.1 [192.168.1.1], anonymous

“ftpd” “FTP LOGIN REFUSED”



Page 21 of 21

Writing Custom Parsing Rules in McAfee ESM

Recommend Documents