1. Overview

This article explains how to create, validate, upload, associate, and troubleshoot URL Files used with URL Filtering. URL Files allow administrators to define URL or fully qualified domain name (FQDN) entries, or regular-expression patterns, and associate each entry with a URL reputation. The resulting file can be used when configuring custom URL Category objects.

A common source of confusion is that uploading a URL File to the Director and making the file available to a VOS appliance are separate steps. The file must be uploaded to the Director first and then uploaded to the target appliance before it can be selected for URL Category configuration.

2. URL File Processing Workflow

The expected workflow is:

1. Prepare and validate the CSV file.
2. Upload the URL File to the Director.
3. Upload the URL File to the target VOS appliance.
4. Associate the URL File with a custom URL Category.
5. Use the URL Category in the appropriate URL-Filtering profile and security policy.
6. Verify the result using URL-filtering logs, statistics, and CLI commands.

 

3. URL File Format

Each line in the CSV file contains three comma-separated values:

The first field must be either string or patterns:

 

Keyword	Second Field	Purpose
string	Fully qualified domain name	Matches a specific FQDN/URL entry.
patterns	Regular expression	Matches one or more URLs based on the supplied regex.

 

3.1  Supported URL Reputations

The supported reputation values range from safest to riskiest:

• trustworthy
• low_risk
• moderate_risk
• suspicious
• high_risk
• undefined

 

3.2  Examples

3.3  Regular-Expression Escaping

When using the patterns format, the backslash (\) and opening curly brace ({) characters must be escaped by preceding them with a backslash.

For example, a pattern containing these characters must be represented in the CSV with the required escaping.

3.4  File Size Limit

A URL File can contain a maximum of 65,535 lines.

4. Uploading a URL File

4.1  Upload the File to the Director

From Director view:

7. Select Configuration.
8. Select Devices > Devices.
9. Select the required organization.
10. Select the target device. The view changes to Appliance view.
11. Select Objects & Connectors > Objects > Custom Objects > URL File.
12. Click Upload.
13. Select the CSV file and verify that its contents follow the supported URL File format.
14. Click OK.

 

4.2  Upload the File to the VOS Appliance

15. From the URL File page, select the Appliance tab.
16. Click Upload.
17. In the Filename field, select the URL File that was uploaded to the Director.
18. Click OK.

5. Associate the URL File with a Custom URL Category

19. Select Objects & Connectors > Objects > Custom Objects > URL Categories.
20. Create a new URL Category or edit an existing one.
21. In the URL File field, select the URL File which was uploaded.
22. Click OK.

The custom URL Category can then be used in URL-Filtering profiles and associated security policies.

6. Verify the Complete Configuration Chain

When a URL File does not appear to work, verify the configuration in the following order:

 

Verification Point	What to Check
CSV file	Format, keyword, reputation, regex syntax, line count
Director	File is visible after upload
Appliance	File is uploaded to the correct VOS device
URL Category	The correct URL File is selected
URL-Filtering profile	The category/reputation is referenced by the profile
Security policy	The URL-Filtering profile is attached to the traffic path
Traffic/logs	The tested traffic actually traverses the VOS device

 

7. Troubleshooting URL Files

7.1  URL File Upload Fails

Check the following:

• The file is in CSV format.
• Each line contains exactly three logical fields: keyword, URL/regex, and reputation.
• The first field is exactly string or patterns.
• The reputation value is one of the supported values.
• The file does not exceed 65,535 lines.
• Regular expressions correctly escape backslashes and opening curly braces where required.
• The file is being uploaded in the correct organization/device context.

 

7.2  URL File Is Visible on the Director but Not on the Appliance

This usually indicates that the Director upload completed but the appliance upload step was not completed or was performed against a different appliance.

• Confirm that the correct appliance is selected.
• Open the Appliance tab under the URL File object.
• Verify that the required file is listed. If it is not present, upload the file to the appliance again.
• Confirm that the appliance is the same device on which the URL-Filtering policy is being tested.

 

The URL file is stored at the following location on the appliance:

The Director can upload the file only if password-less SSH authentication to the device is working correctly.

 

Test passwordless authentication — run on Director:

If it asks for a password, that means password less authentication is not working.

 

 

In that case:

View Director SSH public keys:

The keys above must match the authorized_keys file on the branch/appliance:

 

If the keys do not match, push the keys from the Director shell:

 

Check whether the admin account is locked on the appliance (run from the appliance shell prompt):

If the admin account is locked, unlock it with the following command:

 

7.3  URL File Is Not Available in the URL Category

Check the following:

• The URL File was uploaded to the appliance.
• The URL Category is being configured in the correct organization/template context.
• The URL File is being selected from the correct URL File object.
• The configuration has been committed/applied according to the deployment workflow.

 

7.4  URL Does Not Match the URL File

For string entries, verify that the entry is the expected fully qualified domain name.

For patterns entries, validate the regular expression independently before testing it on the appliance.

• Check for incorrect escaping of backslashes or opening curly braces.
• Verify that the tested URL is actually processed by the VOS device.
• Confirm that the custom URL Category containing the file is referenced by the URL-Filtering profile.
• Confirm that the URL-Filtering profile is applied to the relevant security policy.

 

To view the content of a URL file associated with a URL Category, use the following procedure:

 

Syntax:

Tip: Type ? for tenant_name-ID and scan_context to see available options. For scan_context, choose the latest value.

 

Example — connect to vsmd and run the show command:

 

Example output:

 

7.5  URL Filtering Appears Not to Be Applied

A valid URL File alone does not enforce a policy. The URL File must be associated with a URL Category, and the resulting category must participate in the URL-Filtering and security-policy configuration.

• Confirm the URL File is present on the appliance.
• Confirm the URL File is associated with the expected custom URL Category.
• Confirm the URL Category is referenced by the URL-Filtering profile.
• Confirm the URL-Filtering profile is applied to the relevant security policy.
• Confirm the test traffic traverses the appliance and matches the expected policy.