Adding Support for New Frameworks

Adding Endpoint Detection for New Frameworks

The Hybrid Analysis Mapping (HAM) process parses source code to detect endpoints, which it uses when mapping static and dynamic scans. This parsing is framework-specific, and adding support for a new framework requires being able to parse both the relevant language and API calls to the desired framework.

This page will provide the general steps for creating a new framework parser and registering it with the ASTAM Correlator HAM module.

Not all methods and steps need to be implemented or followed. Some sections are only required if using the HAM module for vulnerability mapping. These sections can be ignored if only endpoint detection is desired. These sections will be marked as Optional.

When adding support for a new framework, most work will be done within the threadfix-ham module.

Getting Set Up

Follow the Build Guide to prepare your development environment. If vulnerability mapping is unneeded, the web and database configuration steps can be skipped - use the HAM CLI tool for testing. (See the Testing section below for details.)

Using a new endpoint generator requires some wireup before it can be used within the HAM module. When testing, you can instantiate your classes directly, or you can follow the configuration steps to have the HAM module recognize your endpoint generator. For the latter, create your Endpoint and EndpointGenerator types as described, and follow the steps listed below to register your EndpointGenerator.

New types should be created in the com.denimgroup.threadfix.framework.impl package, under a sub-package named appropriately for your framework.

The JSP and ASP.NET Web Forms framework parsers are the simplest and are recommended as a reference.

It is recommended to:

• Create your Endpoint and EndpointGenerator types without implementing yet
• Create a FrameworkType
• Register your FrameworkType

This will allow you to run your EndpointGenerator through the HAM API for testing.

Testing

Use the threadfix-cli-endpoints project for easy and robust testing.

Clone the repository to any folder, preferably the same one containing the astam-correlator project. Modify pom.xml in the astam-correlator and add <module>.../attack-surface-detector-cli</module> in the <modules> section. Replace ... with a relative path to the cloned attack-surface-detector-cli repository.

Change the version of the astam-correlator modules and reflect this change in the attack-surface-detector-cli dependencies to ensure that the CLI tool uses your custom build of the astam-correlator modules. The process of changing the version can be simplified by running the command mvn versions:set -DnewVersion=<new-version> in the astam-correlator repository, before adding the attack-surface-detector-cli module as a reference. If the current release version is 1.0.0, the recommended new version would be 1.0.1-SNAPSHOT.

Once these steps are completed, run mvn clean package in the astam-correlator repository to build the attack-surface-detector-cli utility with your custom changes. If developing in an IDE, it should update its project structure after changing the astam-correlator pom.xml; add a JAR Application target using com.denimgroup.threadfix.cli.endpoints.EndpointMain as the main class.

This process will build an executable JAR in the attack-surface-detector-cli/target directory that will run endpoint detection on a given directory.

The tool will output various validation info once complete, such as how many parameters were created, how many parameters are missing their data type, etc. The tool will emit errors if JSON parsing is not implemented; implement JSON parsing (fairly straightforward) or comment out this block while testing.

Create the Endpoint and Generator Types

Create an Endpoint (Required)

Create a class the implements the Endpoint interface. For convenience, it is recommended to extend the AbstractEndpoint class instead of implementing directly. 'get'/'set' methods can be backed by a field, and more complex methods such as getUrlPathNodes and compareRelevance can be ignored until the end.

Nontrivial methods to be implemented are described below. These can generally be a no-op or return an empty data object until the necessary data is available.

compareRelevance(String) (Optional)

This method provides an integer score for the relevance of the current endpoint for the given endpoint string. For example, comparing /foo to /foo/bar may have a relevance of 1, while /foo and /bar may have a relevance of 0. If the relevance is high, then the endpoint object may be associated with the given endpoint string. This is used for fuzzy matching of endpoints.

Scores between different endpoint types should not be compared, but scores should at least be consistent between endpoints of the same type. It is typical for an endpoint to return -1 if an endpoint string is not at all relevant.

isRelevant(String, EndpointRelevanceStrictness) (Optional)

This is a potentially stricter version of compareRelevance. EndpointRelevanceStrictness can either be Loose or Strict. If Loose, this method will generally return true if compareRelevance > 0. If Strict, then this method will only return true if the endpoint is a perfect match for the given endpoint string. For example, /foo/bar and /foo should be relevant for a loose comparison, but not a strict comparison. /foo and /foo?x=y should be relevant for both strict and loose comparisons.

This is typically implemented by cleaning the endpoint string, splitting the string and endpoint object on the / character, and doing a piece-wise comparison. This is only required if .

getUrlPathNodes() (Optional)

This will split an Endpoint object into a List<EndpointPathNode>, where each EndpointPathNode represents a segment of the endpoint path. This is used in conjunction

getVariants() (Optional)

This is used to identify endpoints that are aliases ("variants") of another. For example, /page/ and /page/index.html are variants of each other. The shortest endpoint should be used as the primary, and longer endpoints should be added as variants. The AbstractEndpoint class provides methods for managing these. It is not necessary to use variants but is recommended for clarity.

Create an EndpointGenerator (Required)

Create a class that implements the EndpointGenerator interface. Your new class will have a generateEndpoints method that returns a list of endpoints.

Your EndpointGenerator should have a constructor that accepts a File, which will be the root directory of the codebase to be analyzed. Your EndpointGenerator will be instantiated with a given file later in this guide.

Typically, an EndpointGenerator will gather the set of language-specific files to be parsed, as well as relevant configuration files if necessary. Current implementations make use of a state machine for parsing that accepts tokens. These are implemented using the EventBasedTokenizer interface, and ran using the EventBasedTokenizerRunner utility class.

Example - In ASP.NET MVC, routes are declared based on a string format that maps to a class structure. The DotNetEndpointGenerator finds these API calls in .cs files and collects these format strings, and then uses the discovered files to create different permutations of the format strings that create the endpoints. For the Java JSP/Servlet apps, the JSPEndpointGenerator uses file paths of .jsp files as endpoint paths, and checks web.xml for any servlet mappings that declare endpoints. Logic will always be specific to your framework.

Once your endpoints are generated, they are expected to have the following data:

• Parameters: A named dictionary of parameters - name, data type (string, int, etc), parameter type (ie GET query param, FILE param, etc.), accepted values (ie for enum parameters)
• File path: A path to the file on disk that executes responses to the endpoint, relative to the location of the analyzed directory
• Line range: The start and end line numbers of the code that executes responses to the endpoint
• URL path: A path that effectively represents your endpoint. This can contain wildcards, regexes, etc. as necessary for your framework
• HTTP method: The specific HTTP method that this endpoint responds to. If an endpoint path responds to multiple request methods, an endpoint should be generated for each method type

At this point your EndpointGenerator and Endpoint are ready to be incorporated into the HAM module.

Add JSON Serialization Support for your Endpoint Type (Optional)

The HAM module provides an EndpointSerialization facade that can serialize and deserialize endpoints of any type through JSON. This requires implementing an EndpointSerializer and updating EndpointSerialization.getSerializer with your new serializer. See other EndpointSerializer implementations for reference.

Create a ParameterParser (Optional)

A ParameterParser is optional for endpoint detection but required for vulnerability mapping. It is used to retrieve parameters in a given file without parsing the whole file. It has no effect on the parameters stored in your Endpoint object.

Implement the ParameterParser interface. It contains the parse(EndpointQuery) method, which should return the name of an endpoint parameter that is associated with the file, line number, etc. indicated by the EndpointQuery parameter. Your ParameterParser should have a constructor that takes a (ProjectConfig)[https://github.com/secdec/astam-correlator/blob/master/threadfix-ham/src/main/java/com/denimgroup/threadfix/framework/engine/ProjectConfig.java], which should be used to collect the data necessary to implement parse(EndpointQuery).

After creating your ParameterParser, modify ParameterParserFactory.getParameterParser(ProjectConfig) to return your new ParameterParser when your framework is detected.

Register Your New Types with the HAM Module

Add a new FrameworkType (Required)

Add an entry to the FrameworkType enum that will be used to identify your new framework. This can be found in the threadfix-entities module. You can also modify the getFrameworkType method there for convenience, which will map a string to the appropriate framework type.

Add a reference for your new EndpointGenerator (Required)

Modify EndpointGeneratorFactory.getDatabase to return an instance of your EndpointGenerator.

Add detection for your new FrameworkType (Optional)

Implement a FrameworkChecker that will be used to auto-detect the framework type of a project. Your FrameworkChecker will be ran if the user provides FrameworkType.DETECT to the EndpointGeneratorFactory.

Modify FrameworkCalculator and add a reference to your FrameworkChecker by invoking register(new MyFrameworkChecker());. Pass the rootFile parameter to its constructor.

Summary

After completing these steps you will have successfully created various classes and referenced them in the necessary parts of the codebase. Compile the project and replace your existing assemblies with the new ones. If you are manually specifying a framework, change the FrameworkType to the type that you have created. If you are using FrameworkType.DETECT, then your FrameworkType will be auto-selected if you have implemented a FrameworkChecker.