Connect Data to Influent

Once you have transformed and indexed your transaction and entity data, you must specify how Influent should:

  • Connect to your data sources
  • Map the fields in your data to the transaction and entity properties required by Influent

Specifying the Connection Details

The server.config file in your Influent project's src/main/resources/ folder specifies how Influent connects to your database and Solr instance.

To edit the database and Solr connection details
  1. Open the server.config file.
  2. Add or edit the following values in the Database Server Properties section of the file:
    Property Description
    influent.midtier.user.name Username with which to access your database
    influent.midtier.user.password Password for the username with which to access your database
    influent.midtier.database.type Type of database in which your transaction and entity data is stored (mssql, mysql, hsql or oracle)
    influent.midtier.database.url Location of the database in which your transaction and entity data is stored (e.g., jdbc:mysql://mysql.server.come/MyDB)
    influent.midtier.database.driver Location of the JDBC driver (e.g., org.hsqldb.jdbcDriver) for the type of database to which you want to connect

    NOTE: If you have a Microsoft SQL Server database, you can specify its location using the following legacy properties. However, the presence of the database.url and database.driver properties will automatically override the legacy properties.

    • influent.midtier.server.name (URI)
    • influent.midtier.server.port
    • influent.midtier.database.name

  3. Edit the Solr Server Properties section to specify the URL of the core you added to your Solr instance:

    • For single core instances containing both entity and transaction data:

      influent.midtier.solr.url = http://solr.uncharted.software:8983/solr/aml
      
    • For instances with separate cores for entity and transaction data:

      influent.midtier.solr.entities.url = http://solr.uncharted.software:8983/solr/bitcoin-entities
      influent.midtier.solr.links.url = http://solr.uncharted.software:8983/solr/bitcoin-links
      
  4. Edit the influent.pattern.search.remoteURL value in the Pattern Search Database section to specify the URL of your Graph QuBE server (e.g., http://solr.uncharted.sofware:8805/pattern/search/example).

  5. Save the server.config file.

Connecting to Your Raw Data Source

In addition to requiring access to the LinkFlow and Entity tables created by the DataViewTables script, Influent needs to connect to your raw data source to access your unique transaction IDs.

To connect to your raw data source
  1. Open the database-config.xml file in the src/main/resources/ directory of your project.
  2. Scroll to the FIN_LINK <dataTableSchema> and update the table <memberKey> value to point to the table in your raw data that contains your unique transaction IDs.
  3. Save the file.

Understanding Data Types

Influent Service Provider Interfaces (SPIs) enable you to provide runtime-injected modules for search, data access, clustering and other services. The SPI protocols are defined in cross-language form with Apache Avro.

To map the fields in your data sources to the data types defined in the SPIs, you must edit the property-config.xml file in your project's src/main/resources/ folder.

The following sections describe the data types you should be aware of when editing this file. Complete documentation of each type can be found in the DataEnums_vX.X.avdl file in influent-spi/src/main/avro/.

FL_RequiredPropertyKey

Each Influent app requires the presence of the following case-sensitive property keys. You must ensure that each one is mapped to an appropriate field or fields in your transaction or entity data.

Type Key Description
Entities NAME Entity name. Generally user friendly, and does not need to be unique.
ID Unique entity ID. Should correspond to an entity in the EntitySummary table created when you executed the DataViewTables script.
Transactions ID Unique transaction ID. Should map to your raw data source.
FROM Entity ID of the source account for a transaction.
TO Entity ID of the destination account for a transaction.
DATE Date or range on or during which a transaction occurred.
AMOUNT Value of a transaction.

FL_ReservedPropertyKey

The following property keys are reserved for search keywords and should not be reused in your property-config.xml file:

  • TYPE
  • ORDER
  • MATCH

FL_PropertyTag

The Avro data model includes FL_PropertyTag names defined in the application layer as a taxonomy of user and application concepts independent of the data source. These tags allow application semantics to be reused with new data with minimal design and development.

The following FL_PropertyTags have applications that should be noted before you edit your property descriptors. For a complete list of FL_\PropertyTags, see the the DataEnums_vX.X.avdl file in influent-spi/src/main/avro/.

PropertyTag Description
SHARED_IDENTIFIER Indicates that the property should be used to populate Influent's Advanced Account Search dialog when a user chooses to search for entities that are similar to a selected account.
FROM_LABEL Indicates that the property should be used as a label in the To/From columns of the Transaction search results.
TO_LABEL

Mapping Fields to Property Descriptors

To configure the property descriptors for your app
  1. Open the property-config.xml file in your project's src/main/resources/ folder. There are two sections in this file you must edit:
    • <entities>, which contains the configuration for entity data fields
    • <links>, which contains the configuration for transactions data fields
  2. Create a <type> element for each entity and transaction type in your source data. In the Bitcoin example application, there is one entity type and one transaction type:

    <entities>
        <type key="account" friendlyText="Account"/>
        ...
    </entities>
    ...
    <links>
        <type key="financial" friendlyText="Financial"/>
        ...
    </links>
    
  3. List your entity and transaction attributes as <property> elements.

    NOTE: The order in which properties are defined first is the order in which they are displayed in the user interface.

    <property key="NAME" dataType="string" friendlyText="Name" 
              levelOfDetail="key" searchableBy="freeText" 
              memberKey="Label" multiValue="false">
        ...
    </property>
    

    Where:

    Attribute Required? Description
    key Yes Unique field identifier entered as a custom value or an FL_RequiredPropertyKey. Users can enter the key value into the Accounts or Transactions search field followed by a term to search the corresponding field (e.g., NAME:"daniel").

    NOTE: You cannot reuse any of the reserved property keys FL_ReservedPropertyKey.

    dataType Yes Identifies the type of data contained in the field: boolean, date, double, geo, image, integer, long, real, string.
    friendlyText No Alternate user-friendly field label to display in Influent.
    levelOfDetail No Indicates where in the search results the field should be displayed:
    • key: The field should be a column header in the search results, and should appear in the header of each individual result.
    • summary: The field should appear in the header of each individual result.
    • full: The field should appear only in the expanded view of each individual search result.
    • hidden: The field should not appear in the client.
    If this attribute is not included, the field may appear in the summary view of each individual result as well as the expanded view.
    searchableBy No Indicates how the field can be searched:
    • descriptor: Users can search the field by entering a value in the search bar and prefixing it with the key value followed by a colon. The field will also be available as a drop-down option on the Advanced Search dialog. This is the default value.
    • freeText: When users enter free text in the search bar (i.e., no search field is specified), the field should be searched by default.
    • none: The field should not be available for searching.
    memberKey No If the field name in the data sources does not match the property key, memberKey defines that field name. However, if the field name varies by type or does not exist in every type, <memberOf> elements must be used instead.
    sortable No Indicates whether users can sort results based on the values in the field. Defaults to true.

    NOTE: Sorting is not currently supported for multivalued fields.

    .
    multiValue No Indicates whether the field can support multiple values separated by commas. Defaults to false.
  4. For each property with a field name that varies by type or does not exist in every type, add one or more <memberOf> elements to specify the applicable account or transaction fields.

    <property key="NAME" dataType="string" friendlyText="Name" 
              searchableBy="freeText" levelOfDetail="key">
                  <memberOf typeKey="lender" memberKey="LenderName"/>
                  <memberOf typeKey="loan" memberKey="LoanName"/>
                  <memberOf typeKey="partner" memberKey="PartnerName"/>
                  ...
    </property>
    

    Where:

    Attribute Required? Description
    typeKey Yes Account or transaction type for which the corresponding property is applicable. Must match one of the type keys you entered in step 2 above.
    memberKey Yes Name of the account or transaction type-specific field that contains the values for the element. Make sure you exactly match the field name as it appears in your data source.
  5. For each property that you want to map to an FL_PropertyTag, add one or more <tag> elements.

    <property key="AMOUNT" dataType="double" friendlyText="Amount (USD)"
              levelOfDetail="key" searchableBy="freeText" memberKey="Amount">
                  <tags>
                      <tag>USD</tag>
                  </tags>
    </property>
    
  6. To define a non-primitive property like FL_GeoData (dataType="geo"), list the primitive fields it comprises under a <fields> element. Geo properties can be particularly useful for dynamic clustering.

    <property key="GEO" dataType="geo" friendlyText="Location" levelOfDetail="summary">
        <fields>
            <field name="text" key="LocationText" searchable="false"/>
            <field name="cc" key="CountryCode" searchable="true"/>
            <field name="lat" key="Latitude" searchable="false"/>
            <field name="lon" key="Longitude" searchable="false"/>
        </fields>
    </property>
    

    Where:

    Attribute Required? Description
    name Yes The type of geographic data contained in the field:
    • text: Unstructured text field representing an address or other place reference
    • lat: Latitudinal coordinate
    • lon: Longitudinal coordinate
    • cc: Three-character ISO country code
    key Yes Reference to a unique field identifier defined elsewhere in the property-config.xml file.
    searchable No Indicates whether the field is searchable when the parent meta-property is specified. Defaults to false.
  7. Edit the <defaultOrderBy> element to specify the order in which search results should be displayed. If you specify multiple rules, they are applied according to the order in which they appear in the file.

    NOTE: The defaultOrderBy values are only triggered if users do not explicitly specify a custom order in their search terms.

    <defaultOrderBy propertyKey="ENTITY" ascending="true"/>
    <defaultOrderBy propertyKey="DATE"/>
    

    Where:

    Attribute Required? Description
    propertyKey Yes The key of the field on which you want to order results.
    ascending No Indicates whether to sort results in ascending order (true/false). If not included, results are sorted in descending order.
  8. Edit the <searchHint> element to provide a helpful tip to display on an empty search page.

  9. Save the file.

Handling One-to-Many Transactions

One-to-many transactions are common in email transaction records, where a single entity sends the same message to multiple recipients at the same time. Influent treats these transactions as sets of separate but related transactions.

For example, an email from jsmith@influent.org to jdupont@influent.org and jperez@influent.org is stored as two records:

  • One from jsmith@influent.org to jdupont@influent.org
  • One from jsmith@influent.org to jperez@influent.org

Both records have the same contents and are identified as belonging to the same transaction through a unique EmailRecNo attribute.

To configure Influent to group related transaction records
  • Add a groupByField element in the links section of the property-config.xml file, where the fieldName attribute indicates the unique identifier shared by the related records.

    <groupByField fieldName="EmailRecNo"/>
    

Next Steps

To configure the dynamic entity clustering methods used by Influent, see the Clustering Configuration topic.