IT Visibility Data Export API Usage
Refer to api documentation page for more information on API end points.
Definitions
Parquet File: It is free and open-source file format and column-oriented data file format designed for efficient data storage and retrieval. It is good for storing big data of any kind (structured data tables, images, videos, documents).
CSV File: A CSV (comma-separated values) file is a text file that has a specific format which allows data to be saved in a table structured format.
Snapshot: It is complete export parquet file of the inventory export data associated with all datasets at specific moment in time. Snapshots tend to use more resources because they export all the org’s data at once. It runs once every 24hrs.
Delta: It is delta export parquet file of the inventory export data associated with limited datasets. It runs once every 3hrs. It collects delta data about add/remove/change events from the previous run. It stores inventory data as sequence of events from the previous run.
Delta Opcodes
| Opcode | Description |
|---|---|
| Init | It is initialisation state – what is already inside the database on first request before we start sending changes in subsequent deltas. |
| Add | Instruction to add this resource as a new record. |
| Remove | Instruction to remove this resource. |
| Change | Instruction to update this resource. |
NOTE: Active or ignored status records are retained
Token for API call
Refer to authentication page for fetching access token required to authenticate API requests.
Parquet via Export API
Snapshots export all of an orgs current data at a specific moment in time. Snapshot job runs once every 24hrs.
Delta jobs stream data as it changes every three hours with add, remove and change events and the values within the file are sequential. Each delta file represents a set of changes so a number of delta files may be required to build a full dataset, but the data will be up to 21 hours more up to date than a snapshot. Delta parquet files are appended with timestamp to the filename to ensure correct ordering of files for readability and ease-of-use.
| Snapshot Job | Delta Job |
|---|---|
| Runs once every 24 hours | Runs once every 3 hours |
| Collects orgs latest full export datasets based on org which has access. | Collects the changes from the previous delta run. |
| Runs for entire datasets | Runs against the inventory datasets. |
CSV via queries API
| Snapshot Job | Delta Job |
|---|---|
| Collects full inventory CSV data export for snapshot datasets. | The first delta will contain inits, the original status of the inventory dataset. All deltas after this will contain add/change/remove events to represent a stream of how the data has changed from last export. |
Query API supports Create, Show and Download Endpoints. Refer to query page for more details.
CSV file export for a specific dataset can be created using an Athena query according to the request. The following list of Snapshot and delta query names that are supported:
| Snapshot Query Names |
|---|
| hardware_business_services |
| hardware_contextualized |
| hardware_evidence |
| hardware_inventory |
| hardware_inventory_source |
| hardware_technopedia |
| hardware_technopedia_lifecycle |
| network_adapters |
| software_evidence |
| software_inventory |
| software_inventory_source |
| software_technopedia |
| software_technopedia_lifecycle |
| software_technopedia_versions |
| business_services_applications |
| business_services_assets |
| business_services_dependent_applications |
| software_open_source |
| software_open_source_component_license |
| software_open_source_version_license |
| software_open_source_version_url |
| software_vulnerability |
| cpe_catalog |
| software_vulnerability_cpe |
| software_vulnerability_cve |
| software_vulnerability_kb_article |
| vulnerable_software |
| hardware_model_profile |
| hardware_model_power |
| hardware_model_connectivity |
| software_mashup |
| device_mashup |
| Delta Query Names |
|---|
| hardware_inventory |
| hardware_inventory_source |
| network_adapters |
| software_inventory |
| software_inventory_source |
| vulnerable_software |
| software_mashup |
| device_mashup |