CVE Details Tools

🔗 Scraper for https://www.cvedetails.com/ data.

All the resulting datasets are available at ../../data/cve-details/ (in case you want to check it). There is also a jupyter notebook available in this folder that you can use to explore the datasets.

1. Getting the raw data

The raw dataset is not available throught the repository. There are two ways of getting the dataset:

1. Scrape the entire CVE Details by running the following script:

source scrape_data.sh

This script will save the data by year in ../../data/cve-details/year/.

2. Download our google cloud mirror by running the following command:

gdown https://drive.google.com/uc\?id\=1qGrndhvMBbY15t9VUK7XTyb1UIhHu3mY

If you want to explore the raw data, there is a jupyter notebook available with some data analysis already. If you want to use it run the following command in this folder:

jupyter notebook

2. Scrape CVE Details by year

If you want to scrape CVE Details for a specific year run the follwing command:

python scraper.py --mode=year --year=2022 --folder=../../data/cve-details/year/

To scrape the entire website check section 1.

3. Generate Dataset

Generate the dataset with references to commits in source code hosting websites such as github, bitbucket, gitlab and git.

1. Merge, plot stats and normalize CVE Details data:

source generate_data.sh

4. Filter CVE Details data by source code hosting website (github, bitbucket, gitlab or git):

source filter_data_by_source.sh github