Building a ground truth dataset of real security patches for machine learning and testing activities.
Requirements installation:
virtualenv --python=python3.8 venv
source venv/bin/activate
pip install -r requirements.txt
Jupyter notebooks are available. Charts and wordclouds are saved at notebook/charts.
source venv/bin/activate
cd notebooks
jupyter notebook
Scrapping cvedetails for CVEs in 1999:
source venv/bin/activate
cd scraper
python3 scrape_cve_details.py --mode per_year -file data/1999.csv -year 1999
Merge data from data folder:
source venv/bin/activate
cd scraper
python3 merge_cve_data.py -folder data/ -file cve_details.csv
Clean cve_details data:
source venv/bin/activate
cd scraper
python3 clean_data.py -file cve_details.csv
Merge all datasets:
source venv/bin/activate
python3 scripts/merge_datasets.py --mode merge -file positive.csv
Get CVE Details to complete the data from other datasets:
source venv/bin/activate
python3 scripts/get_metadata.py --source cvedetails -file positive.csv
Configure the github API at scripts/config/.
source venv/bin/activate
cd scripts/config/
cp github_template.json github.json
Add a token and username to the file with permissions for repositories and users information.
Get GitHub data:
source venv/bin/activate
python3 scripts/get_metadata.py --source github -file positive.csv
Adding the extension of files involved in changes (ext_files):
source venv/bin/activate
python scripts/add_features.py --feature ext_files -file positive.csv
Adding programming languages (lang):
source venv/bin/activate
python scripts/add_features.py --feature lang -file positive.csv
To download Scala samples:
source venv/bin/activate
python3 scripts/download.py -file dataset/positive.csv -folder code_samples -language Scala