• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

gyoisamurai/GyoiThon: GyoiThon is a growing penetration test tool using Machine ...

原作者: [db:作者] 来自: 网络 收藏 邀请

开源软件名称(OpenSource Name):

gyoisamurai/GyoiThon

开源软件地址(OpenSource Url):

https://github.com/gyoisamurai/GyoiThon

开源编程语言(OpenSource Language):

Python 98.7%

开源软件介绍(OpenSource Introduction):

GyoiThon: Next generation penetration test tool

Black Hat ASIA Arsenal 2018 Black Hat ASIA Arsenal 2019

Japanese page is here.

Presentation

Documents

Slack

New function!!

The new GyoiThon (version 0.0.4) can list up your subdomain facing on the internet. And if the subdomain is published as a Web service, then GyoiThon executes a health check that a non-destructive vulnerability assessment.

Note
New function uses a Google custom search API. So if you use a new function, then you have to prepare a API key of Google Custom search.
  • ex) Listing up your subdomain.
    First, you have to prepare the domain_list.csv is following:
"Domain Name"
mbsd.jp

And you execute following command.

root@kali:~/GyoiThon# python3 gyoithon.py -i --domain_list

As a result, you get a list of sundomains associated with the specified domain.

Index Domain Sub-Domain IP Address Access Status (http) Location (http) Access Status (https) Location (https) Whois records
1 mbsd.jp mbsd.jp ['40.115.251.148'] 301 https://www.mbsd.jp/ 301 https://www.mbsd.jp/ ***
2 mbsd.jp www.mbsd.jp ['40.115.251.148'] 301 https://www.mbsd.jp/ 200 - -
3 mbsd.jp www2.mbsd.jp ['40.115.251.148'] 301 https://www.mbsd.jp/ 200 - -
  • ex) Listing up your subdomain and executing health check.
    GyoiThon executes a list up your subdomain and if the subdomain is published as a Web service (with port 80 or 443 open), then GyoiThon execute a health check that a non-destructive vulnerability assessment.
root@kali:~/GyoiThon# python3 gyoithon.py -i --domain_list --through_health_check --safety

As a result, you get a list of subdomains and assessment report.

Index Domain Sub-Domain IP Address Access Status (http) Location (http) Access Status (https) Location (https) Whois records Assessment results
1 mbsd.jp mbsd.jp ['40.115.251.148'] 301 https://www.mbsd.jp/ 301 https://www.mbsd.jp/ *** ***
2 mbsd.jp www.mbsd.jp ['40.115.251.148'] 301 https://www.mbsd.jp/ 200 - - ***
3 mbsd.jp www2.mbsd.jp ['40.115.251.148'] 301 https://www.mbsd.jp/ 200 - - ***

Overview

GyoiThon is Intelligence Gathering tool for Web Server.

GyoiThon execute remote access to target Web server and identifies product operated on the server such as CMS, Web server software, Framework, Programming Language etc,. And, it can execute exploit modules to identified products using Metasploit. GyoiThon fully automatically execute above action.

GyoiThon's main features are following.

  • Remote access/Fully automatic
    GyoiThon can fully automatically gather the information of target Web server using only remote access. You only execute GyoiThon once for your operation.

  • Non-destructive test
    GyoiThon can gather information of target Web server using only normally access.
    But, when you use a part of option, GyoiThon execute abnormally access such as sending exploit modules.

  • Gathering various information
    GyoiThon has various intelligence gathering engines such as Web crawler, Google Custom Search API, Censys, explorer of default contents, examination of cloud services etc,. By analyze gathered information using strings pattern matching and machine learning, GyoiThon can identify product/version/CVE number operated on the target web server, unnecceary html comments/debug messages, login page etc,.

  • Examination of real vulnerability
    GyoiThon can execute exploit modules to identified products using Metasploit.
    As a result, it can examine real vulnerability of target web server.

Overview

Note
If you are interested, please use them in an environment under your control and at your own risk.

Installation

  1. git clone GyoiThon's repository.
root@kali:~# git clone https://github.com/gyoisamurai/GyoiThon.git
  1. Get python3-pip.
root@kali:~# apt-get update
root@kali:~# apt-get install python3-pip
  1. install required python packages.
root@kali:~# cd GyoiThon
root@kali:~/GyoiThon# pip3 install -r requirements.txt
root@kali:~/GyoiThon# apt install python3-tk
  1. Edit config.ini of GyoiThon.
    You have to edit your config.ini.
    More information is Usage.

Usage

By using default mode without option and combination of several options, GyoiThon can gather various information of target web server.

usage:
    .\gyoithon.py [-s] [-m] [-g] [-e] [-c] [-p] [-l --log_path=<path>] [--no-update-vulndb]
    .\gyoithon.py [-d --category=<category> --vendor=<vendor> --package=<package>]
    .\gyoithon.py [-i]
    .\gyoithon.py -h | --help
options:
    -s   Optional : Examine cloud service.
    -m   Optional : Analyze HTTP response for identify product/version using Machine Learning.
    -g   Optional : Google Custom Search for identify product/version.
    -e   Optional : Explore default path of product.
    -c   Optional : Discover open ports and wrong ssl server certification using Censys.
    -p   Optional : Execute exploit module using Metasploit.
    -l   Optional : Analyze log based HTTP response for identify product/version.
    -d   Optional : Development of signature and train data.
    -i   Optional : Explore relevant FQDN with the target FQDN.
    -h --help     Show this help message and exit.

Preparation.

  1. Edit target file host.txt.
    You have to write target web server to the host.txt.
    Writting format is protocol FQDN(or IP address) Port Crawling_root_path.
  • Example.
https gyoithon.example.com 443 /

If you want to indicate multiple target information, you have to write below.

https gyoithon.example.com 443 /
http 192.168.220.129 80 /vicnum/
https www.example.com 443 /catalog/
Note
You insert / at the beginning and end of Root Path.
  1. Edit configuration file config.ini.
    Parameters to be changed by the user are defined in the setting file config.ini.
    If you want to change parameters, edit config.ini.
    Detail of config.ini is here.

Execution of GyoiThon.

1. Default mode.

root@kali:~/GyoiThon# python3 gyoithon.py

The default mode gathers following minimum information.

  1. Gathering of HTTP responses by Web crawling.
  2. Identification of product/version using string pattern matching.
  3. Examination of CVE number (from NVD) for identified products.
  4. Examination of unneccesary HTML/JavaScript comments.
  5. Examination of unneccesary debug messages.
  6. Examination of login pages.
  • Crawling setting
    GyoiThon uses Scrapy that Python's library.
    By change the parameters in config.ini, you can change setting of Scrapy.
Category Parameter Description
Spider depth_limit Maximum depth of crawling. Default value is 2 layer.
delay_time Delay time of crawling. Default value is 3 (sec).
time_out Spider close option. Timeout of crawling. Default value is 600 (sec).
item_count Spider close option. Maximum items. Default value is 300.
page_count Spider close option. Maximum items per page. Default value is 0 (no limit).
error_count Spider close option. Maximum errors. Default value is 0 (no limit).
  • Examination speed setting
    The examination number and HTTP response size greatly affect examination times.
    By change the parameters in config.ini, you can adjust examination speed.
Category Parameter Description
Common max_target_url Maximum examination URL number. If the URL number gathered by Web Crawling exceeds this parameter value, excess URL number is discarded. Default value is 100. 0 is unlimited.
max_target_byte Maximum examination response size. If the response size exceeds this parameter value, excess response size is discarded. Default value is 10000 byte. 0 is unlimited.
scramble The URL list gathered by Web crawling is randomly ordered. Default value is 1 (validity). 0 is invalid.
Note
The examination speed and accuracy are trade-off.

2. Examination of cloud services mode.

root@kali:~/GyoiThon# python3 gyoithon.py -s

By add -s option, GyoiThon identifies target web server uses cloud service or not in addition to default mode.
Before execution, you must change the below parameter of config.ini.

Category Parameter Description
CloudChecker azure_ip_range Source URL of Azure Datacenter IP Ranges.

This parameter is source URL of Azure Datacenter IP range. This URL is changed a few per day. So, you must get the latest URL from link "click here to download manually" of page "Microsoft Azure Datacenter IP Ranges" and set it to above parameter before execute GyoiThon.

3. Machine Learning analysis mode.

root@kali:~/GyoiThon# python3 gyoithon.py -m

By add -m option, GyoiThon identifies products/version using Machine Learning (Naive Bayes) in addition to default mode.

4. Google Hacking mode.

root@kali:~/GyoiThon# python3 gyoithon.py -g

By add -g option, GyoiThon identifies products/version using Google Custom Search API in addition to default mode. Before execution, you must set API key and Search engine ID to the below parameters.

Category Parameter Description
GoogleHack api_key API key of Google Custom Search API.
search_engine_id Google search engine ID.
Note
You can use free Google Custom Search API of 100 queries per day. But, if you want to use more than 100 queries, you must pay fee the Google Custom Search API service.

5. Exploration of default contents mode.

root@kali:~/GyoiThon# python3 gyoithon.py -e

By add -e option, GyoiThon explores the default contents of products such as CMS, Web server software in addition to default mode.
By change the parameters in config.ini, you can change setting of exploration.

Category Parameter Description
ContentExplorer delay_time Delay time of exploration. Default value is 1 (sec).
Note
When you use this option, may be affected to heavy load of server because of GyoiThon execute numerous accesses (hundreds accesses) against the target web server. In addition, by numerous 404 error logs are wrote to access log, it may be to caught by SOC (Security Operation Center). So, if you use this option, please notify person concerned such as SOC, administrator and use them in an environment under your control and at your own risk and.

6. Censys cooperation mode.

root@kali:~/GyoiThon# python3 gyoithon.py -c

By add -c option, GyoiThon examines open port number and server certification using Censys.
Before execution, you must set API key and Secret key to the below parameters.

Category Parameter Description
Censys api_id API key of Censys.
secret Secret key of Censys.

7. Metasploit cooperation mode.

root@kali:~/GyoiThon# python3 gyoithon.py -p

By add -p option, GyoiThon examines real vulnerabilities such as DoS and backdoor using Metasploit in addition to default mode.
Before execution, you must launch RPC server of Metasploit and set below parameters in config.ini.

Category Parameter Description
Exploit server_host Allocated IP address to the RPC Server (msgrpc).
server_port Allocated port number to the RPC Server (msgrpc).
msgrpc_user User ID for authorization of msgrpc.
msgrpc_pass Password for authorization of msgrpc.
LHOST Allocated IP address to the RPC Server (msgrpc).
Note
When you use this option, may be heavily affected to server operation because of GyoiThon execute the exploit against the target web server. In addition, this option may be caught by SOC (Security Operation Center) because of exploits are like a real attacks. So, if you use this option, please notify person concerned such as SOC, administrator and use them in an environment under your control and at your own risk and.

8. Stored logs based analysis mode.

root@kali:~/GyoiThon# python3 gyoithon.py -l --log_path="Full path of stored logs"

By add -l option, GyoiThon executes various examination using stored HTTP responses without web crawling.

This mode assumes the web application that GyoiThon cannot execute web crawling.
GyoiThon can execute various examination similar web crawling of default mode using stored HTTP responses gathered by local proxy tool.

Note
Log file's extension is .log.

9. Combination of multiple options.

Combination of "Examination of cloud services mode" and "Machine Learning analysis mode".
root@kali:~/GyoiThon# python3 gyoithon.py -s -m
Combination of "Examination of cloud services mode" and "Google Hacking mode".
root@kali:~/GyoiThon# python3 gyoithon.py -s -g
Combination of "Examination of cloud services mode", "Machine Learning analysis mode" and "Google Hacking mode".
root@kali:~/GyoiThon# python3 gyoithon.py -s -m -g
All option.
root@kali:~/GyoiThon# python3 gyoithon.py -s -m -g -e -c -p -l --log_path="Full path of stored logs"

10. Generating signature/train data mode.

root@kali:~/GyoiThon# python3 gyoithon.py -d --category=CMS --vendor=joomla! [email protected]@_origin.tar.zip

By add -d option, GyoiThon generates signatures/train data for detecting Web products operated on the target Web Server.
The options is following.

Option Description Example
--category Product's category in signature/train data. OS or WEB or FRAMEWORK or CMS.
--vendor Product's vendor name in signature/train data. wordpress, joomla!, drupal.
--package Importing target package file name. You have to separate three fields that product name, version and extension using @. [email protected]@.tar.gz, [email protected]@_origin.tar.zip, [email protected]@.tar.gz.

Check report.

After finished execution of GyoiThon, reports of each target are generated to the following path.

root@kali:~/GyoiThon/report# ls
gyoithon_report_192.168.220.129_80_1082018338.csv
gyoithon_report_192.168.220.129_80_bodgeit.csv
gyoithon_report_192.168.220.129_80_cyclone.csv
gyoithon_report_192.168.220.129_80_vicnum.csv
gyoithon_report_192.168.220.129_80_WackoPicko.csv
gyoithon_censys_report_www.gyoithon.example.com_443_test.csv

GyoiThon generates following two types report.

  • gyoithon_report_target FQDN(or IP address)_Port number_Root Path.csv.
    This is main report that mainly including product name, version, cve etc,.
    Report format is gyoithon_report_target FQDN(or IP address)_Port number_Root Path.csv.
    Each column's detail is following.
Column Description Example
fqdn FQDN of target web server. www.gyoithon.example.com
ip_addr IP address of target web server. 192.168.220.129
port Port number of target web server. 80
cloud_type Cloud service name (Azure or AWS or GCP or Unknown). AWS
method Examination way of GyoiThon. Crawling
url Accessed URL. http://192.168.220.129:80/WackoPicko/admin/index.php?page=login
vendor_name Vendor name of identified products. apache
prod_name Identified products. http_server
prod_version Version of identified products. 2.2.14
prod_trigger Trigger of identified products. Apache/2.2.14
prod_type Product category (Web or CMS or Framework etc..). Web
prod_vuln CVE number according to identified products (desc CVSS score). CVE-2017-3167, CVE-2017-3169, CVE-2017-7668 ...
origin_login Login page is existing or not (Log: Analysis using Machine Leaerning, Url: Analysis using string pattern matching in URL. Log : 37.5 %\nUrl : 100.0 %
origin_login_trigger Trigger of identifed login page. Log : name",<input type="password"\nUrl : login
wrong_comment Identified unnecessary comments. パスワードは「password1234」です。
error_msg Identified unnecessary debug messages. Warning: mysql_connect() ..snip.. in auth.php on line 38
server_header Server header of HTTP response. Server: Apache/2.2.14 (Ubuntu) mod_mono/2.4.3 PHP/5.3.2
log Path of raw data. /usr/home/~snip~/http_192.168.220.129_80_20181112170525765.log
date Examination date. 2018/11/12 17:05:25
  • gyoithon_censys_report_target FQDN(or IP address)_Port number_Root Path.csv.
    This is search result report using Censys that including open ports, certification information etc,.
    Report format is gyoithon_censys_report_target FQDN(or IP address)_Port number_Root Path.csv.
    Each column's detail is following.
Column Description Example
fqdn FQDN of target web server. www.gyoithon.example.com
ip_addr IP address of target web server. 192.168.220.129
category Information category. Server Info or Certification Info
open_port Open web port. 443
protocol Protocol of open web port. https
sig_algorithm Signature algorithm of certification. SHA256-RSA
cname Common name of certification. www.gyoithon.example.com
valid_start Validity start date of certification. 2018-08-15T00:00:00Z
valid_end Validity end date of certification. 2019-09-16T12:00:00Z
organization Organization name of certification. GyoiThon coorporation, Inc.
date Examination date. 2018/11/22 11:19:36
Note
Because Censys needs several days to several weeks to survey the entire Internet, the information obtained from Censys may not be up-to-date.

Tips

1. How to manually add new signature (string matching patterns).

signatures path includes below files.

root@kali:~/GyoiThon/signatures/ ls
signature_product.txt
signature_default_content.txt
signature_search_query.txt
signature_comment.txt
signature_error.txt
signature_page_type_from_url.txt

signature_product.txt

This is string matching patterns for identification of product in default mode.
If you want to add new string matching pattern, you have to write it such following format.

Format: field1@field2@field3@field4@field5
Type Field# Description Example
Required 1 Product Category. CMS
Required 2 Vendor name. drupal
Required 3 Product name. drupal
Optional 4 Version binded with this signature. 8.0
Required 5 Regex of identifying product. .*(X-Generator: Drupal 8).*

If you don't need optional field, you must set * to this field.

  • Example
CMS@wordpress@wordpress@*@.*(WordPress ([0-9]+[\.0-9]*[\.0-9]*)).*
CMS@drupal@[email protected]@.*(X-Generator: Drupal 8).*
Note
If you want to extract product version, you write two regex groups (the second regex is used for version extraction).

signature_default_content.txt

This is string matching patterns for identification of product in Exploration of default contents mode.
If you want to add new string matching pattern, you have to write it such following format.

Format: field1@field2@field3@field4@field5@field6@field7@field8
Type Field# Description Example
Required 1 Product Category. CMS
Required 2 Vendor name. sixapart
Required 3 Product name. movabletype
Optional 4 Version binded with this signature. *
Required 5 Explore path /readme.html
Optional 6 Regex of to confirm product. .*(Movable Type).*
Optional 7 Regex of identifying version. (v=([0-9]+[\.0-9]*[\.0-9]*))
Required 8 Login page or not. Login page is 1, Not login page is 0

If you don't need optional field, you must set * to this field.

  • Example
Web@apache@http_server@*@/server-status@*@Version:.*(Apache/([0-9]+[\.0-9]*[\.0-9]*))@0
CMS@sixapart@movabletype@*@/readme.html@.*(Movable Type).*@(v=([0-9]+[\.0-9]*[\.0-9]*))@0
Note
If you want to extract product version, you write two regex groups (the second regex is used for version extraction).
Note
If GyoiThon cannot confirm the product by just Explore path, you need to indicate the Regex of to confirm product field. GyoiThon accesses the URL that Explore path and examines the HTTP response using Regex of to confirm product. If this regex matches, GyoiThon judges that the product exists.

signature_search_query.txt

This is Google Custom Search query for identification of product in Google Hacking mode.
If you want to add new query, you have to write it such following format.

Format: field1@field2@field3@field4@field5@field6@field7@field8

鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap