Determine the Contents of Encrypted BitTorrent Traffic

September 24, 2017September 24, 2017 marc

There are many ways to determine what someone is doing online by analyzing their network traffic. Capturing network traffic and using Wireshark dissectors and statistics can even help when a large amount of the traffic is encrypted. A domain name system (DNS) query alone could provide enough information to act on. I had talked about determining what someone is doing online with non-technical friends before. This past month I received a text message from a one of them who essentially asked the following: Do I still need to use a virtual private network (VPN) if I select ‘require encrypted peers’ in my BitTorrent client? I had my answer (don’t illegally download anything and yes, use the VPN) but I didn’t have a technical reason for BitTorrent traffic specifically. I decided to experiment so that I could provide that technical reason and to have a more robust understanding of what the BitTorrent protocol is doing.

I regularly use BitTorrent for legal downloads. When Offensive Security released an updated Kali Linux 2017.2 virtual machine image this week, I used BitTorrent to download and share it. Another distribution I download this way is Raspbian. I decided to use qBittorrent, a Windows BitTorrent client, to test what using encryption among peers actually means.

The default peer encryption setting is located in Preferences > BitTorrent > Encryption mode: and has the default value of Prefer Encryption. Below is a screen shot.

The default encryption mode in qBittorrent is to prefer encryption.

Control, Capture, Analyze

I will be doing null hypothesis testing and will change the peer encryption option for two separate packet captures (pcaps) and compare them. Instead of using the default setting I will use disable encryption and require encryption. Using each configuration I will download the Raspbian system image and use hashing to ensure they result in an identical file. I will use a set of bytes from the file to search for it being in each packet capture as clear text. The null hypothesis (H0) is that the download exists in both pcaps in clear text. My hypothesis (H1) is that the download does not exist in the require encryption pcap as clear text. I am hoping to use Wireshark to reject the null hypothesis.

Using tcpdump and a port mirror I captured the network traffic and ended with these two pcaps:

filename, size, sha1
bt_disable_encryption.pcap, 432.4 MB, 65586ad1b23aeaccddbf17c99a7611e0f94c8e97
bt_require_encryption.pcap, 421.6 MB, 4c0024e9aa4ca9fcc0aadf4d9e8903e864e5a6b7

The resulting download from each was identical (as expected) and verified through hashing. I used the first 16 bytes from the download for filtering. I wouldn’t use this as a filter when given a large pcap, but mine is small and was a targeted capture. If I use too few bytes as a filter, I may be filtering for any *.zip file signature. If I use too much data, I may exceed the size of a payload in a single frame. I use xxd for a continuous byte dump and sed to put it in the format that tshark expects for the filter.

$ xxd -p -c 16 ~/2017-08-16-raspbian-stretch-lite.zip | head -n 1 | sed 's/\(..\)/\1:/g' | sed '$s/.$//'
 50:4b:03:04:14:00:00:00:08:00:74:5c:10:4b:da:96
$ tshark -r ~/bt_disable_encryption.pcap frame contains 50:4b:03:04:14:00:00:00:08:00:74:5c:10:4b:da:96

Using the first 16 bytes of our download, we can filter and find the location of the same data within the pcap.

We can see that frame 241939 contains the hex we filtered for. Opening the pcap in Wireshark and filtering on the frame allows us to see the bytes in the payload.

The bt_disable_encryption pcap contains the payload in clear text.

This is good news and what we expected. In the screen shot below we can see that using the same filter against the bt_require_encryption pcap also yields good results; it does not find the bytes in any of the frames.

The same filter yields no result when ran against the bt_require_encryption pcap.

I am not testing the payloads for entropy or signs of encryption, so the most correct statement I can make is that the download is not being sent as clear text when require encryption is selected in qBittorrent. This distinction is important, as it more accurately describes what I am doing. As a result, we reject the null hypothesis. The bt_require_encryption pcap does not contain the file in clear text.

Results

With the testing, I could determine two facts:

When disable encryption is selected the data is transmitted in clear text.
When require encryption is selected the data is not transmitted in clear text.

It was nice to see this in action but I wanted to explore the pcaps to see if there was more information to be had. All I looked for previously was the clear text transmission, which is only a part of the process of sharing data securely. I wanted to answer a new question:

Does using encrypted peers prevent an eavesdropper from determining the payload using only the BitTorrent protocol traffic? (Hint: No!)

Exploratory surgery

I decided to look at each pcap using Wireshark. I’m always amazed at how much information the built-in dissectors can parse out. I started noticing BitTorrent handshake traffic between peers. This appears to occur when peers communicate for the first time after completing a TCP three-way handshake. Both peers send a BitTorrent handshake that contains, among other data, a SHA1 hash of the info dictionary and a Peer ID. Below is a screen shot of a sample from the bt_require_encryption pcap.

Peer ID

The Peer ID is a 20-byte value that is unique to each peer. According to the BitTorrent Protocol Specification, there are currently no guidelines for generating a Peer ID. The observed Peer IDs from both pcaps appeared to be in hex and it’s possible that a quick SHA1 hash of random input is used for generation.

SHA1 Hash of the Info Dictionary (info_hash)

The SHA1 Hash of the Info Dictionary was identical and present in both the bt_disable_encryption and bt_require_encryption BitTorrent handshakes. This a huge red flag and peaked my interest. It makes sense that peers need to verify that they will exchange the correct data, but they appeared to be making this negotiation outside of any encrypted stream.

To better understand the meaning of the info_hash I wanted to find out what input is used for generation. It isn’t a hash of the *.torrent file itself. The info_hash is contained within the *.torrent file, the BitTorrent magnet uniform resource identifier (URI) link, and it isn’t a hash of the file(s) to be exchanged. After some searching, the BitTorrent Protocol Specification helped by defining it as a 20-byte SHA1 hash of the value of the info key from the metainfo file. The value of the info key from the metainfo file is a bencoded dictionary. I was able to parse out the bencoded dictionary and re-create the info_hash for the Raspian torrent with some simple hex editing.

Knowing that BitTorrent traffic with the peer encryption required setting enabled still transmits the info_hash in clear text, I decided to see if I could accurately determine encrypted BitTorrent traffic contents by simply entering in the corresponding info_hash on Google and browsing the top results. This worked.

The Process

Using tshark and bash I can filter info_hash values from BitTorrent handshakes. I could then query Google using those info_hash values and use the results to estimate the payload. I chose to automate this with Python 3 and BeautifulSoup. Below is a screen shot of output using the bt_require_encryption pcap as input:

This proof of concept works against traffic with either encrypted or unencrypted BitTorrent peers.

Conclusion

You have a high likelihood of determining the contents of encrypted BitTorrent traffic by collecting info_hash keys and using public databases. For proof of concept I have created a Python script to automate looking through pcaps and returning the first five Google results of any info_hash present in the traffic.

To my friend: Keep using your VPN. 🙂

Disruptive Technology Theory

July 6, 2017July 6, 2017 marc

Disruptive Technology Theory has come up frequently in my coursework and it is largely misunderstood or errantly attributed to firms or ideas that are successful. There’s a collection of articles I will post soon (I’m finishing up a summer class at the University of Arizona this week!) but thought this podcast episode warranted a share on it’s own. Here is a link and description:

The Disruptive Voice, episode 15: Is Uber Disruptive?

Is Uber disruptive? We asked five experts on the theory of disruptive innovation this question and received varying responses, yet their prescriptions for what lies ahead for Uber and the incumbent taxi companies vary less than you might think. In this episode, we revisit Professor Clay Christensen’s December 2014 article in Harvard Business Review, “What Is Disruptive Innovation,” with co-author Rory McDonald, Innosight Managing Partner Scott D. Anthony, Christensen Institute co-founder Michael B. Horn, and Forum Senior Researchers Tom Bartman and Efosa Ojomo. Also discussed: the platform business model through the lens of disruptive innovation and what’s next for Uber.

Using ScanSnap Manager to OCR non-ScanSnap PDFs

May 31, 2017May 31, 2017 marc

I had some PDFs that I wanted to perform optical character recognition (OCR) processing on. I have a Fujitsu ScanSnap and wanted to use the ScanSnap Manager software to do this. The management software checks supplied PDFs and will only perform procession on those which originated using ScanSnap hardware. I wanted to circumvent this and it ended up being easy.

PDFs created with a ScanSnap have the Exif tag “creator” with the model string value. You can use ExifTool by Phil Harvey to print and modify Exif data. For example:

$ exiftool -creator ~/example.pdf
Creator                         : ScanSnap Manager #iX500

The file example.pdf has the correct tag/value pair and will be processed. The next file, covfefe.pdf, does not. You can add/modify the tag to the PDF which did not originate from a ScanSnap.

$ exiftool -creator="ScanSnap Manager #iX500" ~/covfefe.pdf 
    1 image files updated
$ exiftool -creator ~/covfefe.pdf
Creator                         : ScanSnap Manager #iX500

Voila! The ScanSnap Manager software will now process the PDF. You can certainly use free OCR software but I didn’t find any of them to be quite a slick. Plus this was more fun. 🙂

Welcome to the World of Tomorrow! (Again)

May 31, 2017May 31, 2017 marc

I wanted to follow-up on my previous post about scraping Futurama episode ratings from IMDb. I used tools I was familiar with to get the job done but I was told by someone that I really should check out BeautifulSoup to do it all in Python. It ended up working great and I’ll continue to use BeautifulSoup for web scraping in the future. This is what I did in the IPython interpreter:

import re, requests
import numpy as np
import pandas as pd
import scipy.stats as stats
from bs4 import BeautifulSoup

# create soup object

r = requests.get("http://www.imdb.com/title/tt0149460/eprate?ref_=ttep_sa_2")
soup = BeautifulSoup(r.content)

# scrape scores

scores = []
for score in soup.find_all("td", {"align": "right", "bgcolor": "#eeeeee"}):
    scores.append(float(score.get_text().strip()))

# scrape episodes

titles = []
for title in soup.find_all("a", {"href": re.compile("\/title\/tt")}):
    if len(title["href"]) == 17:
        titles.append(title.get_text().strip())

cols = ["IMDb Rating"]

# build dataframe

frame = pd.DataFrame(scores, titles, cols)

# maths with numpy

np.std(scores)
np.mean(scores)

# maths with pandas

s = pd.Series(scores)
s.std()
s.mean()
s.describe()

pd.Series.describe(frame)

# test for normal distribution

stats.normaltest(scores)

The ISTA 350: Programming for Informatics Applications course at the University of Arizona helped me a lot after my initial post. Additionally, the book Web Scraping with Python by Ryan Mitchell is one I’d recommend keeping handy.

Scraping IMDb Futurama Episode User Ratings

March 28, 2017March 28, 2017 marc1 Comment

Good news, everyone!

This entry is effectively a two-fer. It will show how I used some basic tools and a pinch of Python with numpy to get some of the data I needed for a class project. I took a look at my favorite television show, Futurama. I used the average Internet Movie Database (IMDb) user rating for each episode to see how many standard deviations away from the mean the top four episodes are. The ultimate goal of the project was different but this was a good way to use data to support facts.

Quick. Dirty. Scraping.

IMDb has a page with every Futurama episode and it’s average user rating. The URL is http://www.imdb.com/title/tt0149460/eprate?ref_=ttep_sa_2. Note that the direct-to-video movies are excluded (rightfully) from this list.

Let’s scrape that data!

$ curl -v http://www.imdb.com/title/tt0149460/eprate?ref_=ttep_sa_2 2>&1 | egrep -i 'users rated this' | cut -d' ' -f5 | cut -d'/' -f 1 > /tmp/scores.txt

Oddly enough the OSCP labs had me scrape this way frequently. I didn’t have the time to push the labs hard or take the practical but some information stuck. I’ll hopefully get back to that OSCP soon. 🙂

The above gives us a file (/tmp/scores.txt) with each Futurama episode user score on a new line. All I really want is the mean and standard deviation anyway — It’s easy to do with the Python interpreter.

>>> import numpy as np
>>> scores = []
>>> for line in open('/tmp/scores.txt', 'r'):
...   scores.append(float(line.strip()))
... 
>>> scores = np.array(scores)
>>> np.mean(scores)
7.8798387096774185
>>> np.std(scores)
0.58612723471593964

The mean is ~7.88 and the standard deviation is ~0.59 — I used this information to compare the top four episodes. (The highest rated episode is 2.745 standard deviations away from the mean!)

For those interested here’s a screenshot of the above in action:

The Amazon Echo Dot has a dump-ramdump.sh script

February 25, 2017 marc

The script’s existence is not proof that it is used but the expanded speculation around it is a fun exercise. The only fact I can give about the /bin/dump-ramdump.sh script is that it exists. (For now.)

I can say something different for another script, firewall.sh, on the system. I audited against it with nmap. I enabled features on the Dot (such as using Spotify to open TCP 4070) to test the script’s execution/logic. The ability to audit the script and observe behavior is crucial. The data supports that the firewall.sh script is used. (More would be better!) The images below are part of that audit; TCP 4070 being open after enabling Spotify and then a quick banner grab.

Unfortunately I’m unable to do the same level of observing with the dump-ramdump.sh script. I can’t knowingly trigger it, I don’t have a way to image an Amazon Echo Dot, and I don’t have a way to remotely connect and monitor it’s activity. The script appears to create new memdump logs in the /data/system/dropbox directory. I would love to know the fate of these logs and anything else in the /data/system/dropbox directory.

If you want a copy of the script you can download the system at Amazon Echo Update 567200820 (And where to download it!). Discovery of this script and other fun within the system happened late last year/early 2017. It’s been fun. 🙂

It’s worth noting that recently ArsTechnica ran the story of Amazon refusing to hand over data on whether Alexa overheard a murder, which puts a good perspective on information one could get (possibly) from Amazon about an Echo Dot user if they were motivated to do so. It’s a continuation of the involvement of an Echo in a murder case from 2016.

I wish I had more time to work on this system. Unfortunately taking 19 credits this semester has proven to be the challenge I was expecting. It’s something I still give attention but not at the level of intensity I would like. Hopefully this summer I can focus on it quite a bit more.

Tucson Computer Forensics and Information Security Meetup Slide Deck

February 18, 2017 marc

I recently presented for the Tucson Computer Forensics and Information Security Meetup. It was mostly on the content I’ve already talked about here – RiotPSA, Amazon Echo Dot system software, etc. I had told people there that the slide deck would be available.

I dislike slides/PowerPoint/etc. I generally make whatever I’m going to present that same day. I’d much rather talk and have a natural flow and slides are a bit limiting. In the end they’re more for me than they are for you. They keep me on track and guarantee that I don’t miss an important idea. These slides are essentially screenshots missing a ton of information given verbally and through live demo. In short: Please do not judge me based on these slides. They are here only because I promised them. I made them on a Saturday to show people for fun.

You can download the slides (PDF) here.

Special thanks to Mari DeGrazia (Twitter, Another Forensics Blog) for organizing the Meetup. Thanks to everyone who attended. It’s always a ton of fun. 🙂

Marc Padilla

Menu

Author: marc