Basic HTTP and CURL

CURL is a powerful tool that we can use when we test web applications for security issues. It is a commandline utility for sending HTTP requests, but is rather low-level. To this end, we need to know a bit about HTTP itself to be able to use it, in addition to the command syntax for the application itself.

This article isn't really going to go over any specific attack methodologies, but instead is going to be a deep dive on HTTP and manipulating it with CURL.

Intro to HTTP

HTTP (Hypertext Transfer protocol) is the application layer protocol used to web pages. HTML, CSS, JavaScript, and the various binary files you view over the web are encapsulated in this protocol, which mostly serves to specify which resources are being requested and metadata about the request.

Here's example of a raw HTTP request: 


GET /index.php HTTP/1.1
Host: glowbox.cc
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Authorization: Basic bG9sLCBkaWQgeW91IHJlYWxseSB0aGluayBJIHdhcyBnb2luZyB0byBsZWF2ZSBteSBiYXNpY2F1dGggY3JlZGVudGlhbHMgaW4gcHVibGljIHZpZXc/IGMnbW9uIG1hbi4K
Connection: keep-alive
Referer: https://glowbox.cc/output.php?title=HTB+Legacy
Cookie: PHPSESSID=18h71e575ncr1vbsgqqsoapaf2
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
content-length: 0

And here's an HTTP response from the server to the request above: 


HTTP/1.1 200 OK
Date: Fri, 08 Sep 2023 14:31:10 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 1953
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8

<!doctype html>
<html>
    <head>
        <title>dshel's notes</title>
        <link rel="stylesheet" type="text/css" href="style.css">
    </head>

[...]

We'll refer back to these examples more, but for the most part will be focusing on HTTP requests, since what we'll be using CURL for the most will be to create crafted request packets.

Methods

HTTP has various methods which are used to send data to the server. By default, CURL will use the GET method, as this is what is used to retrieve web pages.

In curl, we specify the method with -X, for example: 


curl -X GET https://example.com/index.php

This would make a GET request to https://example.com/index.php and then output that to STDOUT for us.

GET

GET is the default method which is used for making read-only requests to web servers. This is how you typically make a request for a web page when you're doing conventional browsing.

GET can also send user submitted data to a website. This is handled by URL parameters. As you're reading this article, there's likely a URL parameter in the navigation bar now, as I use GET parameters to select which article to display. They would look like this: 


https://example.com/output.php?title=HTB+Legacy

After the name of the file we're accessing, we'll see ?title=HTB+Legacy. This is the parameter. ? tells the browser that the next things are parameters, title is the variable name and HTB+Legacy is the value.

If we had multiple parameters, we could string them together with & like so: 


?title=HTB+Legacy&somefakeparam=some+data+for+the+variable

Note that after HTB+Legacy we have &somefakeparam= The & shows that we have another parameter to add, and somefakeparam is it's name. Note also the +'s where spaces should be in the value. This is because we should usually send parameters in a URL encoded format. In order to make URL encoded strings in Linux, we can do this: 


urlencode "the thing you want encoded"
the%20thing%20you%20want%20encoded

Note that + and %20 mean the same thing (space); urlencode just likes to use %20.

Since they're just sorta tacked onto the URL, using these in CURL is no brainer: 


curl -X GET https://example.com/index.php?someparam=some+data&otherparam=more+data

POST

POST is another common method you'll encounter in casual browsing which is usually used to submit data via HTML forms. Conceptually, there isn't much different about POST from GET. However, instead of embedding the user parameters into the URL, the parameters are send in the request body. Here's an example of a POST request with 2 parameters: 


POST /index.php HTTP/1.1
Host: 127.0.0.1
User-Agent: curl/7.81.0
Accept: */*
Proxy-Connection: Keep-Alive
Content-Length: 59
Content-Type: application/x-www-form-urlencoded

exampleparam=the+quick+brown+fox&secondparam=some+more+data

We can see the parameters there at the bottom in the request message body.

Doing this in CURL, we need to use -X to specify that we want to use POST, and then -d to tell it the parameters. Notice that the parameters are sent in pretty much the same format as GET parameters, we just don't have to have ? at the beginning anymore: 


curl -X POST http://127.0.0.1/index.php -d 'exampleparam=the+quick+brown+fox&secondparam=some+more+data'

Other Methods

When performing activities such as API enumeration, we may want to also experiment with PUT and DELETE.

PUT functions similarly to POST in that it takes the -d flag to send data (in the context of APIs, this is usually json data).

DELETE functions like GET, but attempts to delete the requested resource.

Modern browsers don't usually implement using these methods, so about the only time you'll ever see them (or any other methods) will be when we're doing API enumeration or other types of low level sever pentesting.

Of course, in CURL, we can select them at any time using -X just like with GET and POST.

Headers

Headers are metadata sent with a request or response that provide useful information relevant to the context of a request. In CURL, we'll use the -H flag to specify headers. Here's some common request headers we'll likely use when enumerating a site:

Header Description example
User-Agent This is a string that's supposed to tell the server what browser you're supposed to be using. curl -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0' https://example.com
Referer If you're arriving on a page through a link, this is the page that link was on. curl -H 'Referer: https://glowbox.cc/index.php' https://example.com
Cookie Cookies are variables stored in files on the client. Often these are used for things like session IDs or login tokens, so if you're sending requests to an authenticated app, you'll probably need to send a cookie value. curl -H 'Cookie: PHPSESSID=18h71...' https://example.com
Authorization This is used to pass credentials to technologies like basic auth. I.e. to authentication systems that are implemented by the server and not by the web app. curl -H 'Authorization: Basic bG9sLCBka...' https://example.com
Content-Type This tells the server what format data you're sending is. This will be formatted as a mime type. If the server expects one format, but doesn't validate that anywhere else, changing this could allow for file upload bypasses. curl -H 'Content-Type: application/x-httpd-php' -d @somefile.php https://example.com

There are many other headers you can take advantage of. Do some deeper diving in Mozilla's HTTP documentation.

In a lot of cases, it's not so much important to have 9001 different headers memorized (although it's not harmful, for sure), but rather moreso useful to be able to observe the headers sent by an application and then modify them on the fly as seems fit for the scenario.

X-headers

Sometimes you may see headers beginning with x- like x-metrics or something. The x- prefix is a convention for denoting nonstandard headers, which are used internally by some platforms and web applications. You'll need to consult the documentation for whatever you're working with to understand what these mean, or else figure it out yourself through enumeration.

Multipart/Form-Data

Multipart forms are used to send a HTML form with many fields. Classically, they're used for file uploads, but really a developer can use them for any situation where they have a form with multiple fields.

To explain these a bit, let's consider the following HTML: 


<form action="handler.php" enctype="multipart/form-data">
    <input type="text" name="user">
    <input type="file" name="upload">
    <input type="submit" name="submit" value="Upload File">
</form>

In the above, we have a multipart form (you can tell this from enctype="multipart/form-data") that has 3 fields: a text field called user, a field upload called upload and the submit button for the form, which is called submit.

To send this with CURL, we'll use the -F flag for each form input, like so: 


curl -F 'user=jim' -F 'upload=@somepic.jpg' https://example.com/handler.php

Notice for the file upload we say @somepic.jpg. The @ operator here means that the input should be populated by the contents of a file on our system called "somepic.jpg". When files are uploaded, they're also sent with the built in parameters filename and content-type. By default, CURL will use the name of the pointed to file, but for various tests, we may need to change this (e.g. for testing path traversal in that parameter). We may also need to change the content-type in order to attempt to bypass mime type filters. In order to manually modify these, we might do something like this: 


curl -F 'user=jim' -F 'upload=@somepic.jpg; filename=../../../../etc/passwd; type=text/html' https://example.com/handler.php

Here we've added the filename= and type= fields manually. Note that like most filetype descriptors, we're going to use mime types to describe the filetype. The example above is a pretty nonsensical madlib of file types and names, etc, but should convey the gist of how you would use these fields.

Other useful CURL options

Here's some other options that can be used which aren't quite as everyday as the above, but should still be known:

Parameter Description
--proxy Use it like curl --proxy "http://127.0.0.1:8080" https://example.com to point curl through a web proxy like Burp or Mitmproxy, where 127.0.0.1:8080 is your web proxy.
-w This is for outputting information rather than the HTTP response. Use it with the variables referenced here like curl -w %{http_code} https://example.com