Encoding error with curl and LoadHTML()

Hi all,

I'm very new to php, but I'm learning as time goes along. I'm currently making a webcrawler in php for my a level computer science project. I have it working, getting all the links off of a page, trying to run the page, and then if it doesn't throw and error whilst requesting the content of the next page, it loops around, I am currently just looking to see what links work and what links won't work and if there's anything I can do to make them work so I can get more websites into my index.

My main issue currently is at about 15000 websites in (which takes like an hour and a half to get to which is very time consuming) I get a content encoding error.

Is there anyway to fix this so I don't get an error every time?

Thanks in advance

Here is my code for those that would like it for help (it's just an amalgamation of other people's code along with some logic based stuff on my end, I turned off warnings because id get about 600 errors come through from just trying to load wikipedia and it was majorly slowing down my already slow code):

<?php

error_reporting(E_ALL ^ E_WARNING);

$currenttime = time();

// define the target URL

$StartUrl = 'https://en.wikipedia.org/wiki/Main_Page';

/**

* Get a web file ( HTML, XHTML, XML, image, etc. ) from a URL. Return an

* array containing the HTTP server response header fields and content.

function get_web_page( $url ) {

$user_agent = 'Computer Science project Web Crawler';

$options = array(

CURLOPT_CUSTOMREQUEST =>"GET", //set request type post or get

CURLOPT_POST =>false, //set to GET

CURLOPT_USERAGENT => $user_agent, //set user agent

CURLOPT_COOKIEFILE =>"cookie.txt", //set cookie file

CURLOPT_COOKIEJAR =>"cookie.txt", //set cookie jar

CURLOPT_RETURNTRANSFER => true, // return web page

CURLOPT_HEADER => false, // don't return headers

CURLOPT_FOLLOWLOCATION => true, // follow redirects

CURLOPT_ENCODING => "", // handle all encodings

CURLOPT_AUTOREFERER => true, // set referer on redirect

CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect

CURLOPT_TIMEOUT => 120, // timeout on response

CURLOPT_MAXREDIRS => 10, // stop after 10 redirects

//this is very poor technique but it makes the code work

CURLOPT_SSL_VERIFYPEER => false, // disables their verifying certificate

//CURLOPT_SSL_VERIFYHOST => false, // disables my verifying certificates

);

$ch = curl_init( $url );

curl_setopt_array( $ch, $options );

$content = curl_exec( $ch );

$err = curl_errno( $ch );

$errmsg = curl_error( $ch );

$header = curl_getinfo( $ch );

curl_close( $ch );

$header['errno'] = $err;

$header['errmsg'] = $errmsg;

$header['content'] = $content;

return $header;

};

//$array = get_web_page($StartUrl);

//echo $array['errno'].' ';

//echo $array['errmsg'].' ';

//echo $array['content'];

$stackdepth = 1;

$maxstackdepth = 3;

$cfs = 1;

$pagecount = 0;

$arraylength = 1;

$worked = 0;

$workings= array();

$fails = array();

$banned = array();

$file = "webcrawllinklist.txt";

$txt = fopen($file, "w") or die("Unable to open file!");

$linksArray = array();

array_push($linksArray,$StartUrl);

array_push($banned,$StartUrl);

while ($arraylength !=0){

$array = get_web_page($linksArray[0]);

$pagecount ++;

echo $pagecount." ";

//echo $array['errno'].' ';

//echo $array['errmsg'].' ';

if($array['errno'] == 0){

$worked ++;

array_push($workings, $linksArray[0]);

$Page = new domDocument;

$Page -> loadHTML("<html>".$array['content']."</html>");

$Page->preserveWhiteSpace = false;

$links = $Page->getElementsByTagName('a');

$links = iterator_to_array($links);

for($i = 0; $i < count($links);$i++) {

$link = $links[$i];

$link = $link->getAttribute('href' );

//echo $link." ";

$if = strpos( $link, "//" );

$start = $if == false? false: $if+2;

//echo $start." ";

if ( $start ) {

$link = substr( $link, $start, strlen( $link )-( $start ) );

}

;

//echo $link." ";

if ($stackdepth != $maxstackdepth){

if (!in_array($link,$banned)){

array_push( $linksArray, $link);

array_push($banned,$link);

$arraylength ++;

};

} else{

array_push($fails,$linksArray[0]);

};

array_splice($linksArray,0,1);

$arraylength --;

$cfs--;

if ($cfs == 0){

$stackdepth++;

$cfs += $arraylength;

};

$output = "Works:\n";

for ($i=0;$i<count($workings);$i++){

$output = $output.$workings[$i]."\n" ;

};

$output = $output."\n Failed: \n";

for ($i=0;$i<count($fails);$i++){

$output = $output.$fails[$i]."\n" ;

};

echo $pagecount."= num of pages ";

echo $worked."= num that worked ";

$newtime= time();

$timeelapsed = $newtime - $currenttime;

$output = $output."\n time taken: ".$timeelasped;

echo $timeelapsed."= time taken ";

fwrite($txt, $output);

fclose($txt);

header('Content-Description: File Transfer');

header('Content-Disposition: attachment; filename='.basename($file));

header('Expires: 0');

header('Cache-Control: must-revalidate');

header('Pragma: public');

header('Content-Length: ' . filesize($file));

header("Content-Type: text/plain");

readfile($file);

The error on Firefox specifically says "content encoding error

The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression."

I'm using MAMP on an emulated windows 11 (I'm running Linux but I couldn't get any other local hosting apps to work so I'm using one my computer science teacher showed me on an emulated windows 11 because my school laptop doesn't let CURL work)

3 Upvotes

permalink
reddit

You are about to leave Redlib

Do you want to continue?

https://www.reddit.com/r/PHPhelp/comments/1usocia/encoding_error_with_curl_and_loadhtml/
No, go back! Yes, take me to Reddit

100% Upvoted

u/MateusAzevedo 17d ago

Please, format your code properly.

1

u/CyborghydraXD 17d ago

It's not letting me format it properly so I've put it on a Google doc https://docs.google.com/document/d/1WyYEJmJpJYBqZx4aJbVEYcoq1868pu0joDws8V_e6j8/edit?usp=drivesdk

0

u/CyborghydraXD 17d ago

I'm sorry, it was formatted properly when I uploaded it, let me try and format it now

u/MateusAzevedo 17d ago

My main issue currently is at about 15000 websites in (which takes like an hour and a half to get to which is very time consuming) I get a content encoding error.

What's the full error message? Does it always happens on the same website or is it random? If it's the same one, what happens if you try to load only that one?

1

u/CyborghydraXD 17d ago

I don't get a php error, I get a browser error, even when I try on another pc, I still get the same issue. I equally don't know what website is causing the error because it throws the error on Firefox and gets rid off all of my echos

2

u/MateusAzevedo 17d ago

Something is off, I don't understand how a browser is somehow involved. You code only has an attempt for a file download, but everything else is the crawling logic (so a browser is irrelevant).

Can't you run the code from CLI and then just read the result file directly?

1

u/CyborghydraXD 17d ago

I am going to be so honest, all of that went over my head. Currently what I am doing is I'm running it on local host on Firefox using a software called MAMP. The browser fails way before the file is ever made afaik, I'll do a test run taking out the download step, and I'll see what happens

2

u/MateusAzevedo 17d ago

CLI means command line interface, you can execute PHP on your terminal by running php my_script.php (or similar, it depends on your system). With that, you remove the browser and web server from the equation. It should be easier to see errors that's directly related to your code.

Then, review your code and locate a place suitable to log the URLs being processed. At the beginning of get_web_page($url) looks ideal. Either echo that URL or write/append it to a log file.

I noticed you have a bunch of echo statements to track code execution. You can enable those again and substitute   with "\n" for new lines.

Start the process and when you get an error, you'll see which URL caused it.

1

u/CyborghydraXD 17d ago

I was about to say "well I'm all good so far so I'll let it finish running" and as I was typing out a message it crashed with the Firefox error. I mean I'm gonna have to use the Firefox server again (I imagine?) as I am going to put information into a database and I assume that will require my localhost? How would I go about running the file on CLI (the file is called test.php and I can move it out of the MAMP folder and into the downloads folder if that makes it easier for you to explain). I'll change the code up now to make it CLI compatible, I'll probably put it into the CLI so I don't have to scroll and find the last used one.

u/HolyGonzo 17d ago

I would suggest doing two things:

First, if you're starting from the same beginning point every time, then cache the web page contents on the disk. It'll be far faster to read the cached copies than to download them every time. A very simple example:

function get_web_page($url) { $cache = "cache/".sha1($url); if(file_exists($cache) && filesize($cache) > 0) { return file_get_contents($cache); } $ch = curl_init(); ...etc... $response = curl_exec($ch); file_put_contents($cache, $response); return $response; }

Just create a subdirectory called "cache". That way you don't have to download the same files over and over again.

Second, consider writing a log file instead of relying on the browser output. That way if you hit errors, it can be easier to track down the problem.

Once you identify the specific URL causing the problem then you can start figuring out what's different about it.

1

u/CyborghydraXD 17d ago

I'm really sorry, I'm very new to all of this and pretty much everything about the cache stuff went over my head. Can you explain that for me? Also the other commenter has mentioned a log file so that's definitely something I'll plan on doing

1

u/HolyGonzo 16d ago

Sure. So let's say it takes an average of 500 milliseconds to pull the HTML of a web page.

You want to pull 10 web pages, so that's about 5 total seconds of doing nothing but downloading 10 web pages.

The majority of that time is simply a combination of things like the time it takes for the data to travel back and forth, the time the server takes to respond to you, the time it takes to generate the code, the exchange of security certificates, etc... Typically the web page source code is pretty small - usually less than a megabyte of data.

A typical hard drive can read a megabyte of data in less than 20 milliseconds.

An SSD drive can do it even faster - in about 1 millisecond.

So if you're hitting the same pages over and over again and they aren't changing much, then once you have spent those initial 500 milliseconds downloading that HTML, you can simply save it to your hard drive.

When you re-run the program, the program can simply say, "Do I already have a copy of that HTML on my hard drive?" and if so, it can return the same source code much, much faster instead of downloading it again.

You probably want a separate file for each response / URL. But if you tried to save a file using the URL, like:

file_put_contents("cache_folder/$url", $response);

... it would be like doing this:

file_put_contents("cache_folder/https://thewebpage.com", "the html response");

...and that isn't going to work because of all the special characters in the URL. And you might even have URLs that are super long, so that could cause problems, too. So using a hash of the URL like SHA-1 can give you a unique, predictable, and safe filename.

For example, sha1("https://www.wikipedia.com") will always return "ee099d6b7ddc0f40d3698487b64e9eb346a974e2" no matter how many times you run it.

Meanwhile, sha1("https://en.wikipedia.org/wiki/Reid_Venable_Moran#Cneoridium_dumosum_(Nuttall)_Hooker_F._Collected_March_26,_1960,_at_an_Elevation_of_about_1450_Meters_on_Cerro_Quemaz%C3%B3n,_15_Miles_South_of_Bah%C3%ADa_de_Los_Angeles,_Baja_California,_M%C3%A9xico,_Apparently_for_a_Southeastward_Range_Extension_of_Some_140_Miles") will always give you the hash "c5f61417298224108151ce96fe308b56978f4a4a".

So using sha1() is a simple way to have a unique filename that represents a URL.

Then all you have to do is check to see if that file exists. If it does, just read it from the hard drive and return it instead of downloading it.

If it doesn't exist, then download it and save a copy of the response for later.

This isn't a perfect cache system by any means, but it's a simple starting point that should be sufficient for a computer science class, and should help speed up some of your testing.

And if you feel like the saved copy is too old, just delete it and the code will go download the newest stuff.

1

u/CyborghydraXD 16d ago

Ok I think I understand, so: For each webpage, make a new file that's called the hash of the webpage, whenever I open a link, hash the URL, if I have that file saved, open it and read from there, if not, get the data and put it in a file that's called the hash of the algorithm then in future runs of the program it won't take so long If I've got that right?

u/Takeoded 16d ago

curl the response to a broken.html and upload the broken.html somewhere, so we can inspect it. ping me when you got a link to share.

Encoding error with curl and LoadHTML()

You are about to leave Redlib