Welcome to the MacNN Forums.

Sanjan · Jan 19, 2007, 03:02 PM

I want a script to preform a seach on a website, then check a box from the results page (the check box is on the same line as the seach result) and finally it a button on the page?

If it is can someone point me in the right direction to make this as easily as pssible

Z's Mac · Jan 21, 2007, 08:42 AM

what's the website?

Brass · Jan 22, 2007, 07:53 PM

Sure it's possible. You've just got to connect to the web server (eg, using telnet) on the HTTP port (usually 80) and send the right commands (usually "GET" or "POST") with the required arguments to suit the URL (script) that usually receives such messages from that form.

The source of the web page containing the form would be required, but any web browser can show you the source of a web page you're viewing.

Z's Mac · Jan 22, 2007, 10:33 PM

Why would you use telnet vs a regular browser in your script?

besson3c · Jan 22, 2007, 11:18 PM

Originally Posted by Z's Mac

Why would you use telnet vs a regular browser in your script?

Because then you could do the operation faceless, although I'm not sure what this user wants to do with the output.

Z's Mac · Jan 23, 2007, 08:58 AM

faceless = no window opening to accomplish the task?

what's "faceless"?

besson3c · Jan 23, 2007, 10:14 AM

Originally Posted by Z's Mac

faceless = no window opening to accomplish the task?

what's "faceless"?

You got it. Unix cronjobs generally run without even the mere presence of a GUI being required. I think the suggestion was more in the spirit of the Unix way of automating tasks, although granted this would be a difficult one to handle this way.

Brass · Jan 23, 2007, 02:55 PM

Originally Posted by besson3c

You got it. Unix cronjobs generally run without even the mere presence of a GUI being required. I think the suggestion was more in the spirit of the Unix way of automating tasks, although granted this would be a difficult one to handle this way.

I think it would be MUCH easier to handle through telnet than to script a browser. Telling a browser to enter this text in this field, then click this button vs telling telnet to send this single message and listen for the response.

besson3c · Jan 23, 2007, 03:11 PM

Originally Posted by Brass

I think it would be MUCH easier to handle through telnet than to script a browser. Telling a browser to enter this text in this field, then click this button vs telling telnet to send this single message and listen for the response.

Yes, but then the challenge would be submitting the resulting form and saving this output into some usable format.

Brass · 03:53 PM

Submitting the form is simple, as is downloading the resulting form. Parsing it is the only really tricky bit. Then all you have to do is submit a "POST" via telnet, and read the response.

Reading the response, it can simply be saved to a text file with a .html extension.

Brass · Jan 23, 2007, 03:49 PM

EG, this is how simple it is, copied and pasted from my terminal window:

Code:
$ telnet www.thingy.com 80
Trying 101.101.101.101...
Connected to www.thingy.com.
Escape character is '^]'.
get
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>501 Method Not Implemented</TITLE>
</HEAD><BODY>
<H1>Method Not Implemented</H1>
get to /index.html not supported.<P>
Invalid method in request get<P>
</BODY></HTML>
Connection closed by foreign host.
$

(subsituted host names and IP addresses for fake ones)

In this case, you can see that I've entered a simple "get" with no arguments. This returns a web page (in this case an error saying that I'm not allowed to request /index.html).

Instead of entering "get", I could have entered a "get" or "post" to the relevant path WITH the arguments that the form would have submitted via a web browser.

besson3c · Jan 23, 2007, 03:57 PM

Originally Posted by Brass

EG, this is how simple it is, copied and pasted from my terminal window:

Code:
$ telnet www.thingy.com 80
Trying 101.101.101.101...
Connected to www.thingy.com.
Escape character is '^]'.
get
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>501 Method Not Implemented</TITLE>
</HEAD><BODY>
<H1>Method Not Implemented</H1>
get to /index.html not supported.<P>
Invalid method in request get<P>
</BODY></HTML>
Connection closed by foreign host.
$

(subsituted host names and IP addresses for fake ones)

In this case, you can see that I've entered a simple "get" with no arguments. This returns a web page (in this case an error saying that I'm not allowed to request /index.html).

Instead of entering "get", I could have entered a "get" or "post" to the relevant path WITH the arguments that the form would have submitted via a web browser.

How do you make POST arguments? I'd assume the GET arguments would simply be embedded into the URL:

Code:
get index.php?var1=blah&var2=blah2

How would you save the output?

Brass · 04:12 PM

Saving the output is just a matter of redirection.

However, you're right in that telnet is probably not the correct tool to use. Particularly, since many web sites will not respond nicely if you don't tell them which browser you are using (or pretending to use).

I've not used lynx before but I'm sure all this could be done with lynx, which is a CLI tool specifically for interacting with web servers (ie, a CLI web browser). It does have a "-post_data" option.

besson3c · Jan 23, 2007, 04:05 PM

Originally Posted by Brass

Saving the output is just a matter of redirection.

If you could download the contents to your Unix shell, game over, but how would one do this from a Telnet session?

However, you're right in that telnet is probably not the correct tool to use. Something more interactive is required. Particularly, since many web sites will not respond nicely if you don't tell them which browser you are using (or pretending to use).

I was thinking more along the lines of fetch, but tools like fetch/curl/wget are used for pulling data from sites primarily. I wonder if something like links or w3m could be scripted?

Brass · Jan 23, 2007, 08:19 PM

Originally Posted by besson3c

If you could download the contents to your Unix shell, game over, but how would one do this from a Telnet session?

Use an expect script?

I was thinking more along the lines of fetch, but tools like fetch/curl/wget are used for pulling data from sites primarily. I wonder if something like links or w3m could be scripted?

lynx is certainly scriptable. It's just a CLI utility, same as any other.

Mithras · Jan 24, 2007, 09:54 AM

I've used Perl's HTTP::Recorder package for this kind of thing in the past. It makes it pretty easy.

Another, possibly simpler option, is Firefox's Chickenfoot extension, which looks neat, though I haven't used it.